in Data Science by
If you are having 4GB RAM in your machine and you want to train your model on 10GB data set. How would you go about this problem? Have you ever faced this kind of problem in your machine learning/data science experience so far?

1 Answer

0 votes

First of all, you have to ask which ML model you want to train.

For Neural networks: Batch size with Numpy array will work.


1. Load the whole data in the Numpy array. Numpy array has a property to create a mapping of the complete data set, it doesn’t load complete data set in memory.

2. You can pass an index to Numpy array to get required data.

3. Use this data to pass to the Neural network.

4. Have a small batch size.

For SVM: Partial fit will work


1. Divide one big data set in small size data sets.

2. Use a partial fit method of SVM, it requires a subset of the complete data set.

3. Repeat step 2 for other subsets.

However, you could actually face such an issue in reality. So, you could check out the best laptop for Machine Learning to prevent that. Having said that, let’s move on to some questions on deep learning.


Related questions

+1 vote
asked Mar 7, 2021 in PL/SQL by SakshiSharma
0 votes
asked May 2, 2020 in Data Science by James
0 votes
asked Mar 31, 2020 in DBMS by amita rallin