Jan 2, 2020 in Data Science
Q: If you are having 4GB RAM in your machine and you want to train your model on 10GB data set. How would you go about this problem? Have you ever faced this kind of problem in your machine learning/data science experience so far?

1 Answer

0 votes
Jan 2, 2020

First of all, you have to ask which ML model you want to train.

For Neural networks: Batch size with Numpy array will work.

Steps:

1. Load the whole data in the Numpy array. Numpy array has a property to create a mapping of the complete data set, it doesn’t load complete data set in memory.

2. You can pass an index to Numpy array to get required data.

3. Use this data to pass to the Neural network.

4. Have a small batch size.

For SVM: Partial fit will work

Steps:

1. Divide one big data set in small size data sets.

2. Use a partial fit method of SVM, it requires a subset of the complete data set.

3. Repeat step 2 for other subsets.

However, you could actually face such an issue in reality. So, you could check out the best laptop for Machine Learning to prevent that. Having said that, let’s move on to some questions on deep learning.

 

Click here to read more about Loan/Mortgage
Click here to read more about Insurance
...