Categories

Jan 2 in Data Science

Q: If you are having 4GB RAM in your machine and you want to train your model on 10GB data set. How would you go about this problem? Have you ever faced this kind of problem in your machine learning/data science experience so far?

1 Answer

Jan 2

First of all, you have to ask which ML model you want to train.

For Neural networks: Batch size with Numpy array will work.

Steps:

1. Load the whole data in the Numpy array. Numpy array has a property to create a mapping of the complete data set, it doesn’t load complete data set in memory.

2. You can pass an index to Numpy array to get required data.

3. Use this data to pass to the Neural network.

4. Have a small batch size.

For SVM: Partial fit will work

Steps:

1. Divide one big data set in small size data sets.

2. Use a partial fit method of SVM, it requires a subset of the complete data set.

3. Repeat step 2 for other subsets.

However, you could actually face such an issue in reality. So, you could check out the best laptop for Machine Learning to prevent that. Having said that, let’s move on to some questions on deep learning.

 

Click here to read more about Loan/Mortgage
Click here to read more about Insurance

Related questions

Madanswer
Jan 17 in Data Science
Nov 29, 2019 in Machine Learning
Feb 26
...