in Artificial Intelligence by

What is ‘training Set’ and ‘test Set’ in a Machine Learning Model? How Much Data Will You Allocate for Your Training, Validation, and Test Sets?

1 Answer

0 votes

There is a three-step process followed to create a model:

1) Train the model

2) Test the model 

3) Deploy the model

Training Set

The training set is examples given to the model to analyze and learn

70% of the total data is typically taken as the training dataset

This is labeled data used to train the model

Test SET

The test set is used to test the accuracy of the hypothesis generated by the model

Remaining 30% is taken as testing dataset

We test without labeled data and then verify results with labels

Consider a case where you have labeled data for 1,000 records. One way to train the model is to expose all 1,000 records during the training process. Then you take a small set of the same data to test the model, which would give good results in this case.

But, this is not an accurate way of testing. So, we set aside a portion of that data called the ‘test set’ before starting the training process. The remaining data is called the ‘training set’ that we use for training the model. The training set passes through the model multiple times until the accuracy is high, and errors are minimized.

Train Set

Now, we pass the test data to check if the model can accurately predict the values and determine if training is effective. If you get errors, you either need to change your model or retrain it with more data.

Test Set

Regarding the question of how to split the data into a training set and test set, there is no fixed rule, and the ratio can vary based on individual preferences. 

Related questions

+1 vote
asked Jan 6 in Structured Data Classification by sharadyadav1986
0 votes
asked Aug 29, 2019 in Artificial Intelligence by Robin