Validation

See also: Machine learning terms

Introduction

Validation checks the quality of the model's predictions by testing the model against the new data in validation set. Validating a machine learning model requires labeled data that can be used for training and testing. Usually, a dataset is divided into 3 sets: a training set, validation set and test set. The training set of data instructs the model how to classify or predict outcomes based on input data, while the validation set evaluates the model's accuracy and performance. Validation prevents the model from overfitting to the training set. Validation can be thought of as the first around of testing and evaluating the model while test set is the 2nd round.

Validation Methods

Validating a model requires different approaches, each with their own advantages and drawbacks. Three common techniques for validation are k-fold cross validation, hold-out validation, and leave-one-out validation.

k-Fold Cross-Validation

K-fold cross validation (kFCV) is a popular technique that involves splitting the data into k equal subsets. One subset serves as the testing set, while the remaining k-1 subsets train the model. This cycle repeats itself k times with each subset being tested once. After averaging these results, an estimate of their accuracy can be made.

Hold-Out Validation

Hold-out validation involves dividing the data into training and testing sets. Usually, a large portion of this information goes toward training the model, while the remainder serves for testing. While this approach is straightforward and straightforward to execute, it may not provide an accurate representation of model performance if the testing set is too small or not representative of all available information.

Leave-One-Out Validation

Leave-one-out validation involves training the model on all but one data point and testing it on the remaining one. This process is repeated for each data point in the set, with results then averaged. This approach works best when working with small datasets but may prove computationally expensive for larger ones.

Evaluating Model Performance

Once a validation method is selected, its performance is assessed using several metrics. These include accuracy, precision, recall, F1 score and area under the receiver operating characteristic curve (AUC-ROC).

Accuracy

Accuracy is the percentage of correctly classified instances within a testing set. It provides an easy-to-understand gauge of a model's performance.

Precision and Recall

Precision measures the percentage of true positive predictions among all predicted positives, while recall evaluates the proportion of true positives among all actual positives. Precision and recall are often combined to assess a model's performance when there is an imbalance in class size.

F1 Score

The F1 score is the harmonic mean of precision and recall. It can be a useful metric when both precision and recall are important factors.

AUC-ROC

AUC-ROC is a measure of a model's capability to discriminate between positive and negative instances. It's calculated as the area under the curve on an ROC plot. A model with a higher AUC-ROC value will be better at discriminating between positive and negative instances.

Explain Like I'm 5 (ELI5)

Imagine you're teaching a robot how to recognize something like a cat.

Show the robot pictures of cats and tell it, "This is a cat." Then test its accuracy by showing another picture of a cat and asking, "Is this this another cat?"

If the robot correctly recognizes a cat, then you know it has learned how to recognize one. But if not, then further practice is needed in order for it to get it right.

Validation in machine learning is like this test; it helps us verify that our robot (or machine learning model) learned how to do something correctly.

We give the model examples of what we want it to learn, and then ask it for its opinion on new examples. We compare its answers with what we know is correct, and if the model consistently gives the right answer then we know it has learned how to perform the task correctly. If not, then additional practice may be necessary.