Cross-validation (statistics)

Cross-validation is a family of model validation techniques used to judge how well the results of a statistical analysis or predictive model will generalize to an independent data set. Instead of trusting how well a model fits the data it was trained on, which tends to be optimistic, the sample is divided into complementary subsets: the model is fitted on a training set and tested on a held-out validation set, and the process is repeated over several different partitions whose results are averaged into one estimate of predictive performance. It is used to flag problems such as overfitting and selection bias, and to give insight into how a model will behave on data it has never seen. The family splits into exhaustive methods, such as leave-p-out and leave-one-out, which test every possible division of the sample, and non-exhaustive methods, such as k-fold, holdout, repeated random sub-sampling (Monte Carlo) and nested variants used when hyperparameters are tuned at the same time. It is also known as rotation estimation or out-of-sample testing.