Out-of-Sample Testing

Out-of-sample testing evaluates a model on data it never saw during training or tuning, which is the only way to estimate how it will perform on genuinely new data. It is closely related to Walk-Forward Backtesting, Overfitting, Cross-Validation and Hit Rate (Directional Accuracy), each defined in the hedgewing.ai glossary.

In-sample performance is what a model achieves on the data it was fitted to. Out-of-sample performance is what it achieves on data held back from every stage of development, including feature selection and hyperparameter tuning. The gap between the two is the most important diagnostic in applied machine learning: a small gap suggests the model learned something general, a large gap means it memorised the training set.

In finance the distinction is sharper than in most fields because the signal-to-noise ratio is so low. It is easy to find a rule that fits the past perfectly and has no predictive value at all. The strongest form of out-of-sample evidence is forward testing: making predictions, recording them with timestamps, and scoring them as outcomes resolve. Walk-forward backtesting simulates this historically; a live, resolved prediction log is the real thing.

This is why hedgewing.ai labels its on-demand backtests as in-sample and illustrative rather than as a track record. They show how the current models describe the recent past, which is useful for understanding behaviour, but they are not evidence of future performance. The evidence that matters accumulates as published forecasts resolve, and investors should hold every tool, including this one, to that standard.

FAQ

What is Out-of-Sample Testing?

Out-of-sample testing evaluates a model on data it never saw during training or tuning, which is the only way to estimate how it will perform on genuinely new data.

Which terms are related to Out-of-Sample Testing?

Out-of-Sample Testing is closely related to Walk-Forward Backtesting, Overfitting, Cross-Validation and Hit Rate (Directional Accuracy). Each has its own entry in the hedgewing.ai glossary.

Is this definition of Out-of-Sample Testing investment advice?

No. This glossary entry explains what Out-of-Sample Testing means. It is educational reference material, not personalised investment advice and not a recommendation to buy or sell any security.

Create a free Hedgewing account to try hedgewing.ai yourself.