Probability of Backtest Overfitting (PBO): How to Tell If Your Backtest Is Edge or Luck

The Probability of Backtest Overfitting (PBO) is the chance, from 0 to 1, that your best backtest falls below the median of its rivals out of sample.

Test a hundred parameter sets, keep the best, and part of that "best" is luck. The question is not whether luck was involved. It is how much. PBO measures exactly that — and replaces a beautiful equity curve with a number you can defend in a decision.

The short answer

• What PBO is: the probability that your selection process — try N configurations, keep the best — delivers a below-median choice out of sample.
• How it is computed: with CSCV (combinatorially symmetric cross-validation). Split the data into S blocks, try every half/half combination as in-sample/out-of-sample, and count how often the in-sample winner falls below the median outside it.
• What a good value looks like: near 0 means your selection carries real information. Around 0.5 means it is no better than a coin flip.
• Primary source: Bailey, Borwein, López de Prado and Zhu, "The Probability of Backtest Overfitting," Journal of Computational Finance.

The problem: a backtest is not an experiment, it is a contest

In what makes an automated trading strategy good, I argued that a rising equity curve is not evidence. The reason is statistical, not aesthetic. Once you test several configurations, the backtest stops being a measurement and becomes a contest — and the winner of any contest, even among worthless entries, posts a large number.

This can be computed exactly. Suppose you have N independent strategies and all of them have a true Sharpe ratio of zero. How high is the best one's annualized backtest Sharpe, on average? The table gives the Minimum Backtest Length (MinBTL): how many years of data you need before the best of N worthless candidates is no longer expected to show an annualized Sharpe of 1.

In practice: with two years of data and only seven parameter tweaks, you should expect at least one version to show a Sharpe of 1 — even if none has any edge. A routine optimization run, such as hyperopt in Freqtrade, tries thousands of configurations overnight. (The table uses the expected-maximum formula from "Pseudo-Mathematics and Financial Charlatanism," checked against a Monte Carlo simulation.)

Why the usual defences fail

A single in-sample/out-of-sample split is not enough. If you change parameters after seeing the out-of-sample result — and nearly everyone does — the out-of-sample data has become training data. A single split is also one draw of the price path; it may simply have been kind to you.

Walk-forward is still one path. It is useful, but it tests one time sequence and says nothing about how dependent your selection process is on luck.

Fix data errors separately. PBO measures overfitting from selection, not data defects. Look-ahead bias and broken OHLCV data inflate every configuration at once, and PBO cannot see them. Clean the data first, then measure overfitting.

CSCV, step by step

1. Build a return matrix. For every configuration you tried — all of them, not just the good ones — keep the series of period returns. The result is a T×N matrix: T periods, N configurations.
2. Split the rows into S contiguous time blocks. S is even, typically 8 to 16. Time order is preserved within each block.
3. Form every combination of S/2 blocks. For S = 16 that is 12,870 combinations. Each combination plays in-sample; the remaining blocks are out-of-sample.
4. In each combination, do what you do in reality: pick the best in-sample configuration by your own metric (for example, Sharpe).
5. Find that configuration's relative rank out of sample — a number between 0 and 1 — and take its logit: λ = ln(ω / (1 − ω)).
6. PBO is the share of combinations where λ ≤ 0: the in-sample winner dropped into the bottom half outside it.

CSCV's strength is its symmetry: every block is out-of-sample as often as it is in-sample, so the result does not depend on one arbitrary split.

A Python implementation

This minimal version contains the whole method. The input is the return matrix of every configuration you tested:

import itertools
import numpy as np


def sharpe(r):
    sd = r.std(axis=0, ddof=1)
    return np.where(sd > 0, r.mean(axis=0) / sd, 0.0)


def pbo_cscv(returns, n_splits=16, metric=sharpe):
    """Probability of Backtest Overfitting via CSCV.

    returns: array (T, N) - one column of period returns per configuration tried.
    n_splits: S, even; the number of IS/OOS combinations is C(S, S/2).
    """
    T, N = returns.shape
    blocks = np.array_split(np.arange(T), n_splits)
    logits = []
    for is_ids in itertools.combinations(range(n_splits), n_splits // 2):
        oos_ids = [b for b in range(n_splits) if b not in is_ids]
        is_rows = np.concatenate([blocks[b] for b in is_ids])
        oos_rows = np.concatenate([blocks[b] for b in oos_ids])
        best = np.argmax(metric(returns[is_rows]))        # what you would pick
        oos_perf = metric(returns[oos_rows])
        rank = (oos_perf < oos_perf[best]).sum() + 1      # 1 ... N
        w = rank / (N + 1)                                # relative rank in (0, 1)
        logits.append(np.log(w / (1 - w)))
    logits = np.array(logits)
    return (logits <= 0).mean(), logits

Sanity check: on fifty configurations of pure random noise, mean PBO across forty independent samples came out at about 0.48 — the theoretical 0.5 (a single sample can swing between roughly 0.3 and 0.7). In a separate test with a hundred configurations, giving five of them a small real edge drove PBO to zero. If your implementation returns something near zero on pure noise, data is leaking somewhere. In R, the pbo package on CRAN implements the same method.

How to read a PBO number

These thresholds are my working rule, not part of the original paper, which defines no universal cut-off. Alongside PBO, read two other CSCV outputs: performance degradation (the slope of out-of-sample on in-sample performance — negative means the better it looks inside, the worse it does outside) and probability of loss (the share of combinations where the winner lost money out of sample).

PBO, Deflated Sharpe and Minimum Backtest Length: which for what?

These are not competitors; they are layers of one gate. MinBTL tells you, before research starts, how many trials you can afford. PBO tells you, after the search, whether the selection was sound. Deflated Sharpe tests significance on the single finalist.

Where PBO sits in a quant system

PBO is not a report you write once. It is a gate in the strategy pipeline. In quant system design, the path from hypothesis to real capital has several gates, and this one sits between research and paper trading:

1. Log every trial. Every configuration that ran is stored with its returns. Without that log PBO cannot be computed — the most common reason teams never compute it.
2. Trial budget. Before searching, use MinBTL to set how many independent configurations your data can support.
3. Statistical gate. PBO and Deflated Sharpe on the search output. Failure means back to the hypothesis, not another round of tuning.
4. Paper trading against expectation. Only candidates that clear the gate run live without capital; that is where the gap between backtest and reality gets measured.

The subtle trap: if a high PBO makes you change parameters and recompute PBO, that loop is itself a fresh source of overfitting. A failed gate should send you back to the source of the edge, not into more search.

Common mistakes

• Counting only the "serious" configurations. Any trial whose result you saw and acted on is a trial — including the ones you deleted from your notebook.
• Blocks that are too small. If each block holds only a handful of trades, the performance metric is noise and PBO is meaningless. Each block needs enough trades for its Sharpe to be a stable number.
• Near-identical configurations. A hundred variants differing slightly in one parameter are not a hundred independent trials. PBO absorbs some of this, but for MinBTL you need an estimate of the effective number of trials.
• Treating a low PBO as profitability. Low PBO means your selection is not luck. It does not say you are profitable after costs. That is a separate test.

Frequently asked questions

What exactly does PBO measure?

The probability that the in-sample best configuration ranks below the median of all tested configurations out of sample. Its subject is the selection process, not a particular strategy.

How many configurations do I need to compute PBO?

At least two, but the number only means something when you include every configuration you actually tested. Dropping failed trials artificially lowers PBO.

What should S (the number of blocks) be?

Usually 8 to 16. A larger S gives more combinations (12,870 at S = 16) but smaller blocks. Each block must hold enough data for the performance metric to stay stable.

How is PBO different from the Deflated Sharpe Ratio?

PBO evaluates the selection process across all trials. Deflated Sharpe adjusts one strategy's Sharpe for the number of trials and for non-normal returns. The first asks "was the selection sound?", the second "is this number significant?".

Does walk-forward analysis replace PBO?

No. Walk-forward tests one path through time. CSCV tests every symmetric split of the data and returns a distribution, not a single number. The two are complementary.

Does it apply to crypto strategies?

Yes, arguably more so: shorter histories, shifting regimes and push-button optimizers mean many trials on little data — exactly the conditions in which overfitting thrives.

References

• Bailey, D. H., Borwein, J. M., López de Prado, M., Zhu, Q. J. — The Probability of Backtest Overfitting, Journal of Computational Finance.
• Bailey, D. H., Borwein, J. M., López de Prado, M., Zhu, Q. J. — Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance, Notices of the AMS, 2014.
• Bailey, D. H., López de Prado, M. — The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality, Journal of Portfolio Management, 2014.

If your system tests more than a few dozen configurations and still has no statistical gate between research and execution, start with quant system design: diagnosis first, then tools.