Ludwig J. Marx.

How the tests work

The rules below come from a mistake. In an earlier study a filter silently dropped two thirds of the stocks, and a return formula overstated the result. Nobody noticed until a second implementation produced different numbers. Each rule is meant to make that kind of error visible earlier.

Reporting

Bottom line first, with a level of confidence

A paper is written like an assessment. The first page gives the result and the key judgments. Each judgment carries one of three levels, and the level follows from the tests it passed, not from how convincing it feels.

High
Holds out of sample, in both sub-periods and in the independent replication (R against QuantConnect).
Moderate
Holds in sample and in the replication; weaker or not yet tested out of sample.
Low
First finding, few cases or a single period. Stated so that nobody builds on it.
Rules

Written down before the first result

Hypothesis, key figure, threshold, samples and costs are fixed before any result is seen. The test sample is kept apart and opened once.

Tested against chance

Every claim is compared with a null model: random entries with the same timing, random portfolios with the same size and turnover, block bootstrap or shifted signals. A rule only counts if it sits outside that distribution.

A positive control for every test

An artificial effect is built into the data, and the procedure has to find it. A test that cannot find a known effect cannot show that there is none.

A filter log

Raw data is never changed. Every step that removes rows is logged with its count. A filter that removes more than half of the data stops the run until the limit is raised on purpose. Delisted stocks stay in the universe.

Count the attempts

The number of variants tried is recorded, and results are reported with the deflated Sharpe ratio, which corrects for that number.

Real costs

Results are net of the costs of the broker that would actually trade the strategy.

Two independent implementations

Every study is computed in R and rebuilt in QuantConnect. If the two disagree in size or sign, nothing is reported until the reason is found.

Failures stay on record

A finding that is later refuted stays in the study notes, struck through, with the reason and the date.