How the tests work
The rules below come from a mistake. In an earlier study a filter silently dropped two thirds of the stocks, and a return formula overstated the result. Nobody noticed until a second implementation produced different numbers. Each rule is meant to make that kind of error visible earlier.
Bottom line first, with a level of confidence
A paper is written like an assessment. The first page gives the result and the key judgments. Each judgment carries one of three levels, and the level follows from the tests it passed, not from how convincing it feels.
- High
- Holds out of sample, in both sub-periods and in the independent replication (R against QuantConnect).
- Moderate
- Holds in sample and in the replication; weaker or not yet tested out of sample.
- Low
- First finding, few cases or a single period. Stated so that nobody builds on it.
Written down before the first result
Hypothesis, key figure, threshold, samples and costs are fixed before any result is seen. The test sample is kept apart and opened once.
Tested against chance
Every claim is compared with a null model: random entries with the same timing, random portfolios with the same size and turnover, block bootstrap or shifted signals. A rule only counts if it sits outside that distribution.
A positive control for every test
An artificial effect is built into the data, and the procedure has to find it. A test that cannot find a known effect cannot show that there is none.
A filter log
Raw data is never changed. Every step that removes rows is logged with its count. A filter that removes more than half of the data stops the run until the limit is raised on purpose. Delisted stocks stay in the universe.
Count the attempts
The number of variants tried is recorded, and results are reported with the deflated Sharpe ratio, which corrects for that number.
Real costs
Results are net of the costs of the broker that would actually trade the strategy.
Two independent implementations
Every study is computed in R and rebuilt in QuantConnect. If the two disagree in size or sign, nothing is reported until the reason is found.
Failures stay on record
A finding that is later refuted stays in the study notes, struck through, with the reason and the date.