Strategy research

How we test GSDAQ strategies with automated trading

We have run strategies in automated trading since March 2026. Why backtests overstate, and how pre-registration, a holdout and forward tests decide.

By June Kim · How we research · 2026-09-27

Why do backtests look better than real results?

Usually because the rule was chosen after seeing the same data it is graded on. Try enough moving-average lengths or exit rules and one will fit the past well by luck alone. Backtests also tend to skip what the real world charges for: companies that were later delisted, the delay between a signal and a fill, spreads and slippage. Harvey, Liu and Zhu argue that after decades of data mining, a newly reported return factor should clear a t-ratio above 3.0 rather than the customary 2.0. We therefore read any backtest as an upper bound, not a forecast. Since March 2026 our in-house agent system has also run our strategies in automated trading, so our rules are judged against live markets, not only history. This guide describes the research process GSDAQ uses as of September 27, 2026. It reports no strategy results and is not investment advice. Harvey, Liu & Zhu (NBER w20592): … and the Cross-Section of Expected Returns

What does pre-registration mean for a trading rule?

It means writing down the hypothesis, data, costs and exact pass line before a test runs, and not moving that line after seeing the result. The Center for Open Science describes preregistration as specifying a research plan in advance so planned work can be told apart from unplanned work, because the same data cannot both generate and test a hypothesis. In our research the pass line is written into the study file before it is executed. If a candidate misses it, the current rule stays. If we decide a standard was wrong, we close that entry and register a new one rather than quietly editing it. The record is internal, not a public registry or an outside audit, which is one reason we do not publish strategy returns. Center for Open Science: What is preregistration?

On narrow screens, scroll the table sideways. Keyboard users can focus the table and use the arrow keys.

What we write down before a test runs
Item	What we record	Why it matters
Hypothesis Source	One sentence on what should happen and why	A vague idea can be reinterpreted after any result
Universe	Past index membership, including companies later delisted	Today’s survivors make the past look better
Timing and costs	Signal from a close, trade in the next session, a cost on every trade	A same-close fill with no cost is not achievable
Pass line	Beat the current rule, survive a resampling test for luck, agree in the holdout	Fixed first, so the goalpost cannot move
Paper terms	Budget, minimum days and trades, decision date	An early lucky streak cannot be declared a win

What is a holdout, and why is it used only once?

A holdout is a stretch of data, usually the most recent years, that stays untouched while a rule and its settings are chosen, and is then used once to judge them. If you look at it, adjust a setting and look again, it has quietly become training data. In our studies the holdout start date is fixed in the code before the run, and a candidate must show the same direction of result there as in the training years. Some studies repeat this across several training and holdout splits. We also check that neighboring settings point the same way, so the choice sits on a plateau rather than a lucky spike. A holdout alone is not enough: Bailey, Borwein, López de Prado and Zhu note that standard holdout methods can be unreliable for investment backtests. That is why a forward test comes next. Bailey, Borwein, López de Prado & Zhu: The Probability of Backtest Overfitting (PDF)

Our testing sequence from a written plan to a human decision — The order our research follows. Passing one stage does not skip the next.

Why do forward results get the final say?

Because they are recorded after the rule is frozen, so nothing from the future can leak in. We have run strategies in automated trading, in live and paper accounts, since March 2026. Each paper test in our registry states its hypothesis, a budget, a minimum number of trading days and trades, the conditions it must meet and a decision date. Until the minimum sample is reached the verdict is “not enough data”: the numbers can be shown, but no pass or fail is declared. A proposed change runs as a separate variant beside the original on the same days instead of replacing it. Paper fills are still optimistic. Broker documentation such as Alpaca’s says paper trading does not account for market impact, slippage from latency or order queue position. Some inputs, such as scores built from data with no historical snapshots, cannot be backtested honestly at all; only a forward record can speak for them. Alpaca documentation: paper trading and its limitations

How do we try not to fool ourselves?

We count our attempts and put proposals through a separate critical review. Test 20 rules that have no real edge at a 5% significance level, and on average one will look significant by chance. By late August 2026 our research folder held more than 1,600 tested variants, so a best-looking result has to be judged against that many tries. Our larger studies report measures built for this problem: the Deflated Sharpe Ratio, which adjusts for selection among many trials and for non-normal returns, and the probability of backtest overfitting. Some studies also compare a ranking rule with random picks from the same universe. The critical review looks for data snooping, look-ahead, costs and code errors, and rejected ideas stay on record so they cannot return under a new name. Most proposed changes do not pass, and a pass makes only a candidate: a person decides. Bailey & López de Prado: The Deflated Sharpe Ratio (PDF)

Are the signals on GSDAQ tested?

Yes, continuously and in real time. The strategy lists in GSDAQ are produced by the same selection code our automated trading uses, and each list is also run forward as a tracked book every trading day. Growth-trend rules like those behind the Growth Trend list have traded automatically since March 2026. The Long and Short chart marks are drawn by the same code as our in-house trading dashboard, whose signals feed one of our automated strategies. Tested is not the same as passed. A signal on your screen is what a rule found in past prices and data, not an order, a recommendation or a guarantee. In a stock’s detail view, some indicators carry an “observational” label noting that they include items that did not pass pre-registration. Research tests decide which rules change, and we publish no strategy returns or win rates. GSDAQ: How GSDAQ signals work

How can you check any strategy claim the same way?

Ask the same questions of any backtest you see, including ours. The table is a checklist, not a score: a claim that cannot answer these questions has not yet shown more than a fit to the past. Historical and paper results do not establish future results, and nothing here recommends buying or selling any security.

On narrow screens, scroll the table sideways. Keyboard users can focus the table and use the arrow keys.

Questions to ask of a backtest
Question	Why it matters
Was the pass line written before the test?	Otherwise the goalpost can move after the result
Does the universe include companies later delisted?	A survivors-only list flatters the past
When is the trade assumed, and at what cost?	Same-close fills without costs are not achievable
How many variations were tried?	Many tries make a lucky fit likely
Was an untouched period used only once?	Repeated peeking turns it into training data
Is there a forward record since the rule was fixed?	Only results after the freeze are free of hindsight

Sources

Open GSDAQ ↗