Backtesting Trading Systems for CMT Candidates: Specify, Simulate, Challenge

What a backtest can and cannot answer

A backtest applies a fixed decision rule to historical data. It can reveal how that rule would have behaved under explicit assumptions. It cannot prove future profitability. The 2026 CMT Program Guide includes scientific-method steps, signal tests, survivorship bias, trigger and filter rules at Level I; Level II adds system evaluation, performance measures, stops, and robustness; Level III includes statistics of backtesting. The public objectives make this a curriculum topic, but no third-party strategy or parameter is an exam requirement.

Before calculating a return, write the rule so another analyst could reproduce it. A statement such as “buy strong trends and use a sensible stop” is not testable. “At the next session open after a daily close above the prior 20-session closing high, buy up to a fixed risk amount; exit at the next open after a close below the prior 10-session closing low” is closer. It still needs universe, adjustment policy, position size, transaction costs, and treatment of gaps.

Signal test before full strategy

A signal test asks whether a defined event has informative outcomes, before adding many portfolio and money-management choices. For example, record the next 20-session return after every close above a 20-session high, including overlapping events under a declared rule. Compare with a benchmark or matched non-event periods. This isolates the idea more clearly than a complex strategy with multiple optimized exits.

A full backtest then simulates orders, capital, cash, overlapping positions, and costs. If the signal test is weak, a profitable backtest may owe its result to sizing or a favorable sample rather than the supposed entry. Signal and strategy evaluation answer different questions; both need a holdout period.

Worked trade economics

Suppose a rule buys 100 shares at 50, sells at 55, pays 0.10 per share in combined commissions and spread, and incurs 0.05 per share slippage. Gross gain is (55 − 50) × 100 = 500. Round-trip per-share friction is 0.15, so total estimated friction is 15 and net gain is 485. If the 0.10 charge was actually per side rather than round trip, this calculation would understate costs. The test specification must say how each cost is applied.

For a loss, imagine entry 50 and exit 47. Gross loss is (47 − 50) × 100 = −300; after the same 15 friction, net loss is −315. If a 20-trade sample has 8 wins averaging 485 net and 12 losses averaging −315 net, total net = 8×485 + 12×(−315) = 3,880 − 3,780 = 100. Average net per trade is 100/20 = 5. The win rate is 8/20 = 40%. That small positive average can easily be erased by modest cost-estimate error or a different market period; a 40% win rate alone is neither good nor bad.

A risk-based position example: with 50,000 equity and 1% intended capital risk, budget is 500. If entry is 50 and planned exit 46, price risk is 4 per share. Maximum whole shares under a simple calculation are floor(500/4) = 125, before gaps and correlated holdings. This is a sizing rule, not a loss guarantee. A stop may execute below 46.

Biases that manufacture attractive history

Look-ahead bias: using tomorrow's close to decide today's opening trade, or using a centered moving average that contains future bars. Align timestamps so every decision uses only data available then.

Survivorship bias: backtesting only today's index constituents omits securities that failed, delisted, or left the index. The historical universe should reflect what was eligible at each date.

Selection and overfitting: testing many lookbacks and reporting only the best creates a result partly selected for luck. Freeze a small candidate set, use a development period, and validate on later untouched data. Parameter stability matters: if 19 days loses badly while 20 days wins spectacularly, investigate.

Execution bias: assuming fills at a price no trader could obtain, ignoring spread, or treating a limit order as filled merely because the day's range touched it. State fill priority and gaps. Volume constraints matter if the position is large relative to daily trading.

Data problems: split-adjusted versus raw prices, stale quotes, survivorship of ETFs, futures contract rolls, and missing bars can create false trades. Preserve raw data and adjustment logic.

Metrics that answer different questions

CAGR summarizes compounded annual growth; maximum drawdown is the largest peak-to-trough decline in the tested equity path. If equity rises to 120, falls to 90, then recovers, drawdown from that peak to trough is (90/120 − 1) × 100 = −25%. Profit factor is gross profits divided by absolute gross losses; if profits are 3,880 and losses 3,780, it is about 1.026. A near-one factor is fragile after costs.

Sharpe relates excess return to total return volatility; Sortino emphasizes downside variation; Calmar relates annualized return to maximum drawdown. None should be reported without period, return frequency, risk-free convention where relevant, and an out-of-sample check. A high ratio from a few smooth observations can be misleading.

A decision and invalidation protocol

Predeclare a go/no-go standard. For example, advance a signal only if it remains net positive in a later untouched period, does not depend on one security, and survives a reasonable range of cost and parameter assumptions. If it fails any one, return to research rather than silently weakening the standard. In live monitoring, define a separate failure rule such as sustained divergence from expected risk or drawdown beyond a pre-set tolerance. The standards should be chosen for the strategy's context, not copied from an article as universal thresholds.

Common mistakes are changing the rules after seeing a drawdown, treating in-sample optimization as validation, and confusing a high backtest return with a well-controlled decision process. See Statistics for CMT Candidates for the sampling concepts behind these cautions.

Self-check

A strategy has four net trades: +120, −80, +60, and −70. What are net total, win rate, and profit factor? Answer: Net is 120 − 80 + 60 − 70 = 30. Two of four trades win, so win rate is 50%. Gross profit is 180, gross loss magnitude 150, so profit factor is 180/150 = 1.2. Four trades are far too few to infer durability.

Next, write one complete entry and exit rule with exact timestamps and cost assumptions. Hand-simulate five consecutive historical signals before using software; this often reveals ambiguous rules immediately.