How we test, and what a test cannot tell you.
A backtest is a model of the past. A forward test is a short sample of the present. We publish both, in separate tables, with the methods and the limits next to the numbers rather than under them.
Methodology
Every number in the backtest table comes from this set-up. If any of it changes, the table is rerun and the change goes in the changelog.
| Item | How it is modelled |
|---|---|
| Price data | Tick data supplied by the partner broker's own servers, with gaps checked and logged |
| Period | xxxx – xxxx |
| Spread | Variable spread replayed from the tick data, plus x.x points added on every entry |
| Slippage | x.x points against us on every market order and stop; x.x on stops during listed releases |
| Commission | $x.xx per lot per side, as charged on the partner broker's raw account |
| Swap | The broker's published swap rates for each period, charged at rollover, triple on the broker's triple-swap day |
| Data split | xx% in-sample for development, xx% out-of-sample kept back and run once |
The easiest way to build a beautiful backtest is to fit it to the past.
With enough inputs and enough tries, any EA can be tuned until its history looks smooth. That history then describes the noise it was tuned on, not the market it will meet. These are the checks we run against it.
Few inputs that matter
Each EA has a handful of inputs that change its decisions. Every extra input is another way to fit noise, so new ones need a reason written in the changelog.
Out-of-sample data, run once
Part of the history is locked away during development. The finished EA is run on it once. If that run is poor, the EA does not ship; it is not tuned and run again.
Plateaus, not peaks
We vary each input around its default. If results collapse when a value moves slightly, the default sits on a peak of luck and is rejected. We choose values where neighbours behave alike.
Shuffled order of trades
The same trades in a different order give a different drawdown. We resample the order many times and quote the worse end of the range, not the drawdown history happened to produce.
Costs pushed against us
Spread and slippage are set worse than typical. An EA that only works with perfect fills does not work.
Hypothetical, out-of-sample
Default settings, RiskPerTrade at its default, costs as modelled above. Drawdown is printed before return.
| EA | Period | Max drawdown | Net result | Trades | Sessions sat out | Profit factor | Cost per trade |
|---|---|---|---|---|---|---|---|
| Lull | Out-of-sample, xxxx – xxxx | −x.x% | xx.x% | xxx | xx% | x.xx | $x.xx |
| Hinge | Out-of-sample, xxxx – xxxx | −x.x% | xx.x% | xxx | xx% | x.xx | $x.xx |
| Ledge | Out-of-sample, xxxx – xxxx | −x.x% | xx.x% | xxx | xx% | x.xx | $x.xx |
Backtest results are hypothetical. They come from running the EA on historical prices, with costs modelled rather than paid, and with the benefit of hindsight in how the rules were chosen. No backtest accounts for every effect of real trading, such as liquidity at the moment of an order or a broker's execution on a given day.
All figures on this showcase are shown as x.
What the backtest cannot model
Every one of these has, at some point, made a live result worse than its backtest. They are the reason the forward test exists.
- Liquidity at the moment of the order. Tick data shows a price, not how much could be traded at it. Around releases, a stop can fill well beyond its level.
- Gaps. Weekend and holiday gaps open past a stop. The backtest fills at the gap price; a live account can do worse.
- Spread that is not in the data. Tick data records the spread at each tick, but not every broker's widening at rollover is captured the same way.
- Your broker is not our broker. Different servers, prices, execution and symbol specifications produce different trades from the same EA.
- Terminal and connection. A backtest never disconnects, restarts or misses a bar. A real terminal can.
- Hindsight. We designed the rules knowing what the past looked like. However careful the method, that knowledge leaks in.
- Regime change. Markets change how they behave. A rule that worked for years can stop working without warning.
A separate record, on a live account
Since xx.xx.xxxx. Live account at Ferncastle Markets, default settings, RiskPerTrade x.x%, VPS in the broker's data centre. Never restarted to reset a drawdown, never trimmed, and never merged with backtest figures.
| EA | Period | Max drawdown | Net result | Trades | Sessions sat out | Profit factor | Cost per trade |
|---|---|---|---|---|---|---|---|
| Lull | xx.xx.xxxx – today | −x.x% | xx.x% | xxx | xx% | x.xx | $x.xx |
| Hinge | xx.xx.xxxx – today | −x.x% | xx.x% | xxx | xx% | x.xx | $x.xx |
| Ledge | xx.xx.xxxx – today | −x.x% | xx.x% | xxx | xx% | x.xx | $x.xx |
Forward-test results are past performance on one live account at one broker. Your results will differ with your broker, account type, terminal, settings and start date. Past performance is not a reliable indicator of future results.
All figures on this showcase are shown as x.
On a live site the forward-test account's statement is published here as the broker issues it, so the trades can be checked against the table. Ferncastle Markets is a fictional broker, so the showcase publishes none.
We do not show equity-curve screenshots or third-party badges as proof. A picture of a curve proves nothing about how it was made.
Where the two disagree, we believe the forward test.
When a forward-test figure drifts away from the backtest, it is usually costs: wider spreads and worse fills than the model assumed. When that happens we change the model, not the forward test, and we often change the EA so it trades less.
The Lull spread-cap change is an example: trades near the old cap lost more often live than in the backtest, so the default cap came down and Lull now sits out more nights. See the release
Read the automation risks