Basis: This is not a measurement but a reference text. Every number in it comes from a linked study on this site, from a documented client case (anonymised), or is marked as an internal, not yet published measurement. The error classes are not specific to AI. We have built every one of them ourselves, with years of market experience and a review workflow. The difference is the speed at which a model builds them, and that nobody is watching while it does. No trading recommendation.
The scene has become routine: a rule in your head, a prompt to ChatGPT or Claude, and three minutes later an expert advisor for MetaTrader 5 or a Pine strategy for TradingView. The code compiles. The backtest is green. What is missing is nothing you could see in the code. It is what the market knows and the model does not: when a piece of information exists, what a fill costs, how a broker counts, when a session really begins.
This text sorts the resulting mistakes into three groups: the backtest lies, the code is right but the market is different, the idea does not hold. It ends with the checklist that finds them. It is the same list that runs here before every client project.
1. What the model optimises for
A language model knows syntax, structure and a platform's documentation. It writes boilerplate faster than any human, knows the signatures of OrderSend and strategy.entry, and makes fewer typos than we do. That is real value, and we use it too.
But it optimises for something other than the market: the satisfaction of the person in the chat. "Make the backtest profitable" is an instruction a model can satisfy in three ways, and all three look like success in the chat. It can build in information that did not exist at entry time. It can turn parameters until the curve fits. It can build the simulator so that fills appear that would never happen for real. None of these is fraud. They are the three natural shortcuts anyone takes who builds a backtest without a counter-check, and a model takes them without the unease an experienced developer feels at that moment.
Market experience, at this point, does not mean predicting the market better. It means knowing the list of things that appear in no API documentation: that the 30-minute opening range does not yet exist at 09:05, that the DAX spread at 07:38 is a different one from 10:00, that two brokers quote the same index on different point scales, that Europe and the US change their clocks on different weekends. Every item below is one entry on that list.
2. The backtest lies
2.1 Information from the future
Lookahead is the mistake everyone warns about and everyone builds anyway. Never crudely, never with tomorrow's close. But with a quantity that is defined for the day and read at a point where the day has not got that far yet.
It happened to us with four context features, despite a review routine: the boundaries of the 30-minute opening range were read for entries inside the first 30 minutes, a directional tendency relative to the cash open for entries before the cash open. 60,129 of 123,193 episodes were affected. The forecast spread of a model trained on them fell from +0.726 R to +0.191 R after the fix. The full diagnosis is in the post-mortem.
In generated code it looks like this: iHigh(_Symbol, PERIOD_D1, 0) as "the day's high" at 10 a.m. An opening range used as a filter from the first minute. In Pine, a request.security call to the higher timeframe without [1] that reads the running daily bar. And the whole family of centred indicators: TMA bands, ZigZag, centred averages, anything that averages a candle's value from the candles after it.
A client case from September 2026: a recognisably AI-generated skeleton calling TMA bands with HalfLength 32 on H4 via iCustom. The chart showed three arrows in seven weeks, all neatly at the band crossings. Except that the bands are redrawn for past candles as new ones arrive; the arrows are hindsight. The non-repainting variant at that setting lags by more than five trading days. The client would have received an EA that never reproduces his chart, and the complaint would have arrived after the first live comparison.
Check: For every input, write down the moment from which it is known and compare it with the entry time. Diagnostic signature: the "edge" concentrates in the trades where the feature was still unknown.
2.2 The trigger comes late, the entry stays early
The second class is subtler. A filter waits for a confirmation, volume for instance. The condition only becomes true later, but the entry price stays, arithmetically, at the original level. The backtest then buys at a price that no longer existed at the moment of the decision.
That is exactly what a setup in the US midday lull handed us: +0.90 R at a 60% hit rate with a volume gate, the best setup in the whole system. Without the gate: −0.11 R at 29%. The gate had shifted the trigger by a median of 100 minutes while the entry stayed at the old level. The entire edge was lookahead in disguise. The study: Phantom edges from a volume gate.
Check: Log the gap between "level reached" and "condition true" (a median above 10 minutes is an alarm), and the gap between fill price and level in R (a median above zero means a fictional entry).
2.3 Fills that never happened
A bar-based backtester has to guess what happened inside a bar. If stop and target both lie inside the same bar: which was touched first? Is a limit filled at the level or at the next bar's open? A generated simulator decides these in the strategy's favour when in doubt, not by intent but because the optimistic variant is the simpler one.
Our evidence: a doji pattern on the 2-minute chart showed +0.07 R net on real data with a t-value of +16. A synthetic random walk sent through the same bracket chain produced +0.16 R gross for the same pattern. The edge was a property of the chain, not of the market. Documented in The doji trap on the 2-minute chart.
Check: Run the same engine on a random path. Whatever it "finds" there is the engine. That is the random-walk anchor, and it is mandatory for minute-scale brackets. And where the order of events inside one minute decides the result, guessing does not help, only the tick series does: in our DAX evaluation of the opening candle, both sides were broken within the same minute on 250 days. 223 of them could be resolved with tick data at millisecond level; the remaining 27 stayed undecided even at tick level. Minute data is the base, ticks are the tool for exactly these cases.
2.4 Costs that are missing or flat
Generated code runs without a spread, or with a flat percentage from a stock-market tutorial. Both are worthless for intraday trades. The number that decides before the backtest is the spread relative to the stop distance, because every trade pays that toll up front.
Testing an index expansion, a spread of 0.80 points looked small. Relative to the setup's stop it was about a quarter of the risk per trade, and the market was dead before the edge question was even asked. The index with the smallest absolute spread on the list was the most expensive market. Everything from the Euro Stoxx 50 upwards sat at seven to nine times the DAX toll. The arithmetic: SMI, CAC and CA60 under test.
Check: State costs per trade as a share of R, and every metric net. What only works gross does not work.
2.5 A hit rate without a yardstick
The model reports "67% hit rate" as success. With a stop twice as far away as the target, 67% is exactly what a coin flip delivers: stop divided by (stop plus target). The formula is over 300 years old, and 35,000 coin-flip trades per market on DAX, NQ and SPX confirm it within one or two percentage points. The derivation in plain language: The random-walk yardstick.
Check: Hit rate minus the chance hit rate, trade by trade, because every trade has its own stop-to-target ratio.
2.6 Two years are one regime, not proof
The standard backtest from the chat runs over whatever data is free right now, usually one to two years. Six breakout setups over eleven years show why that is not enough: 2015 to 2017 produced +15 R at a drawdown of −449 R, the eight years after that +621 R per year at −136 R drawdown. Same rules, two distributions. Anyone who has seen only one of them does not know what a bad month is. The study: A breakout book over eleven years.
And even eleven years do not protect against decay: in our deployment study the backtest stayed green while the real account lost from 2025 onwards. Phase 5 is the most honest chapter of the series for exactly that reason.
Check: Every regime the data offers. Walk-forward instead of a single optimisation. And an assumption about how fast the edge decays, before money is attached.
3. The code is right, the market is different
No backtest finds the mistakes in this section, because the backtest shares them. They only show up on the account.
3.1 Session hours and daylight saving
if(Hour() == 15) is the most common line in generated session EAs. It is correct as long as server time and exchange time keep the same distance. But Europe and the US change their clocks on different weekends. Two or three times a year the US open shifts by an hour relative to server time, for two to three weeks. An EA that does not know this quietly trades a different strategy in those weeks. Add the small things no model derives from the prompt: Xetra closes at 17:30, the CFD runs until 22:00; US half days; holidays where the price stands still but the spread does not.
Check: Derive session boundaries from the exchange time zone, never from the server hour. And look at the week of the DST offset in the tester, once.
3.2 Points, pips, tick value, symbol
At one broker an index point is ten broker points, at the next a hundred. The same EA with the same inputs is therefore off by a factor of ten on the second account: the spread filter filters nothing, the trigger demands two instead of 0.2 index points. We worked through this ourselves when moving between two brokers; every point-based input had to be divided by ten. Add the tick value in euros or dollars, and symbol names that differ between GER40, GER40.cash and DE40. Generated code takes "100 points = 10 pips" from a forex tutorial and applies it to indices.
Check: Log SYMBOL_POINT, SYMBOL_TRADE_TICK_VALUE and the account currency at start-up, think of every distance in index points or in R, never in broker points.
3.3 State instead of event
Back to the client case from 2.1. Besides the repaint, the skeleton had four more mistakes in perhaps 150 lines. The buy condition checked whether the green band is below the white one, not whether it crossed below. As long as the band stayed below, the EA would have opened the next trade right after every closed one; on the chart it was one arrow. The bar-close gate only became active after the first trade; before that the code evaluated every tick. The same indicator name received four parameters in one call and five in another; iCustom passes them by position, so at least one call returned nonsense. And the H1 bands on the H4 chart were read with shift=1, the last H1 candle instead of the H4 candle.
None of these mistakes is exotic. They are the standard mistakes every beginner builds once. The model builds all four at once and adds a comment block reading "Copyright 2026".
Check: Phrase every entry condition as an event (false before, true now), observe the gate before the first trade, hold every external indicator call against its input list.
3.4 Gross and net
A client EA was supposed to close all positions once the cycle profit reached a target. It closed too early: the sum counted only DEAL_PROFIT, without swap and commission. Gross, the target was met; net, it was not. After the fix the client found the same trap one level deeper, in the recovery logic, because we had fixed the bug as a location instead of as a class. DEAL_PROFIT is the field the model picks because it is the obvious one.
Check: At every place where money is summed, decide deliberately whether swap and commission belong in. And search the whole code for every bug found, as a class.
3.5 The four life cycles
Terminal restart, chart closed, EA removed, parameters changed: in each of these four cases the state in memory dies. What stays, what resets? Generated code keeps counters in static variables and is a different EA after the first restart. Add a case from the MQL5 forum in September 2026: an EA ran correctly in the tester and wrongly on the prop-firm account, because that server rewrites or empties deal comments and the EA had identified its trades by the comment.
Check: Trigger all four life cycles once and observe the tick after each. Identify trades by magic number and ticket, never by comment.
4. The idea does not hold, and the code cannot know
The third group has nothing to do with programming. The model builds what the prompt demands, and if the prompt demands "eleven indicators, arrow only on full consensus", it builds that cleanly. We measured such a consensus algorithm (internal measurement of 18 September 2026, not yet published): the eleven votes from the same close series are correlated at 0.41 on average, effectively 3.3 independent votes. The consensus arrives a median of 16 candles after the first flip, and by then price has already run 2.7 ATR in the signal direction. Net, the result was negative in all 15 market-timeframe cells tested.
The second trap in this group is selection. Whoever lets the model build twenty variants and takes the best one has, with high probability, selected noise. With twenty attempts a t-value above 2 is expected with nothing behind it. The yardstick is then not "t against zero" but "how strong is the best of twenty random runs".
Check: Before building, ask what information the rule contains that is not already in the price. And count the number of variants tried.
5. The checklist
Twelve questions, each answered yes or no before money hangs on a rule. It is the same list that runs here before every client project.
| # | Question | Where it goes wrong |
|---|---|---|
| 1 | Is it written down, for every input, from when it is known? | 2.1 |
| 2 | Does an indicator repaint (centred, HalfLength, ZigZag, running HTF bar)? | 2.1 |
| 3 | Is the median gap between level and trigger under 10 minutes, and the fill at the level? | 2.2 |
| 4 | Does the engine find an edge on a random walk? | 2.3 |
| 5 | Are costs stated as a share of R per trade, and is every metric net? | 2.4 |
| 6 | Is the hit rate measured against stop / (stop + target)? | 2.5 |
| 7 | Does the test cover several regimes, with walk-forward instead of one optimisation? | 2.6 |
| 8 | Do session boundaries come from the exchange time zone, not the server hour? | 3.1 |
| 9 | Are point, tick value and account currency logged, all distances in index points or R? | 3.2 |
| 10 | Is every entry condition an event, and was every gate observed before the first trade? | 3.3 |
| 11 | Is it decided at every money sum whether swap and commission belong in? | 3.4 |
| 12 | Were restart, chart close, EA removal and parameter change each triggered once? | 3.5 |
Anyone who can answer all twelve with yes has a system that deserves the backtest it shows. That is not yet an edge, but it is the prerequisite for measuring one at all.
6. The code is the cheap part
Nothing in this list argues against language models. They do in minutes the part that used to cost days, and that is a gain. The expensive part was never the code. It was the list above, and that list comes from backtests that went wrong, from accounts that reacted differently from the tester, and from brokers that count differently from what the prompt assumes.
Anyone building on their own has the most important part in hand with the list. Anyone who wants to skip that round finds validation as a separate step before development under Services, with a verdict that can also read "no edge". The cheapest result is a no before development begins, and a no takes less than a week.
Caveat
This text measures nothing new. The numbers come from the linked studies, which carry their own caveats, and from two client cases, which are an anecdote and not a sample. The measurement on indicator consensus is internal and not yet published. Language models change faster than this text; which of these mistakes a model still makes in a year, we do not know. What remains are the error classes themselves, because they are properties of backtests and brokers, not of models.
Disclaimer: Historical statistics are no guarantee of future market behaviour. This analysis is not investment advice. Trading carries a risk of loss up to and including total loss.