Verifying trading scripts and claimed backtests — mine included. Author, Trading as a System. Free: the 10 checks I run before I believe a backtest ↓
Joined February 2021
- Tweets349
- Following2.5K
- Followers750
- Likes7.3K
Pinned Tweet
Most backtests fail on arithmetic, not strategy.
60% wins over 25 trades: the 95% interval is roughly 41–77%. A profit factor of 2.0 at 35% wins needs winners 3.7× the size of losers.
The 10 checks I run before I believe a backtest, free on one page:
quanticator.app/go/checklist…
1
140
Before testing another entry rule, test the holding period. That check closed a research programme of mine in an afternoon.
Required hit rate = 0.5 + cost / (2 x the average absolute move over your hold). BTC, ETH and SOL, two venues: below four hours at 60 bps it exceeds 100%.
ALT Card: A hit rate above 100% is a closed question. Required hit rate equals 0.5 plus cost divided by two times the mean move over the hold. Below 4 hours at a 60 bps round trip it is over 100%, so no entry rule clears it. BTC, ETH and SOL across two venues; an arithmetic bound, not a fitted result.
24
My own published backtest states its costs as 0.05% and two ticks. The two ticks is the half that means nothing alone.
Two ticks a side on a ten-tick stop is 40% of your risk per round trip. On a hundred-tick stop, 4%.
A tick count is not a cost until you know the stop.
1
42
The fee you were quoted is half the fee you pay.
A tenth of a percent a side is two tenths a round trip. On a strategy averaging 0.30% gross a trade that leaves 0.10%, not the 0.20% the one-sided number suggests.
Double the fee before you subtract it.
45
A 50% drawdown in a long/short ratio doesn't map to one loss. On $1 long and $1 short held unrebalanced: longs -50% with shorts flat costs $0.50; longs flat with shorts +100% costs $1.00. Same ratio, twice the loss. Check which leg moved. teapawt.net/eliant_capital/status/…
Goldman Sachs’ High Beta Momentum Pair has now suffered a staggering 50% drawdown, the worst in the 15-year history of the dataset, surpassing even the '21–'22 collapse
51
One question ran through this week: how much does a trade count actually buy you?
Less than it looks. It sets how wide the interval is, and nothing else. Not where in the sample the profit sat, not whether the trades were independent, not whether the costs were real.
ALT Card: What the trade-count week came down to. 1, the count sets the interval, nothing more. 2, ask where in the sample the profit sat. 3, count independent events, not fills. Trade count is check 1 of ten.
1
60
Trade count is check 1 of the ten I run before I believe a backtest. The whole page, free: quanticator.app/go/checklist…
43
180 trades over five years sounds like a sample. Then count the chances it had.
Five years of US sessions is about 1,260. A strategy that could trade any of them took 180: one in seven. Whatever picks that one is the biggest parameter in it, and it never appears.
ALT Card: Count the chances, not just the trades. Five years of US sessions is about 1,260 (252 a year times 5). 180 trades taken. That is one session in seven, and nothing in the report says which. Worked example, not a result.
40
Ship log: I killed my Alpaca paper crypto bot on 30 Sep, then wrote the rule that allowed it on 26 Oct.
Frozen config, replayed on 145 unseen days (7 May to 29 Sep 2026): 93 trades, gross +4.7 ± 8.3 bps a trade, net -55 against a 60 bps round trip.
Evidence first, rule second.
🤖 Made with AI
1
68
Four hundred trades is not four hundred times the evidence of one. It is twenty times.
That ratio decides how many of the backtests you are shown mean anything, and almost nobody does the arithmetic. Seven posts.
ALT Card: Same 58% win rate, three sample sizes. 50 trades: 44 to 71%. 100 trades: 48 to 67%. 400 trades: 53 to 63%. Wilson 95% score intervals on 29/50, 58/100 and 232/400. Width falls with the square root of n.
Trade count is not sample size. Twenty fills taken over one weekend in two instruments that move together are closer to one observation than twenty.
Count independent events. I have made exactly this mistake on my own ledger.
1
20
Nine winners out of ten. The textbook interval puts the win rate between 71% and 109%.
A hundred and nine percent. That is the normal approximation falling over at small n, and it is the formula most calculators use.
Wilson gives 60 to 98: still wide, and possible.
🤖 Made with AI
40
A big trade count is good evidence right up until you try to audit it.
TradingView's docs say it plainly: on the default testing range, a strategy preserves individual data for up to the latest 9000 trades. Past that, the list you'd check is a tail.
Large n, small audit.
ALT Quote from the TradingView Pine Script docs, Strategies, Trades tab: 'If a strategy uses the default testing range, it preserves individual data for up to the latest 9000 trades.' Note: Deep Backtesting keeps every trade; on the default range, past 9000 the list you can audit is the latest tail.
32
I'd want the trade count next to any PF. A true PF of 1.5 (half the trades win 1.5R, half lose 1R) prints anywhere from 0.84 to 2.67 over 50 trades, and below 1.0 about 6% of the time. Over 500 trades: 1.26 to 1.79. teapawt.net/PeterLBrandt/status/21…
Unlike all the braggarts on X touting their "moon-shot" success, I do not release company ROR data - but, by example, highly encourage aspiring traders to be data rich on the much more meaningful measures such as PF, Calmar, EV, DDs and other things.
I have released data in the past - a long time ago, because my trading performance from 1981 through 2010 had been audited by a CPA/auditing firm using month end brokerage statements, a CFTC guidelined performance table and IRS tax statements.
I don't release this anymore because it is old and I am against releasing un-audited ROR data.
We do our trading through ADMIS, IBKR and plan to test-drive Stone soon.
The company enters trades in units sized to multiples of $1M - so perhaps, six Bean Oil contracts per $1M notional.
The notional is all backed by cash and T-Bills, but we only keep about 20% of notional value on deposit with our brokers.
Seldom is the margin-to-equity ratio of our open trades more than 10-15% of total notional.
We are angled to make about 150 or so trades per year across all the globe's liquid commodity futures exchanges from Chicago to NY to London to Zurich to Hong Kong to Paris to Malaysia.
We make all of our decisions based on monthly and weekly price charts. No fundamentals. No indicators. Simple charts. We trade about 5 to 10 different set ups that we have traded hundreds of times each over the 50 years.
I gave a speech to the MTA (now CMT certification) in 1991. I recently came across a copy and our trading style has changed almost none since 1981. Whereas correct interpretation was the payoff for us then, sizing, risk and trade management and engineered processes are now where our edge lies.
I believe many of you young chartists way way way over-complicate your charting. Somehow many of you think that adding "just one more" indicator or tool to your efforts will offer the magic sauce.
The magic sauce is SIMPLICITY.
You've seen the charts I post. Any more than one marked line is too often too many to be worthwhile.
A final comment to you if you are a day trader. Good luck. The data is solid - no more than 3 in 1,000 of you who seek to make trading your livelihood will succeed. Disbelieve me at your own risk (and years wasted trying)
Only you have the answer for you. Believing this last sentence and not wasting time time, money and energy looking for shortcuts will shorten your search.
That's it for today
27
My paper equity record read +$91.88 over six closed trades. Then I read the logs and found five more nobody had written down. All five lost.
IBIT and FBTC, 9/14 to 10/6, Alpaca paper, bot-recorded prices before fees: +$45.16 over eleven.
A trade count is a reading, not a fact.
🤖 Made with AI
1
50