FinnyBench: Measuring What Actually Breaks Trading Agents
FinnyBench scores coding agents on the failures that matter in trading research: hallucinated numbers, lookahead bias, sloppy arithmetic, and unverifiable memos. Here is how Finny, Claude Code, Codex, and OpenCode compare.
By Jaimin
Every coding agent demos well. You ask for a strategy, you get clean Python, and the terminal fills with confident output. The problem is that trading research does not fail at the code-generation step. It fails quietly, three steps later, when a backtest silently peeked at tomorrow's close, or a memo cited a Sharpe ratio that no dataset ever produced.
General coding benchmarks do not catch this. Passing a unit test says nothing about whether a strategy is decision-safe. So we built FinnyBench: a task suite that scores agents on the specific failure modes that make quantitative work untrustworthy.
How the benchmark is set up
The comparison is deliberately narrow so the result means something. The underlying model is held constant across all four harnesses, so the scores measure the harness — its context handling, its validation, its guardrails — rather than the raw model.
- Four harnesses: Finny, Claude Code, Codex, and OpenCode.
- Same model everywhere, so harness design is the only moving variable.
- n=3 scored trials per cell, which is why several scores land on thirds like
51.7and81.7. - 20 verifier checks per task, each a programmatic assertion rather than a subjective rating.
- Scores run 0–100 and represent the mean share of verifier checks passed.
The five tasks
Each task isolates one way trading work goes wrong in practice.
Hallucination resistance
Does the agent invent tickers, fields, prices, or API surfaces that do not exist? This is the task where a confident wrong answer is far worse than an admitted gap.
Numeric discipline
Returns, position sizing, percentage-versus-basis-point handling, and compounding. Arithmetic that looks fine and quietly misstates performance.
Repair lookahead
Given a strategy that peeks at future data, can the agent find the leak and fix it without breaking the strategy's intent? Lookahead bias is the single most common reason a backtest looks brilliant and a live deployment does not.
Research memo
Can the agent turn a run into a memo a portfolio manager could act on, with claims that trace back to actual evidence rather than narrative?
BTC strategy
An end-to-end build: take a crypto strategy idea from description to working, testable implementation.
Results
| Task | Finny | Claude Code | Codex | OpenCode |
|---|---|---|---|---|
| Hallucination resistance | 25 | 25 | 28.3 | 21.2 |
| Numeric discipline | 30 | 45 | 30 | 45 |
| Repair lookahead | 70 | 51.7 | 45 | 35 |
| Research memo | 60 | 55 | 62 | 55 |
| BTC strategy | 81.7 | 55 | 55 | 36.7 |
| Mean across tasks | 53.3 | 46.3 | 44.1 | 38.6 |
What the numbers say
Finny leads on the mean across all five tasks at 53.3, ahead of Claude Code at 46.3, Codex at 44.1, and OpenCode at 38.6. But the average is the least interesting part of this chart. The shape of the wins matters far more than the ranking.
Where Finny wins, it wins on structure
The two largest margins are the two tasks that require holding a whole workflow in view. On repair lookahead, Finny scores 70 against a next best of 51.7 — an 18.3 point gap. On BTC strategy, Finny scores 81.7 against 55 for both Claude Code and Codex, a 26.7 point gap and the widest margin in the suite.
Both are multi-step tasks where the agent has to keep the thesis, the data, and the validation connected. That is exactly what the Finny harness is built to do, and it is where a general-purpose coding agent has the least context to work with.
Where Finny does not win
Finny loses two of the five tasks, and the losses are worth stating plainly. On numeric discipline, Finny and Codex both score 30 while Claude Code and OpenCode both reach 45 — a 15 point deficit. On research memo, Codex edges ahead 62 to 60, which is inside the noise of a three-trial sample and should be read as a tie.
Hallucination resistance is effectively a four-way loss. Codex takes it at 28.3, Finny and Claude Code tie at 25, OpenCode trails at 21.2. Every harness fails roughly three out of four checks.
The most important finding is the low ceiling
The headline result of FinnyBench is not that one agent is ahead. It is that every agent is bad at the two tasks that matter most for trust. Hallucination resistance tops out at 28.3 and numeric discipline tops out at 45. No harness in this comparison — including ours — is close to reliable on either.
That is the argument for keeping a human in the loop and keeping generated code inspectable. An agent that builds a strategy well and still miscounts a percentage is useful, but only if you can see what it did.
Why we published the losses
A benchmark that only shows favourable results is marketing, not measurement. FinnyBench exists because we needed to know where our own harness breaks, and two of these five tasks tell us exactly that. Numeric discipline is now a concrete target rather than a vague worry.
The same standard applies to the wins. Finny's repair-lookahead and BTC-strategy margins hold up because they were scored by the same verifiers, against the same model, as everyone else's.