In my last post, The best LLM benchmark is the one you write yourself, I shared tips for building your benchmark. I kept two things back: the model names, and the benchmark itself. Today you get both.
Meet finrec-bm. It hands AI agents a common job in finance, reconciliation, and checks whether they get the books right: right to the cent, with the paper trail to prove it. In order to do that, the agents need to navigate emails, spreadsheets, a transactional database and screenshots.
Financial reconciliation?
Every company keeps its own record of money in and out: the ledger. Everyone involved in any transaction keeps their own version too: the bank, your suppliers, your customers, the colleague with the spreadsheet.
Reconciliation means lining those versions up, matching every item, and explaining every difference. Mismatches are common.
Sounds like adding numbers. It isn’t, and it often requires a good dose of investigation. Some examples:
One payment settles three invoices.
The bank keeps a fee.
A euro invoice gets paid in dollars.
A June sale lands in July.
Somebody types 8,420 instead of 8,240.
Adding up numbers is the easy part. Deciding which numbers count is the job.
Internal teams fix this every month, under deadline, across emails, CSV exports, Excel files and screenshots.
That makes it a great test for AI agents: lots of reading, a bit of arithmetic, and zero tolerance for “close enough”.
What the benchmark tests
Each case reads like a message from a colleague: “Internal audit wants us to confirm this net cash flow. The email and the spreadsheets are in the folder.”.
The model gets tools and decides where to look:
query_ledger: the ledger lives in a SQL database the model has never seen. It explores the tables and writes its own SQL on the fly, up to ten queries per call. Across all runs, the models wrote 4,723 of them.read_msg_fileextracts what an Outlook.msgemail holds: sender, recipients, dates, subject, body, HTML body and attachments. 10 of the 22 cases include at least one of these binary files;more file readers open plain text, CSVs and Excel
.xlsworkbooks, with their cells, formulas, fill colours and embedded pictures;read_image_fileshows it screenshots;calculatedoes exact decimal arithmetic, because floating point and money don’t mix.
It gets 30 turns to return a structured answer: a verdict, the value the evidence supports, and every record, cell and row behind it.
Four test families check the analysis capabilities of the models:
No LLM judges the answers. A rule-based scorer compares the cited evidence with the expected list. A case passes only on an exact match: every expected item, source, the right verdict and the right value to the cent.
Your turn: spot the trap
Here’s a real case from the Excel family, and one of the hardest in the set. Three .msg emails land in your inbox, in this order:
Here’s the workbook the emails talk about:
The freight charges live in the ledger. The model has to write SQL queries to find them: the customer sits in sales_orders and the booking entity in entities, so it takes two joins.
Does EUR 10,760.80 hold up? Which cells and which charges make up the right number? A human reviewer would need some time to answer this questions.
Some of the LLMs were able to provide the answer (more on this later).
Expected answer: not confirmed. The evidence does support EUR 15,121.25.
Follow Priya. Tom wrote after her, but the controller wrote last and backs Priya.
Keep D5, the only green cell: 642.75. Drop C5 and C7 (yellow: July fees, already invoiced) and D4 (tan: another customer’s fee, under review).
Keep every entity’s freight, not just the BV’s.
Read each date in its entity’s format. ENT-NL (the Dutch BV) and ENT-DE (the German GmbH) write day first. ENT-US writes month first. So 08/04/2025 means 4 August and 07/08/2025 means 8 July.
Keep the seven August charges: FC-7002, 7003, 7010, 7011, 7020, 7021 and 7023, for 14,478.50. Add 642.75 and you get 15,121.25.
And the draft’s 10,760.80? Read every date day first. The US entity’s 8 July charge sneaks into August, and three real August charges drop out.
Result: Only 5 of 25 configurations passed, all of them DeepSeek. Most of the others found 15,121.25, then cited an email as a source record, which the scorer counts as wrong evidence. Three GPT-6 Luna settings wrongly confirmed the draft’s 10,760.80 immediately.
Boss level: every format in one case
Let’s make it even harder.
Here’s a case that combines an Outlook .msg email, an Excel workbook, a screenshot and ledger postings the model has to find with its own SQL:
Only six configurations in the benchmark passed the evaluation above.
DeepSeek V4 Pro never got that far. It failed every screenshot case at every setting, while its smaller sibling DeepSeek Flash passed 8 of 12.
Here’s how GPT-6.1 Sol at low reasoning nailed it.
The results
The benchmark runs with three different model providers by default, each one configured with low, medium, high and max reasoning.
GPT-6.1 Sol at low reasoning takes the top spot: 19 of 22 for 3.7 cents a case. Cranking it up to xhigh reasoning level buys the exact same 19, for 54% more.
Now put every configuration on one chart, score against price:
Five observations
More thinking rarely pays: The top reasoning level beat every cheaper setting in exactly one family: Claude Sonnet, which jumped from 14 to 18 passes at 4.3× the cost. In the other five settings, it tied or lost to a cheaper setting.
DeepSeek Flash is really good: DeepSeek Flash at high reasoning passed 17 of 22 for 0.67 cents a case. The whole benchmark cost it 15 cents.
Models “fix” the spreadsheet they were asked to check: One receipt has transposed digits, and the payments formula includes the July subtotal, so it counts July twice. Some models got it right.
Passed test, with significant cost implications. Right number, wrong paper trail: 42 of 86 failed answers had the right verdict and the right value to the cent. They failed on evidence. Tracking the error source was part of the tests.
Hard cases stay hard: I rate each case’s difficulty from its sources, formats and reasoning steps, never from model scores.
Which model would you take?
The variety of checks and results render the main idea: one single model might not be the best model in every single case.
Also, in this particular benchmark, the performance tier of the flagship models at max reasoning (cheap at a scale) get very close to (expensive) more capable models.
for high-volume matching, DeepSeek Flash at high reasoning, with a human on potential misses.
Cases involving SQL and image processing: GPT-6.1 Sol at low reasoning.
In any case, skip the max reasoning settings and insist on the evidence trail.
The benchmark code is available on Github: https://github.com/rubenafo/finrec-bm.
Build your benchmark! It will save you time and budget.