One of the most tedious and time-consuming parts of commercial insurance operations is submission intake: reading the emails brokers and agents send a carrier or MGA requesting a quote, and pulling out everything an underwriter needs to price the risk. A single submission can bundle a broker email, a statement of values listing every building at risk, and a stack of supporting documents. The addresses, values, and construction details in them often disagree from one page to the next.
Most carriers and MGAs handle this today with a dedicated intake team, either outsourced or in-house. Both have real drawbacks. Outsourcing means overnight turnaround at best, so a broker waits a day for a reply, and somebody still has to manage and retrain the external team. Keeping it in-house fixes turnaround but costs more, and retaining people to do repetitive work is its own problem.
That is why submission intake is one of the first places carriers are applying AI. Increasingly it means an engineering team calling a frontier model API directly, handing over the broker's files, and trusting that a strong enough model can read the spreadsheet.
We wanted to know whether that holds up. So we ran a benchmark, built around the question underwriting and IT leaders actually have to answer: build this in-house on a general-purpose model, or buy a system built for the problem?
The three systems
We compared three systems on the same 157 real broker submissions, with the same ground truth and the same scoring applied to each one.
ResiQuant Agent is our intake system, which takes a submission from the broker's email through to the fields an underwriter prices from.
Claude Opus 5 is Anthropic's frontier model, called directly through the Anthropic API.
GPT-5.6 is OpenAI's frontier model, called directly through the OpenAI API.
Each system received the same thing a carrier receives: the documents exactly as the broker sent them. That is the comparison a carrier is actually weighing, because a team evaluating this will drop the broker's attachments straight into a model and see what comes back.
None of the systems had any advance sight of the answer key. The ground truth was labeled and verified by insurance domain experts who had no hand in building any of the three systems, ResiQuant's included, so no system had an advantage in how it was scored.
What the systems left out
Across the 157 submissions, ResiQuant Agent returned 98.7% of the buildings in the ground truth, missing 43 of 3,481. Claude returned 90.0% and GPT-5.6 returned 79.6%, missing 349 and 710 buildings respectively.
That gap compounds at the file level. A complete schedule means every building in the submission was accounted for. ResiQuant Agent produced one on 146 of 157 submissions. Claude Opus 5 managed 137 and GPT-5.6 managed 117.
On the fields themselves, we checked every extracted value against the answer key: roughly 60,000 field checks per system across the benchmark. ResiQuant Agent got 91.2% of them right, against 80.9% for Claude and 71.7% for GPT-5.6.
It is worth being precise about what "missed" means, because it is different from "wrong." When a system omits a building or a value, it returns nothing for it: no answer, no flag, no sign that anything was skipped. A misread construction class gets caught the moment an underwriter opens the file. A building that was never returned is not caught by anyone. It becomes exposure that nobody prices.
Bigger schedules, more left out
The pattern that matters most for a carrier is what happens as schedules get larger, because that is where the exposure concentrates.
ResiQuant Agent stays between 94% and 100% across every size band. Both models fall away as the schedules grow, and they stay 12 to 23 points behind ResiQuant Agent on everything larger than a single building. On the 18 submissions with more than 50 buildings, which hold more than half of all the buildings in the benchmark, Claude returns 88% and GPT-5.6 returns 77%. On a 100-building schedule that is twelve to twenty-three buildings that never come back, and nothing in the output says which ones.
A clean-looking spreadsheet is not a guarantee
The obvious assumption is that this is a messy-document problem, confined to application forms, scanned tables, and unusual layouts. Layout does matter, and it is worth seeing how much.
A plain grid is the easy case: one header row, one row per building. On those, Claude returns 93% of buildings and GPT-5.6 returns 82%, against our 99%. On every other layout the models drop to 86% and 78%, while we hold at 96%.
The layouts that cause the trouble are not obvious in advance. Thirteen submissions came back with more than half the schedule missing under the better of the two models, and six of those thirteen were ordinary single-row-per-building spreadsheets, with no merged cells, no missing columns, and nothing that would make a reviewer look twice. "Most of our submissions are clean spreadsheets" is not, on its own, a reason to expect a general-purpose model to get this right.
On what they do find
When Claude or GPT-5.6 returns a building, it is usually accurate about the coverage values attached to it. Both score about 90% on the value fields they fill in. ResiQuant Agent scores 96.6% on the same measure, so the three systems are closer here than anywhere else in the benchmark.
The difference is in how much they fill in at all. Of the component values that genuinely exist in the source documents, ResiQuant Agent leaves 0.3% blank. Claude leaves 5.0% and GPT-5.6 leaves 5.3%. Those blanks carry no warning either.
There is one more difference the accuracy numbers do not capture: where a value came from. ResiQuant Agent links the values it extracts back to their source across the document types a submission bundles together, down to the row and column of a spreadsheet cell or the exact spot on a PDF page, so a reviewer who wants to verify a number clicks once and is looking at it. That holds whether the value came from the SOV, the broker's email, or a supporting document. A model API returns a number with no trail. When something looks off, the only way to check it is to go find it in the documents yourself.
This is the practical problem for a carrier. If a model skips one building in ten before accuracy has a chance to matter, someone still has to review the whole schedule to find what is missing, which is most of the manual work the carrier was trying to remove. Getting the found rows right is necessary but not enough on its own. Field accuracy has to be high enough that a reviewer can spot-check rather than re-key, and completeness has to be high enough that spot-checking is a safe thing to do, and both only pay off if checking a value is fast.
What this means for build vs. buy
If your team is weighing whether to build submission intake in-house on an off-the-shelf model, the question is not whether Claude or GPT-5.6 can read a spreadsheet. They can, and they improve with every release. The question is what a ten to nineteen point accuracy gap costs you once the work reaches a person.
Think about what those points mean in a review queue. At 91%, a reviewer can spot-check: look at the fields that matter, trust the rest, and move on. At 72% to 81%, roughly one field in four or five is wrong, and there is no way to know which one. That is not spot-checking any more. That is opening every property, reading every value against the source document, and correcting as you go, on every submission that arrives.
The cost of that lands twice. You are paying a person to re-key work you already paid the model in tokens to produce, and you are back to the turnaround time you adopted AI to remove. The efficiency gain a carrier is buying does not come from the model reading the file. It comes from a reviewer being able to trust the output enough not to check it line by line. Below a high accuracy bar, that gain does not exist at any price.
Completeness is the same argument one level up. A shorter schedule that still looks complete gives a reviewer nothing to react to, so the only safe response is to reconcile the whole building list against the source by hand.
This is why we think near-perfect extraction, rather than good extraction, is what unlocks the efficiency. It is also why we do not expect a general-purpose model to get there on this problem by prompting alone. The failures in this benchmark are not the model misreading a cell it looked at. They are the model deciding, with no signal to the reader, that a row or a value was not there.
