CoreStory V3 Benchmarks · October 2026
AI coding agents get more changes right with CoreStory V3
We tested seven AI models, four frontier and three open-weight, with and without CoreStory V3, and scored them against answer keys the agents never saw. With V3, every model fixed or flagged more hard tickets, kept more unstated rules intact on everyday changes, and spent less per rule found. Separately, we compared one full V3 extraction pass with each frontier agent working alone.
Key results
The results at a glance
We compared each model only with itself, first working alone and then with CoreStory V3. We never average results across models.
of complex changes done right or flagged with V3, compared with 4% to 36% without it.
of business rule locations found by V3, compared with 42% to 88% for agents alone.
lower cost per business rule found.
What we tested and what we saw
| Test | Result |
|---|---|
| Mapping business rules 256 known rules at 515 places in a 250,000-line codebase | One full V3 pass found 98% of those places. Agents working alone, one business area per session, found 42% to 88% per run. |
| Complex changes 14 change requests across 4 codebases, each depending on a rule the request never states | Fixed or flagged in 40% to 74% of attempts with V3, compared with 4% to 36% alone. Fixed outright in 19% to 44%, compared with 0% to 4%. |
| Everyday changes 8 or 9 everyday changes per model, each with several unstated rules | Claude Sonnet 5.5 kept 80% of unstated rules intact with V3, against 50% alone. Claude Haiku 5.5 kept 67%, against 41%. |
| Finding every place a rule lives 250,000-line codebase | 17% to 35% lower cost per rule found with V3. Places found held about level for Sonnet, Opus and gpt-6.1-sol; Haiku's fell from 42% to 31%. |
Complex changes
With V3, agents catch the rules a ticket doesn't mention
Each model worked through 14 change requests across four codebases, and every request depended on a business rule it never stated. We counted a change as fixed when it passed tests for both the requested change and the hidden rule. We counted it as flagged when the agent didn't fix it but pointed to the exact place and value that conflicted, so a reviewer would know where to look.
| Model | Fixed or flagged, alone | Fixed or flagged, with V3 | Fixed outright, alone | Fixed outright, with V3 |
|---|---|---|---|---|
| Frontier models | ||||
| Claude Haiku 5.5 | 20% | 60% | 1% | 19% |
| Claude Sonnet 5.5 | 14% | 66% | 0% | 32% |
| Claude Opus 5.5 | 36% | 74% | 1% | 44% |
| gpt-6.1-sol* | 4% | 40% | 4% | 34% |
| Open-weight models | ||||
| DeepSeek-V4.1-Flash | 43% | 76% | 26% | 58% |
| GLM-5.3 | 29% | 73% | 14% | 50% |
| Kimi-K3.1-Pro | 29% | 61% | 24% | 50% |
We chose these changes because agents working alone fail them, so their solo scores are a low baseline rather than typical results. Each frontier model made 70 attempts alone (5 per change) and 140 with V3 (10 per change). * gpt-6.1-sol ran in Codex, which doesn't support the hooks CoreStory uses to check edits as they happen, so its V3 results don't include those checks.
Open-weight models
We also ran three hosted open-weight models through the same 14 changes, with the same setup, rules and scoring. Only the model changed, and we used default settings with no tuning. With V3, every open-weight model caught more hidden rules than any frontier model working alone.
of complex changes fixed or flagged by DeepSeek-V4.1-Flash with V3, close to Claude Opus 5.5 with V3 at 74%.
the cost per fix of Claude Opus 5.5 for DeepSeek-V4.1-Flash, with both running V3. It's also about one-fifth the cost per fix of Claude Sonnet 5.5 with V3.
Cost per fix is total spend at list prices divided by the number of fixes. With V3, DeepSeek-V4.1-Flash completed 139 attempts, GLM-5.3 completed 140 and Kimi-K3.1-Pro completed 133, and each made 70 attempts alone. We checked the open-weight runs for leaked answers and reran two frontier models through the same setup to confirm it doesn't inflate scores.
Mapping business rules
V3 maps your business rules once, and every agent reuses that map
One full CoreStory pass vs. agents working one business area at a time. We built a 250,000-line COBOL and Java codebase with 256 known business rules spread across 515 places. V3 read the whole codebase once, in a pass of about eight hours, and wrote down every rule it found. Each frontier model worked alone, one session per business area, so this is not a same-budget comparison. The same automated scorer, with no AI judge, checked V3 and every model against one answer key.
| System | Places covered (of 515) | Details exactly right (of 169) |
|---|---|---|
| CoreStory V3 | 98% | 85% |
| gpt-6.1-sol | 88% | 81% |
| Claude Opus 5.5 | 76% | 80% |
| Claude Sonnet 5.5 | 54% | 63% |
| Claude Haiku 5.5 | 42% | 40% |
Agent results are per run. A finding counts only if it points to 80 lines of code or fewer. A place counts as found when the rule is written down within five lines of where it lives, and a separate check tests whether every value in the rule is right. We left out the 87 rules that pull their values from database tables, since those values aren't in the code for anyone to read.
Everyday changes
Everyday changes are ordinary feature work. Each one relies on several business rules it doesn't mention, and agents working alone usually get some of them right. We measured how many of those rules still worked after the change.
| Model | Unstated rules kept intact, alone | With V3 |
|---|---|---|
| Claude Sonnet 5.5 | 50% | 80% |
| Claude Haiku 5.5 | 41% | 67% |
| Claude Opus 5.5 | 71% | 80% |
| gpt-6.1-sol* | 64% | 73% |
Each model ran nine everyday changes, and Claude Haiku 5.5 ran eight.
Cost efficiency
Before an agent changes a rule, it has to find every place that rule lives. With V3, each place cost less to find. Sonnet, Opus and gpt-6.1-sol found about as many places as alone (54%, 77% and 88%); Haiku found fewer (31%, against 42% alone).
| Model | Lower cost per rule found with V3 |
|---|---|
| Claude Opus 5.5 | 34.6% |
| Claude Sonnet 5.5 | 34.4% |
| Claude Haiku 5.5 | 32.2% |
| gpt-6.1-sol | 17.3% |
Cost per rule found is total model spend across all searches divided by the number of places found. In this test, agents couldn't look up each rule's specific values.
How we measured
How we kept the tests fair
- Hidden answer keys. Every test has an answer key the agents never see, listing where each rule lives and what its values are. Code does the scoring, and no AI model acts as judge.
- Each model against itself. On every with-and-without test, we compared each model only with its own results without V3, and we never average across models. The extraction test is the one exception: it ranks one full V3 pass alongside each model working alone.
- Codebases with known answers. We built three test codebases with known rules (COBOL banking, Python billing, and a COBOL and Java freight system) and added one real open-source Java e-commerce app. We didn't use any customer code.
- Agents alone had the full codebase. Without V3, agents could search and read the whole repository with their usual tools. V3 added only CoreStory's business rules, and the agent, model and test setup stayed the same.
- Models. Claude Haiku 5.5, Claude Sonnet 5.5 and Claude Opus 5.5 from Anthropic and gpt-6.1-sol from OpenAI ran every test. Three hosted open-weight models ran the complex changes.
- One deep read, reused. To find business rules, V3 reads the whole codebase once in a thorough pass, while each agent gets one session per business area. Every later task reuses that read.
Results measured with CoreStory V3, October 2026. Model names belong to their respective vendors.
Give your agents the rules your tickets leave out
Try CoreStory free on your own code, download the full benchmark report, or talk to us about what V3 finds in your codebase.