Ship Math: Measure AI Engineering Productivity by What It Ships

Stop counting AI activity. Start measuring what ships.

AI coding tools make it easy to produce a large number: tokens consumed. The number rises quickly, fits neatly into a dashboard, and looks like adoption.

It also tells a weak story about productivity.

A wasteful agent run can consume more tokens than a successful one. Repeated attempts, irrelevant retrieval, and regenerated work inflate the metric. A concise run that receives the right context, makes the right change, and moves a ticket to done may consume less. Token volume measures consumption; it does not tell a leadership team whether the work contributed to an outcome.

The question worth answering is simpler: what did the AI help the team ship?

Ship Math is a measurement method for answering that question without treating a model’s activity log as proof of value.

The measurement gap: consumption is not contribution

A token chart can tell you that an organization is using AI. It cannot tell you whether the usage advanced software delivery, where the useful work happened, or which practices to fund.

That gap matters most in large, legacy codebases. The work is rarely a clean, isolated generation task. It involves understanding existing behavior, hidden dependencies, business rules, and regressions that are not obvious from a ticket alone. An AI agent can look busy while it navigates uncertainty. The agent becomes more useful when it starts from accurate context about the system it is changing.

That is why a durable productivity measure needs to be connected to the work system, not just the model system. It needs to link calls to tickets, tickets to outcomes, and outcomes to a defensible economic value.

Ship Math does that in five steps.

1. Tag: make every call attributable

Start by instrumenting the work. For each agent action and MCP call, capture enough context to understand what it was trying to advance:

The key decision is to capture the ticket relationship at the point of work, not reconstruct it weeks later from vague logs.

A spec retrieval against a billing service is not equivalent to a generation attempt against an authentication component. Both are AI activity, but they play different roles in the work. Tagging preserves that distinction.

This is not an argument for collecting everything indefinitely. It is an argument for collecting the fields that make a call auditable against an engineering outcome.

2. Bind: connect calls to the ticket they advanced

Next, group tagged calls by the ticket they were intended to advance. Attach the ticket’s story-point size or the organization’s equivalent work-size measure.

Binding changes the unit of analysis. A call is no longer just a request to a model. It has an outcome address: the ticket it served. It also has the work’s intended scope.

Calls that cannot be bound should be visible as unbound. That is useful operational data, not an embarrassment to hide. It can reveal exploratory work, missing workflow discipline, or instrumentation gaps. The point is to keep the chain honest.

3. Baseline: estimate the manual effort with discipline

Story points describe relative size. They do not establish how long the work would have taken without AI assistance.

Ship Math uses a baseline estimate: for a closed ticket, ask one or more qualified reviewers or models to assess the manual effort required for a competent engineer without agent assistance. The assessment should rely on the ticket, the final change, and the touched code—not on a generic prompt.

Use consensus and preserve the range. A baseline is an estimate, not discovered truth. Treating it as precise down to the minute destroys credibility. What matters is a repeatable method, the evidence used, and a visible confidence range.

The resulting calculation is straightforward:

If the baseline is uncertain, label it as uncertain. If the available evidence is insufficient, keep the ticket out of financial rollups until it is resolved.

4. Credit: distinguish the calls that moved the work

Not every call associated with a ticket deserves equal credit.

A useful measurement system separates calls that materially moved the ticket toward completion from retries, dead ends, and re-reads. The test is deliberately strict: a call should receive credit only when its output appears in the completed work or demonstrably unblocks a later contributing step.

This is the credit-assignment problem in engineering terms. The goal is not to assign blame to unproductive calls. It is to identify the patterns that repeatedly help work move.

For complex software, those patterns often begin with understanding. The agent needs verified context about how the relevant system behaves, what depends on it, and where a change could create risk. Raw generation becomes more valuable after that context is correct.

CoreStory is designed for that part of the workflow. It builds persistent code intelligence from the codebase and makes that intelligence available to people and agents. A CoreStory Spec describes component behavior; its intelligence model gives teams a shared, code-grounded foundation for questions that otherwise trigger guesswork.

5. Bank: translate the chain into operating metrics

Once calls, tickets, baseline effort, and contributing actions are connected, teams can calculate measures that an engineering leader and a finance partner can discuss together:

The conversion to dollars requires a loaded engineering cost per hour. That figure is specific to the organization. It should be sourced from finance, documented, and versioned with the reporting period—not hard-coded in a dashboard.

A representative Ship Math calculation might use:

It then reports the result as a chain, not a magic ROI number. The chain is what makes the output defensible in a budget discussion.

Why this matters for AI engineering programs

An AI program can have high activity and low delivery impact. It can also create meaningful delivery impact while using fewer tokens than a less disciplined program. Usage metrics alone cannot distinguish those cases.

Ship Math gives leaders several practical advantages:

It rewards outcomes over exhaust

A team is not incentivized to maximize calls or tokens. The metric favors work that progresses a ticket to completion with evidence behind the contribution.

It exposes where context improves results

When credited calls are analyzed by type, teams can see whether verified retrieval, dependency tracing, generation, or testing most often precedes completed work. This creates a more useful investment decision than “which model had the most requests?”

It creates a common language for engineering and finance

Engineering can examine call patterns and delivery flow. Finance can examine the documented assumptions behind recovered capacity and spend. Neither side has to accept a token chart as a proxy for business value.

It improves the feedback loop

The measurement chain does not only justify AI investment. It guides improvement. If a large share of calls are unbound, improve workflow instrumentation. If many calls are retries, inspect agent context and task framing. If verified retrieval is consistently present among contributing calls, make that context easier to access.

Build the measurement layer on trustworthy system understanding

The quality of the metric depends on the quality of the evidence beneath it.

In an unfamiliar or long-lived codebase, the most consequential question is often not “can the model generate code?” It is “does the agent understand the system it is about to change?” Manual documentation is often incomplete, stale, or too fragmented to answer that reliably.

CoreStory provides a persistent intelligence layer for large and legacy codebases. It analyzes code to create a queryable intelligence model, including Specs that describe how components behave. Through MCP, agents can retrieve governed context as they work rather than relying on broad prompt stuffing or unverified guesses.

That makes Ship Math more than a reporting exercise. The same context layer that makes AI work safer and more grounded can become evidence in understanding why certain calls contributed to delivery.

Start with one ticket, then scale carefully

Do not begin by promising a quarterly ROI number. Begin with a closed ticket and make the chain work end to end.

Only then should a team roll the approach across a portfolio of tickets. Consistency, evidence retention, and clear treatment of uncertainty matter more than a fast dashboard.

Move past the token chart

AI engineering deserves a productivity measure that reflects engineering outcomes.

Ship Math connects AI activity to the ticket, the effort, the contribution, and the economic result. It helps teams identify not merely how much AI they used, but which work it helped ship—and what conditions made that work successful.

For teams working in complex codebases, the condition that matters most is reliable context. CoreStory provides the persistent code intelligence that helps humans and AI agents understand systems before they modify them.

Explore the Ship Math method and model the inputs for a representative ticket: https://ship-math.corestory.ai/calculator/