What Tool Responses Can Actually Teach an AI Agent

Lessons from CoreStory Frontier’s PATOIS study

A tool response can contain more than data. It can carry a cursor, an identifier, a link, a recovery hint—or advice about what the agent should do next.

That last category is attractive. The server knows the current application state, so perhaps it is better positioned than a model’s standing instructions to explain the next move. This is the premise behind PATOIS—Prompts As The Orchestration Intelligence Strategy—a controlled research project documented through CoreStory Frontier.

The result is narrower than a simple “yes” or “no.” A response-scoped channel can teach an agent something it could not infer. It adds much less when it merely restates conventional engineering practice. And the same channel that is useful for carrying arbitrary state can become a liability when that state is derived from untrusted content.

The question is not “can the model read?”

PATOIS deliberately focuses on the part of an agent workflow that begins after the first successful tool call. The initial decision is 0→1: user intent to first tool. The research question is 1→n: once the tool has returned, can the response teach the agent how to continue?

The design compares five conditions:

The test server returned a byte-identical underlying data payload across conditions. The design separately varied structural handles and the form or location of guidance. That invariant matters: without it, an apparent guidance effect could actually be a payload effect.

The measurements were also intentionally mechanical. The study did not ask one model to grade another model’s answer. It scored recorded tool-call sequences: correct tools, correct arguments, valid ordering, unnecessary calls, stopping, recovery, and token usage. Compliance with advice was treated as a mechanism diagnostic, not as proof of benefit.

The clearest result: agents need values they cannot invent

The strongest probe placed a continuation token only in the response-scoped guidance channel. The token was absent from the data payload, absent from the handles, and not derivable from a schema or function signature. Completing the task required presenting it on the next call.

The result was decisive:

• Conditions C and D presented the token in 30 of 30 trials, with 95% Wilson intervals of 89–100%.
• Conditions A, B, and E presented it in 0 of 30 trials, with intervals of 0–11%.
• Successful runs took exactly two calls, on the first attempt.

This shows that the channel was read and acted upon. It does not, by itself, show that advice improves orchestration quality. That distinction is important. A channel can be legible without being useful for a task whose correct continuation is already inferable.

The paging scenario makes the distinction concrete. When the agent received a cursor handle, it completed the task in four calls. When the cursor value was withheld, the agent still understood the convention: it knew the result was paginated and that the parameter was called cursor. It tried thirteen plausible formats across ten trials, but none could substitute for the actual value.

The model knew the rule. It did not know c2.

That is the category where response-scoped state earns its keep: opaque identifiers, cursors, tokens, spans, and other facts that are not present in the model’s prior knowledge and cannot be reconstructed from the response shape.

Conventional orchestration was already free

PATOIS also built four situations intended to need continuation guidance. None produced a useful lift from advice.

The agent already knew how to:

• pass back a page cursor it had been handed;
• stop once the answer was available;
• validate before writing when the requirement was not stated in the schema; and
• hold a write lock for a short window.

In the lock-ordering pilot, advice even moved the structured-guidance condition toward a worse ordering: it acquired the lock before validation, holding the lock across an extra call, while the no-advice condition validated first and then locked. Both completed, so this is an illustration rather than a measured effect, but it captures the risk of treating compliance as success.

Why did these scenarios ceiling? They were built from ordinary engineering practice. A capable model already carries strong priors about pagination, completion, validation, and lock duration. If a constraint feels natural to an experienced engineer, it may not be an unlearnable local fact at all.

The paper explicitly rejects tuning a scenario until guidance became the only path to success. That would create a lock-and-key experiment: technically demonstrable, but not evidence that a real system needs the mechanism.

The practical implication is not that models never need orchestration context. It is that the context must be genuinely local and non-obvious. Repeating “call the next tool” is not the same as returning the identifier that makes the next call possible.

Structured guidance did not justify its cost

PATOIS compared prose guidance with a structured object whose vocabulary was closed: tool names, parameter names, reason codes, scope, and an advisory authority. The intended benefit was better orchestration and stronger resistance to instruction injection.

On normal orchestration, the two renderings behaved identically:

• 30/30 successful token retrievals in both conditions;
• exactly two calls and two turns in both;
• input sizes within five tokens of each other.

The cost appeared in generated output. Prose averaged 323 output tokens; structured guidance averaged 448—about 124 additional tokens, or 38%, using the unrounded means. The structured condition also showed greater variance: standard deviation 105 versus 50, with a 4.3× variance ratio.

Under attack, the structured rendering was more fragile in the observed experiment. But the paper correctly stops short of attributing that result to typing alone. The prose rendering included an explicit sentence saying that the guidance was advisory and did not override system, developer, or user instructions. The structured rendering expressed that constraint only as a field value. The comparison therefore confounds structure with the presence of a plain-language disclaimer.

That is not a reason to hide the result. It is a reason to phrase it correctly: structured guidance, as implemented, never outperformed prose, cost more output, and did not demonstrate a protective effect. A follow-up that equalizes the disclaimer is still needed before making a clean claim about structure itself.

Integrity held. Availability did not always.

The security question was not whether a compromised server could issue commands. It was whether a legitimate server could return a value derived from content someone else controls—a filename, slug, URL fragment, or machine-generated label—that contains an embedded instruction.

Across 600 corrected runs—one no-attack control and three attack styles, across all five conditions—the model made zero out-of-scope calls. The target function was declared and callable, and the no-attack control established a base rate of zero. In a separate blunt-injection set, the model also named the embedded instruction as untrusted and sanitized the identifier before using it.

That is the integrity result: the poisoned value did not redirect tool selection.

Availability tells a more complicated story. Two attack styles disrupted completion, one did nothing, and the mechanisms differed:

• A false-precondition payload was a channel effect. Conditions C and D both fell to 0/30, while the out-of-channel conditions retained some completion.
• An operator-authority payload was a rendering effect. Prose condition C completed 30/30; structured condition D completed 8/30.
• A user-intent reframe did not disrupt completion.

The study reports the provider’s response as a refusal from its safety layer, not as the agent quietly abandoning the task. The exact refusal rate differed between the in-harness runs and isolated replays, and that discrepancy remains open.

This is the uncomfortable collision at the center of PATOIS: arbitrary state is where response-scoped context has demonstrated value, but arbitrary state is also the category most likely to contain attacker-influenced text. A channel can be useful without being safe to treat as instruction.

What this means for code intelligence systems

For systems that help agents understand and modify large codebases, the boundary is especially important. A persistent intelligence layer should make relevant facts available—entities, references, spans, relationships, provenance, and opaque identifiers—without turning content-derived text into an instruction stream.

The PATOIS evidence suggests a conservative design rule:

Return the handle. Keep the orchestration advice out of the handle.

In practice:

1. Put unguessable values in explicit data fields or typed handles.
2. Keep values derived from ingested content separate from authority-bearing instructions.
3. Measure completion and failure behavior, not only whether an agent followed a recommendation.
4. Treat structured output as a hypothesis to test, not an automatic safety improvement.
5. Preserve traces, refusals, voided runs, and design corrections so the evidence can be audited.

This is also where CoreStory’s broader approach matters. Code intelligence is useful when it gives humans and agents a persistent, queryable understanding of a system grounded in the codebase. That does not require pretending that every local fact should become a prompt. Sometimes the most valuable thing an intelligence layer can do is return the exact reference an agent cannot guess—and leave the next action to the agent’s tested capabilities.

A research artifact that keeps its own corrections

The PATOIS research was conducted and authored by Bob McWhirter / CoreStory.ai. Its pinned source repository includes the proposal, design, research diary, results, current paper draft, Rust harness, and JSONL traces. The harness is offline-testable; in a local verification of that pinned source, its mock surface, guidance construction, and metrics passed 81 tests.

The public site currently exposes the results, an experimental design, the originating brief, and an older pre-draft paper. The public pre-draft points readers to a later PAPER.md, but that linked path currently returns a 404 on the public site; the current paper remains available in the pinned research source. It corrects the pre-draft in two important ways: it separates the authority-spoof rendering effect from the channel effect, and it distinguishes the original proposal from the later typed-guidance elaboration.

That distinction is not housekeeping. In empirical work, a correction that narrows a claim is part of the result. The paper’s strongest conclusion is not that response-scoped guidance is universally good or bad. It is that the useful unit is smaller: arbitrary state can be load-bearing; conventional advice often is not.

The next experiment is the one that resolves the confound

The cheapest remaining test is also the most important for the typed-guidance question: make the prose and structured renderings carry equivalent natural-language subordination, or remove it from prose, then rerun the authority-spoof row.

Other open questions remain:

• Does the distinction hold with weaker models?
• Does it survive a real retrieval system rather than a synthetic server?
• How common are attacker-influenced values in production tool surfaces?
• What availability budget is acceptable when a poisoned value causes a refusal?

The right response is not to patch the conclusion into a product claim. It is to keep the experiment honest, then extend it where the evidence says the uncertainty lives.

Read the research

Read the public Frontier artifact at frontier.corestory.ai, including the results, experimental design, and research brief. The results page currently links the published round-1 trace file. The full trace set and current paper draft are retained in the pinned research source.

CoreStory builds a persistent intelligence layer that helps humans and AI understand, modify, and modernize complex software systems. If your team is working through the context problem in a large codebase, talk to a CoreStory expert.

Further reading

PATOIS draws its framing from Roy Fielding’s description of hypermedia as the engine of application state in the REST architectural style. The analogy is useful here as a design question, not as a claim that an AI tool surface is RESTful by definition.

FAQ

What is PATOIS?

PATOIS stands for Prompts As The Orchestration Intelligence Strategy. It is a controlled research study asking whether an agent can use guidance attached to a tool response to continue a multi-step workflow.

Did the study prove that response-scoped guidance improves agents?

No. It showed a decisive benefit for one class of content: an unguessable token available only through the response-scoped channel. It did not show incremental benefit from conventional orchestration advice once the required handle or state was available.

Is typed guidance safer than prose?

The study does not establish that. Structured guidance matched prose behaviorally, cost 38% more output tokens, and was more fragile in one attack comparison. That comparison is confounded by an explicit advisory disclaimer present in the prose rendering but not the structured rendering.

What are the study’s main limitations?

The experiment used one model and endpoint, a synthetic test server, unpinned sampling, varied cell sizes, one post-observation scenario, and one provider safety layer. The results show a regime and a design boundary; they do not establish prevalence across models or production systems.

How does this connect to CoreStory?

CoreStory provides persistent, queryable code intelligence so humans and AI can work with grounded context from complex codebases. PATOIS sharpens one design question inside that broader problem: return the facts and handles an agent cannot guess, but do not assume that more response-scoped instruction is automatically better.