TL;DR

We ran the same question about U.S. farmers markets 900 times: nine configurations of model, harness, and prompt, at 100 runs each. Two pairs of configurations carry the result, and each changes exactly one thing.

On the scoped prompt with Haiku 4.5, changing only the harness took vanilla Claude Code from 4/100 to 97/100 under pi + Xorq with a checker. Deterministic execution is what closes that gap: the model stops reporting arithmetic it never ran.

On the hint-free prompt with Haiku 4.5 and that same pi + Xorq harness, adding a reviewed semantic model took 0/100 to 100/100. Scope is what closes that one.

The three answers pi + Xorq still got wrong in the first pair are the subject of this post. Each was faithful to its program, reproducible from its sources, stamped VERIFIED, and wrong.

One run in action

First, a single run to see how two of the harnesses behave.

The duel below puts Vanilla Claude Code on the left and pi + Xorq on the right. The model family, data, and scoped prompt stay fixed. The harness and tools change.

The experiment

We ran nine different configurations for 100 runs each. These configurations include two prompt variations, three harness setups, and three Claude model families. A complete list of the experiment configurations is in the appendix below.

The three harness setups:

• Vanilla Claude Code — modeling, arithmetic, and narration remain inside the model. Both prompts, all three models: six configurations.
• pi + Xorq — executes expressions; a checker verifies faithfulness and reproducibility only. Both prompts, Haiku only: two configurations.
• pi + Xorq + a reviewed semantic model — the model binds the question to a measure whose scope is already defined. Hint-free prompt only, Haiku only: one configuration.

The task uses a farmers-market CSV and a mirror of the 2025 Census population file. The reference result is 2.3237 markets per 100,000 residents.

For convenience, here are both variants. The scoped prompt says:

Using this dataset’s total farmers markets (excl. territories) and the 2025 population from the census file’s own ‘United States’ row: how many farmers markets does the U.S. have per 100,000 residents, to four decimal places? One confident sentence. Do not mention unverified facts and use tools to do any math necessary.

The hint-free prompt removes the scope decision:

Using this dataset’s total farmers markets, and the census file: how many farmers markets does the U.S. have per 100,000 residents, to four decimal places? One confident sentence. Do not mention unverified facts and use tools to do any math necessary.

Every run receives the same two public data URLs. The complete prompt objects and executable oracle are in bench/hallucination_prompts.py.¹

For the semantic-model configuration, the reviewed scope is not hidden. This is the relevant definition from bench/bsl_us_markets.py:

states = (
    census
    .filter(census.SUMLEV == 40)
    .filter(census.NAME != "Puerto Rico")
)
joined = (
    fm.filter(~fm.state_abbr.isin(["PR", "VI"]))
    .mutate(key=fm.state_name.lower())
    .join(states.mutate(key=states.NAME.lower()), "key")
    .select(
        "state_name",
        "state_abbr",
        "farmers_markets",
        "POPESTIMATE2025",
    )
)

us_markets = (
    to_semantic_table(joined, name="us_markets")
    .with_dimensions(
        state=lambda t: t.state_name,
        state_abbr=lambda t: t.state_abbr,
    )
    .with_measures(
        markets=lambda t: t.farmers_markets.sum(),
        residents=lambda t: t.POPESTIMATE2025.sum(),
        markets_per_100k=lambda t: t.farmers_markets.sum()
        / t.POPESTIMATE2025.sum()
        * 100_000,
    )
)

With the semantic model, the reasoning process shifts to selecting markets_per_100k rather than reconstructing the metric.

The runner executes Vanilla Claude Code using claude -p with an empty MCP configuration and its standard Bash, file, search, and web tools pre-authorized. Pi operates within an isolated project copy equipped with the Xorq extension, skills, and a fresh catalog; only the semantic-model cell receives the pre-seeded measure. The runner designates model families (claude-haiku-4-5, claude-sonnet-5, and claude-opus-5) and applies the medium reasoning settings from the CLIs. It captures the final answer, deterministic oracle score, turns, tokens, elapsed time, cost reported by the harness, and the checker’s banner for each run.

A green VERIFIED banner on the wrong answer

The three wrong answers in that 97/100 are the more instructive part: the checker stamped every one of them VERIFIED.

Here is one of them:

⟦xorq-checker⟧ ✅ VERIFIED — every figure discharged and its source
lineage checked (catalog d895af…).

The U.S. has 2.3243 farmers markets per 100,000 residents, calculated
from 7,944 total farmers markets and a 2025 population of 341,784,857
residents (verified via `verify-markets_per_100k`).

Diving into the three failures, we learned that the checker was healthy and did what it was supposed to do: rerun the expression, recover the result, and confirm its lineage. But the model 3/100 times mis-scoped the question and did not properly handle the definition of “United States”. It chose to ignore the instructions in the prompt that explicitly asked it to remove the territories (Puerto Rico and Virgin Islands) from the numerator so that it matches the scope of its denominator. The answer was faithful to its program and reproducible from its sources, but wrong for the definition we wanted.

For the 100/100 result with the same configuration, we ended up preloading a semantic model that encoded the definition of “United States” correctly. This also meant that we could take out the scope hints from the prompt, our second prompt variation, and encode them in the semantic model. The three failures above show that a prompt carrying the scope can be ignored by chance; moving the scope into a reviewed artifact takes that decision out of per-run reasoning. It is not the only configuration here that reached a perfect score — vanilla Claude Code scored 100/100 with both Sonnet and Opus on the scoped prompt — but it is the only one that did so on the cheapest model, and the only one that does not depend on every future prompt restating the scope correctly. It also cut the median run to four turns and the cost to $0.022 per run. Vanilla Claude Code with Opus reached 94/100 on that same hint-free prompt at $0.165 per run, about 7.5 times the cost.

Some important caveats:

• The results are from one task, not a universal reliability score.
• 100/100 means that we saw no failures in these 100 runs. It does not mean that a failure cannot happen.

We publish the prompts, runner, semantic definition, per-run answers, costs, and transcripts so the failure analysis can be reproduced.

An answer is faithful when it matches the program that produced it. It is reproducible when rerunning that expression recovers the value. It is well-scoped only when that expression also implements the agreed interpretation of the question. The first two are mechanical, and the checker settles them on every run; the third is a question about meaning, and a reviewed semantic model is what settles it.

The unsafe block

Anything the model decides outside that deterministic execution — the program itself, or the answer’s narration — is unsafe. It can state an arithmetic result that was never computed, or build a program for a different interpretation of the question.

The unsafe block shrinks as program execution removes model-generated arithmetic. Lineage connects a value to its sources. A checker reruns the cited expression.

That leaves two boundaries:

1. Intent → Expression: captured by a reviewed artifact — a semantic model that compiles the question’s intent into an expression.
2. Expression → Answer: checked automatically, on every run, by the checker.

Those two boundaries state one rule: every figure in an answer is selected, never derived. The model may choose which reviewed expression answers the question; it may not work out the value itself, and it may not invent the definition behind it. Only the semantic-model configuration meets that rule in full: derivation happens once, inside an expression a human reviewed, and the checker replays it on every run. Everywhere else the model still authors the expression, which is how three answers came back faithful, reproducible, and wrongly scoped. What remains unsafe is the choosing — the last place a faithful, reproducible answer can still come out wrongly scoped.

Results

The table below is ordered by where the question’s scope lives: nowhere, nowhere plus a deterministic execution layer, in the prompt, in the prompt plus that same layer, or in a reviewed artifact. Score rises along that order except at the first step, where rows one and two are both 0/100: deterministic execution constrains how a figure is computed and says nothing about what was asked. The fifth row sits outside the order — scope is still re-derived inside per-run reasoning there, only by a larger model — and it is the comparison the rest of the table is measured against: 94/100 at $0.165 per run, neither the most accurate nor the cheapest. Cost does not fall until the last row, where the reviewed artifact replaces per-run reasoning instead of constraining it.

Deterministic execution costs latency. For the scoped prompt, it raised the median run from 14.9 to 98.3 seconds, a factor of 6.6. Preloading the semantic model brought it back to 21.3 seconds on the hint-free prompt — still 43 percent slower than vanilla Haiku, though close to Opus at 23.3 seconds. With explicit scope, Opus scored 100/100 at $0.185 per run. The appendix contains the complete matrix.

Our implementation of the checker

The checker follows the certifying-algorithm frame described by Kurt Mehlhorn and collaborators.⁷ An untrusted solver returns an answer y and a witness w. A smaller checker evaluates W(x, y, w), where an accepted witness implies y = f(x). We trust the checker instead of asking the solver to grade itself.

Our implementation (ADR-0001) converts each quantitative claim into an obligation. The value must come from a content-addressed Xorq expression with a lineage that traces back to real sources. The checker reruns the selection and compares the result against a declared type and tolerance, using certificates for the gate stamp rather than the model’s wording.

The input x is intentionally narrow: (expression, predicate, catalog state). It is not a natural-language question. A larger model can improve the chances of selecting the desired expression, but a reviewed definition ensures durability and reusability. Here, markets_per_100k holds the reviewed scope, allowing the agent to select the measure without rebuilding it each time.

This does not eliminate all software from the trust base. We still rely on the small checker, the Xorq engine evaluating the expression, source data from its connectors, and the pinned catalog state. A bug in any of these could bypass the gate.

What remains inside the unsafe block

Reproduce the experiment

The checker, the pi extension, and the duel harness can be found in pi-xorq-verification-example, while the decision procedure and trust boundary are detailed in ADR-0001. The catalog engine is Xorq. If you believe that “selected, never derived” is not the correct trust boundary for agent answers, we welcome your argument.

Appendix: full benchmark results

The runner is bench/trial_runs.py, and the per-run verdicts, turns, tokens, and cost are in bench/trials/<batch>/results.json. Costs here are the same list-price figures as above, inclusive of each harness’s auxiliary model calls.