A Commit vs. a Ledger Entry

TL;DR

Agents hand work to each other through code repositories and the markdown beside them. A commit is whatever files were staged: it pins the code, but not which computation those files amount to or what it produced, so the next agent works that out again from the source, and may work out something different. An entry in a Xorq ledger is one computation, defined before it runs and named by a hash of it, its schema and lineage derived from it: an agent proposes entries on a branch, a person approves each one, and the next agent selects an approved entry by name, or composes on its schema, with nothing to work out from source. In our 900-run experiment, the same contrast on a metric definition was 4 turns and $0.02 a run against 21 turns and $0.09. TabPFN makes the gap concrete: a commit holds code with nothing tying it to the number it produced, and a TabPFN fit returns an id for the training rows it uploaded to Prior Labs, with nothing tying those rows to the sources and the split they came from. The entry ties both: the number to its computation, the rows to their source.

Which split did the last agent use?

The pipeline below prices French motor policies with three TabPFN models. It rests on one rule, written in the repository’s markdown. Hand that repository to an agent:

Fit the claim model on ~30k policies, split by hashed policy id, and the
severity model on the ones that claimed. Fit the calibration model on a
held-out slice they never saw.

The agent writes a split and commits it. The next agent, handed the same repository, has the rule and a file; either way it works the split out from source, so it writes one, and it may or may not be the same. The history now holds two splits, neither tied to the number it produced. A reviewer’s questions have no answer in the repository: which split did the last agent use? Which rows did the calibration model see? Nor does the repository say what the number was run against: a connection profile and a read path can be committed, but nothing ties the run that produced the number to the ones it used. The markdown files and memory stores agents hand each other are the same kind of unit: whatever was written, tied to nothing it produced.

In the ledger the rule is not a sentence but entries. fit-slice is the split. calibration-model is trained on calibration-slice, scored through the claim and severity models, which were fit on fit-slice; its lineage shows all of that. The next agent selects calibration-model instead of refitting it on a split of its own. It can still propose a different split; that lands as a new entry with its own hash, on a branch, for a person to approve or not, and every model fit on it names it in its lineage. (The agent still gets instructions for using the ledger; the difference is the unit, not the instructions.)

A Xorq ledger is an append-only record of entries. Each change adds or removes exactly one entry and pins the environment it was built in. Each entry also records the connection profiles and read paths it was built with; credentials stay references to environment variables, read at run time and never stored. The ledger is stored as a git repository, so you can clone it, branch it and review it with the tools you already use, but what makes it a ledger is the unit, not the storage. The CLI that writes one is xorq catalog, and its files on disk say catalog; this post says ledger.

An entry is a deferred computation: the definition, not a run of it. It is named by a hash of its computation, its parameters and the data it reads and writes, so a result is tied to exactly the computation that produced it, and the same computation built again in the same environment lands on the same entry. Everything else a reviewer reads about an entry (lineage, schema in and out, compiled SQL) is derived from the computation and the pinned environment when the entry is built, and stored beside it; reading any of it needs no run. Because schemas are derived, two entries compose without asking either entry’s author.

In Faithful, Reproducible, Wrong we asked one metric question 900 times across three Claude models. Re-deriving the metric from the source files, Claude Haiku was right 0 times in 100 when the prompt left out the metric’s scope (21 median turns, $0.09 a run) and 97 times when it spelled the scope out (24 turns, $0.10). Selecting the definition from a reviewed entry, with the scope removed from the prompt, it was right 100 times in 100, in 4 turns at $0.02. That was a metric definition, not a model fit, but what varied, working a definition out from source, is what a split shares.

TabPFN keeps the rows; the ledger keeps where they came from

TabPFN is Prior Labs’ foundation model for tables. Their insurance cookbook uses it to price freMTPL2, 678,000 French motor policies from the CASdatasets R package, served by OpenML.

Our pipeline, xorq-template-tabpfn, runs TabPFN through Prior Labs’ API. A fit uploads its training rows and returns an id for them, which Prior Labs call the fitted train set; a predict uploads the rows to score against that id, and the account can list, download and delete what it uploaded.¹ TabPFN has no fit-specific weights. A fit is a forward pass over its training rows, recomputable from the checkpoint, the rows and the config, and Prior Labs hold all three: the checkpoint on their servers, the rows under the id, the config in the fit request. What the id does not hold is where the rows came from. The rows under it are the end of a pipeline: two source files joined, claims capped, a split by hashed policy id, one slice of it. None of that is in the upload. The id is per fit call, not per rows: a second fit on the same rows and config skips the upload, because the client checksums what it sends, and gets a new id.² A checksum of the rows says nothing about the pipeline that produced them. The entry that defines the fit is that pipeline, named by a hash of it: the same fit built again lands on the same entry, and its lineage runs from the slice back to the source reads. An entry fixes the fit, not its result; where the result must not move, xorq catalog pin freezes every cache the entry reads through (source rows, fits with their ids, scored rows) into a new entry whose lineage still runs back to the source reads.

The pipeline runs the cookbook as one computation: the claim model, a classifier for whether a policy claims; the severity model, a regressor for the amount on policies that claim; their product at a full year of exposure; and, where the cookbook applies a constant correction, the calibration model, which learns the correction from a slice the first two never saw. Its tests, and the entries themselves, answer what a reviewer would otherwise take on faith:

Lineage names the fit but cannot see inside it. Library code an entry calls is opaque to lineage: each scoring step shows as a UDF[_predict_…] node, and each fit as the step that produces its model entry’s one column (dumps_of__inner_fit_0 for the claim model), a pickled object. So the pipeline’s tests check what that object is: the stock TabPFN estimator, with no API token inside it. The more of a pipeline an agent pushes into such functions, the more a reviewer has to take on the tests’ word.

The ledger, opened

The pipeline’s ledger is ten entries. To list them yourself:

$ xorq catalog clone https://github.com/xorq-labs/xorq-catalog-template-tabpfn --path tabpfn-catalog
$ git -C tabpfn-catalog checkout ledger-dated-checkpoint
$ xorq catalog --path tabpfn-catalog list-aliases

The command prints the aliases; the descriptions are ours. The entries are tables, slices, three fitted models, a scorer with no input table bound, and the scored test set:

tabpfn-insurance        the scored test set: raw and calibrated premiums
policies                one row per policy, claims summed and capped
fit-slice               ~30k policies the claim and severity models condition on
calibration-slice       ~10k policies held out for the calibration fit
test-slice              ~10k policies scored for the metrics
calibration-staged      the calibration slice scored through the claim and severity models
claim-model             the fitted classifier: a one-column table
severity-model          the fitted severity regressor
calibration-model       the fitted calibration regressor
tabpfn-scorer           the pipeline, unbound: nine columns in, priced rows out

The ledger in xorq catalog tui. The selected entry is the scored test set; its lineage runs through the three model entries down to the two source reads, and the git log is one entry per commit.

The fitted classifier as an entry: one column, named after the features and target it was fit on, with the fit slice as its lineage. The uploaded rows live on Prior Labs' servers under an id; the entry holds the fit's definition and names the cached result that holds the id.

$ xorq catalog --path tabpfn-catalog log
--- replay plan ---
[init]  5406c195  initial commit
[init]  adbdbac3  add catalog.yaml
[add]   123ae67b  entry=caf37c092526  aliases=[tabpfn-insurance]
[rm]    c67bd90b  entry=caf37c092526  aliases=[tabpfn-insurance]
[add]   671d3632  entry=caf37c092526  aliases=[tabpfn-insurance]
[rm]    4b2d3dca  entry=caf37c092526  aliases=[tabpfn-insurance]
[add]   d873fb59  entry=bec90e687291  aliases=[tabpfn-insurance]
[add]   f201aeb4  entry=b044b5f8694b  aliases=[policies]
…
[add]   13a1ab7f  entry=8647dccdcbaa  aliases=[tabpfn-scorer]

The first rm and the add after it are the same computation, caf37c092526, rebuilt against a later revision of the pipeline repository, which the pinned environment records: the hash did not move, the pinned revision did. The second rm is a different kind of change. The same pipeline on a dated checkpoint is a different computation, so it lands as a new entry, bec90e687291, and the nine after it are its parts.

What the reviewer sees

The ledger is public at xorq-catalog-template-tabpfn, with the ten entries on its ledger-dated-checkpoint branch. Its main still holds only the first entry, from before the dated checkpoint. The ten reach main when someone merges the pull request, and that merge is the review. A second pull request, stacked on it, proposes an eleventh entry, the pinned scored test set; it waits on the first.

Next month a new slice of policies arrives and an agent is asked to refit. It builds the fit slice on the new rows, fits the claim and severity models on it and the calibration model on the rows held out, and proposes the entries on a branch. The pull request the reviewer opens lists entries, not Python: a new hash on fit-slice; three model entries with the same one-column schema; a scorer with the same nine columns in and priced rows out; and a lineage in which the calibration model’s rows never appear under the claim or severity fit. Had the agent calibrated on rows those models had seen, the lineage would say so; had it drifted to the server’s default checkpoint, the hash would have moved. The reviewer approves computations, each named and typed, not whatever was staged. They merge, or do not, and what is on main is what the next agent selects from and composes on.

Every fit has TabPFN’s gap, an artifact that says nothing about where its rows came from, and the same need: a ledger to select from and compose on. The pipeline is at github.com/xorq-labs/xorq-template-tabpfn, and the script the ten entries are built from is ledger_expr.py; talk to us about where a ledger sits in your process.