The ideas, briefly
A deterministic app
The app itself fills by rules: every field’s meaning comes from them, so the same fields always map the same way. That matters to engineering as much as to trust: when a form breaks, the fix is a rule or a code change plus a regression test that proves it. A language model was built into the app as a tie-breaker and then measured: it got far more wrong than right, so it was switched off.
Run by AI agents, checked by scripts
The pipeline is where the models work. AI agents run each round: they drive the scripts, label the answer key, review every audited field and write the fixes. The agents that label can read only the worksheet they’re handed, and anything a model changes in the app, a rule or a fix, has to pass checks that aren’t models before it ships: the rule validator, the test suite and the scorecard. Rulings on what the app should do are made by a person and written down.
The pipeline, and the canon it builds
The pipeline is the instrument. It lives outside the app and never runs in it. What it knows is the canon: a profile of how each applicant-tracking system builds its forms, and an answer key of real postings labelled field by field. Every fact in a profile records where it came from, and measured facts expire and are either measured again or downgraded, never quietly re-dated.
An answer key the engine can’t influence
Real postings are captured and frozen, then labelled field by field by AI labelling sessions that are never shown what the engine decided. A second, independent pass re-labels a sample, and any disagreement is settled by a person answering a plain question that never says which side gave which answer. The blinding is built into the tooling rather than promised: a labelling session’s tools can’t open the engine’s output, and every label is sealed with a hash that each tool checks before reading it.
A scorecard that leads with precision
The shipped engine is graded against the answer key by a deterministic script; no model does the grading. The first question is how often a value the engine proposes is right. Coverage is reported at that precision, three ways, never as one bare number. A wrong value and a value where none belonged count as different harms, and the engine is compared with the cost of filling nothing at all.
Rules proposed by a model, decided by evidence
A local decision model, a language model rebuilt to choose from a fixed list of answers rather than write text, proposes rules for fields the engine leaves unmatched. A deterministic validator tests each proposal across the whole answer key, and it must still hold with each platform left out in turn, under a policy fixed before the run, and a person accepts or declines each mined rule in an append-only ledger. Rules can also change when an AI agent fixes a finding; that change is reviewed adversarially by a separate model and pinned by a test before it merges. A model’s answer may add a field to review, never remove one.
Proved in the real app
Frozen pages only go so far, so the pipeline’s last step has two halves. The fill audit drives the real app on iPhone simulators against live postings and records every step, and AI reviewers judge every field. The device check compares three views of each posting: what the canon captured, what the app read and what the phone actually showed. Whatever the app missed becomes a finding that an AI coding agent fixes in the app. Whatever the canon missed is fixed in the capture tools and the platform’s profile, and becomes a saved field when an applicant would want it kept.
What the full notes add
- The measured figures behind each step, and how each one is estimated.
- Real excerpts from the work: a platform profile, a sealed label, a rule decision, an audit record.
- The invariants the app keeps, and the tests that hold each one.
- Frames captured by the fill audit.
I share the full notes, and a password to read them, with people who have signed an NDA.