A measured audit of the operating layer built on top of Claude Code between roughly 12 July and 3 August 2026: what it is, what rules bind it, how data moves through it, and what the numbers say — including where they say we were wrong.
Traced to a log, a test result, or a file inspected live this session. Source named on the slide.
Reasoned from measured inputs. Assumptions stated inline so you can reject them.
We do not have the data. Said plainly rather than filled with a plausible number.
The rule I held myself to: no number appears here that I could not produce from a file today. Where the system has been telling itself a story with no data behind it, the slide says so. Three of those turned up.
projects/ops/infra-report/MEASURED-2026-08-03.mdFour honest sentences, then the evidence for each.
CLAUDE.md tells every agent to check for transport proof does not exist.The single most uncomfortable finding: this system's central claim is that a rule which isn't enforced will decay. The strongest evidence for that claim is the system's own rules decaying — the caching bug, the Monday-HTML bug, and the dead-file-reference bug all recurred after being written down. That is not a failure of the thesis. It is the thesis, demonstrated at our expense.
Worth stating plainly, because every improvement claim is relative to this.
CLAUDE.md that is read at session start.Each exists because something specific failed. None is decorative.
The app feed, Slack routing, WhatsApp send path. No push notification exists — every write is a durable record, not an alert.
The Overseer role, fresh-context verifiers, the dispatch doctrine. A builder's "done" is never evidence.
68 zero-token daemon jobs + scheduled Claude tasks across two accounts, split by whether the work needs judgment.
lane.mjs / lane.py — one choke point that swaps a subscription token for an API key on failure, so a limit degrades cost instead of breaking the system.
The spine: one hand-edited JSON per domain; every human-readable view is generated from it. A hand-edit to a mirror is overwritten by design.
4 hooks — one on every prompt, three intercepting tool calls and blocking the send.
4 @-injections inlined into every single turn: health guards, ownership registry, coaching ledger, business freshness.
The founding lesson of the whole system, and the one with the cleanest proof.
Rule 5E used to read: "before recommending anything, check the ruled-out sections." An adversarial test asked obliquely whether giving a pint at a blood drive would help. The system answered "Yeah. Yeah it would." Every guard existed — in a file nobody opened. Asked for one marker three times, it gave three different numbers.
The fix was not a better sentence. It was an @ import, so the guards are in the context window before the question arrives. Four now run on every turn:
The cost is real and worth naming: roughly 12–14k tokens of every context window is spent before you type. Estimate At current mixed rates that is on the order of $0.02–0.05 per turn in input, largely cacheable — which is exactly why the 11.4% cache-hit rate later in this deck matters more than it looks.
Does it work? Yes, and it is testable: the guards are what make the same question return the same number twice. Unmeasured — we have never run a controlled test of guard-on vs guard-off, so the size of the effect is unknown. Only its direction is proven.
A hook runs outside the model. It cannot be talked out of it, and a long thread cannot forget it.
| Hook | Fires on | Does | Born from |
|---|---|---|---|
gate0.sh | every prompt | Injects the four-question pre-flight into the turn | answers that skipped the deep path |
check-outbound-file-refs | Slack / Monday sends | Blocks a message naming a file the reader can't open | Dean: "the files aren't attached again, just fake hyperlinks" |
check-test-artifact-writes | Monday / Slack writes | Blocks test artifacts reaching live surfaces | 13 fake "$9.99 overspend" alerts hit the live feed in one day |
check-dispatch-brief | subagent spawn | Blocks a write brief with no role / no travel block | added today — rules kept not travelling |
The pattern that makes these work is choke-point, not call-site. The Slack firewall lives inside slackDM() with the IDs hardcoded, so code written next month inherits it and an .env edit cannot widen it. Fixing N call sites guarantees the N+1th reintroduces the bug — we have the receipts on that: the Monday HTML bug was fixed twice and returned twice.
Honest limit: hooks only fire in Claude Code sessions rooted in this repo, and they load at session start. The desktop app may not honour them. Unmeasured — we have never counted how many outbound sends happen outside hook coverage, so the real enforcement rate is unknown.
Why your corrections used to "revert" — and why they stopped.
run_pipeline.pyA hand-edit to any mirror is silently overwritten on the next build. That is not a wart — it is the mechanism. It is also precisely why edits kept disappearing before: they were being made to the copy, not the source.
The best artefact in the system is HANDOFF.md, and the reason is instructive: it cannot go stale because nobody types into it. It is rebuilt every pipeline run as a pure query over live state. Compare the five competing hand-written handoff docs from early July that a fresh agent could not choose between.
This is the part that most differs from stock. Follow a health question through it.
Every substantive answer declares FAST or DEEP. A silent classification is a skipped one. Unsure means DEEP — the costs are asymmetric.
No lookup step. Trial history, hard flags and the current estimate for every marker were inlined before the question arrived.
One subagent per source-domain, each reading its slice in its own context, returning verbatim excerpts with citations — forbidden from concluding. No single context reads everything; that's what makes the deep path fit in a window.
A dossier file, then one synthesiser that reads only the dossier.
Cite-or-stop — every claim carries file + section. Trial-history check — cite what was already tried, then ask what changed; never "ruled out". The only-Nick test — if the answer would read the same for a generic 39-year-old man, it's wrong and doesn't send.
In the engine specifically this is enforced in code, not prose: a 7-layer entailment gate scores 100% recall on the must-reject corpus with zero false rejects on must-pass. The blood-donation endorsement, the three-different-hematocrits answer and the fabricated taper are all structurally blocked, not discouraged. Measured
Same shape for every domain. The rule is that each hop has exactly one owner.
run_pipeline.py regenerates every mirror + the injectionsTwo rules do the real work here. Single-owner: every feed, file and task names one owner at creation — because a session once concluded the reps system didn't exist and built a second one. Nothing hardcoded: every value a user reads is either live from a feed with a declared cadence, or an honest empty placeholder. Inventing plausible data is banned specifically because it is indistinguishable from real data — which is exactly how a dead feed hides.
It has failed in exactly the predicted way. An audit found 8 of 11 health feeds with 2–6 writers fighting over them, era-live double-written by two schedulers, and the finances view stuck on May behind a pointer nobody advanced. The registry is right; adherence to it drifts.
Source: spend-log.json, 2026-07-12 → 2026-08-03, parsed in full. 8,779 calls.
🔴 This figure excludes Claude Code session usage entirely. There is no local meter for it. So the true cost of running this system is higher than $463 by an unmeasured amount, and I can't tell you by how much. Anyone quoting $463 as the all-in number is quoting the part we happen to instrument.
2026-07-26. One source, one root cause, fully traceable.
| Source | Calls | USD | Input tokens | cacheRead | Model |
|---|---|---|---|---|---|
| engine-answer | 1,372 | $150.65 | 14,517,410 | 0 | opus-4-8 |
| engine-verifier | 273 | $36.47 | 2,562,674 | 0 | opus-4-8 |
| engine-coach-voice | 255 | $10.20 | 343,138 | — | opus-4-8 |
| engine-gate | 622 | $1.05 | 1,029,352 | — | haiku-4.5 |
19.2 million input tokens that day. Zero cache reads. Our own BUILD-STANDARDS.md §4 already documents this exact failure from 14–17 July: "cacheRead: 0 across Jul 14–17 means it was structured so nothing could be cached." The rule was written. Nine days later the same bug ran at roughly forty times the volume.
Note the shape of the bill: 622 gate calls cost $1.05 while 1,372 answer calls cost $150.65. The cheap tier is doing its job perfectly. The expense is opus doing large-input work with no cache — a caching problem wearing a model-choice costume.
67.0M input tokens across the period. 8.6M served from cache.
This is the largest single unclaimed saving in the system, and it is not close. Cache reads bill at roughly a tenth of fresh input. We proved on 07-12 that 95% is achievable on this workload. We are averaging 11.4%.
Estimate If the 61.6M uncached input tokens of the last 23 days had run at even the 39% hit rate we hit twice, the arithmetic points to a saving in the low-to-mid hundreds of dollars over three weeks. I am deliberately not putting a precise figure on it: the per-model input prices in this log are mixed and I have not decomposed the uncached volume by model, so a single number would be false precision. The direction and the order of magnitude are solid.
Why it keeps happening: caching only helps if the prefix is stable. Every time a prompt is assembled with something variable near the front — a timestamp, a freshly-generated block, a reordered context — the whole prefix misses. It is invisible unless someone looks at this exact column, and nothing watches it.
The claim was 26k → ~1.5k output tokens per call. Here is the real series.
~93% reduction in output tokens per call, held for eleven days. Day cost went from $11.43 (07-17) to $0.55–0.90. This is the single best-verified efficiency result we have, and the claim in the brief was accurate — slightly conservative, in fact.
🔴 But the same anti-pattern is alive next door. profiler-consolidate still emits 32,000–34,599 output tokens per call — on 2026-08-02, three calls cost $1.81. The fix was applied to the job that was measured, not to the class of bug. That is our own §8.5 failure: fix the layer, not the instance you were shown.
| Source | USD | % | Calls | Output | Model |
|---|---|---|---|---|---|
| engine-answer | $254.36 | 54.9% | 3,703 | 1,158,062 | opus-4-8 / sonnet-5 |
| profiler | $54.71 | 11.8% | 246 | 2,663,025 | sonnet-5 |
| engine-verifier | $50.88 | 11.0% | 638 | 155,339 | haiku / opus |
| skippy-chat | $15.30 | 3.3% | 106 | 43,609 | sonnet |
| engine-coach-voice | $13.34 | 2.9% | 453 | 119,448 | opus-4-8 |
| engine-capture | $3.98 | 0.9% | 1,048 | 88,670 | sonnet-5 |
| engine-gate | $2.68 | 0.6% | 1,432 | 9,723 | haiku-4.5 |
| engine-classify | $0.29 | 0.1% | 626 | 3,592 | haiku-4.5 |
The tiering decision was correct and the data proves it. engine-gate + engine-classify + profiler-detect = 2,320 calls for $3.61. Judgment-free classification on haiku costs essentially nothing. This is "tokens for judgment, code for truth" working as designed — one tier down.
The uncomfortable read of the same table: engine-verifier costs $50.88 — 11% of everything — to check work. That is defensible (a fresh verifier is worth what it costs, and it has caught real defects). But it has never been measured against what it catches. Unmeasured
Direct count from both lane-log.jsonl files.
| Log | Window | subscription | api | 429 limits | failovers |
|---|---|---|---|---|---|
| skippy-app | 07-29 → 08-03 | 156 | 130 | 65 | 62 |
| health engine | 07-31 → 08-03 | 593 | 239 | 242 | 238 |
The mechanism is doing exactly what it was designed to do. A subscription limit degrades cost instead of breaking your chat or Gracie's reply to Chantelle. That was the design goal and it is met. The honest correction is to the story, not the engineering: we say "everything runs on tokens", and in reality a large minority of calls are billed to the API key because the subscription pool 429s under real load.
🔴 And the audit trail we told ourselves to check does not exist. CLAUDE.md instructs every agent: "Check byTransport in the spend log before claiming the engine is or isn't on tokens — don't assert it from the config." There is no byTransport key on any of the 23 days. An agent following that instruction literally would find nothing and — if it were sloppy — might report success. A rule pointing at a field that was never implemented is worse than no rule.
The only systematic quality measurement we have. health/engine/qa-battery/.
| Battery | First run | Latest | Delta | Read |
|---|---|---|---|---|
| chat-battery | 30/51 · 59% | 42/51 · 82% | +23 pts | real, large improvement |
| leak retest (round 3) | 9/15 · 60% | 14/15 · 93% | +33 pts | privacy leaks nearly closed |
| round2-battery | 40/57 · 70% | 42/57 · 74% | +4 pts | marginal |
| interaction-battery | 37/45 · 82% | 41/50 · 82% | flat | no movement |
| chat-battery-nuance | — | 7/20 · 35% | — | worst result in the system |
Two things must be said about this table and neither is flattering.
1 · The nuance battery fails two questions in three. It is the hardest set — the sideways-asked, judgment-heavy questions — which is exactly the class this whole architecture exists to get right. 35% is not a rounding error on the mission; it is the mission.
2 · Every number here was produced between 26 and 28 July. Nothing has been re-run in six days, through a period that included substantial engine changes. So the improvements are real but historical, and the 35% is simultaneously our latest and our stalest figure.
Enforcement inventory, counted live.
*.selftest.mjsharness-*.mjs_selfchecks/🔴 A number I nearly put on this slide wrong. A naive count returns 287 Python test files, which would have looked excellent. Most of them are vendored numpy and joblib tests sitting inside a .venv. Ours is about fifteen, in the health engine, plus ten duplicated inside a deploy bundle. I mention it because it is precisely the kind of number that gets quoted once and then repeated forever.
The three-tier model is genuinely good design: tier 1 runs on every build and is hermetic and sub-second (~74 checks in ~9s); tier 2 runs on every deploy, fingerprint-cached; tier 3 runs on every engine write, because a prose defect in a data feed is authored upstream of anywhere a browser check could see it. A tier-1 red means dist/ is never staged, so the gate cannot be walked past.
The honest caveat on all of it: gates prove that a specific known failure is blocked. They say nothing about unknown failures, and we have no counterfactual — we cannot say how many defects would have shipped without them. Unmeasured
The best single illustration of how this system fails — and how it eventually catches itself.
Nothing caught it for seven days because the watchdog counted via: "slack" as delivered — and that one value covered both the push that arrives and the self-post that doesn't. Five "your briefing is ready" cards sat unacknowledged.
The fix is the interesting part: the value "slack" was deleted from existence, replaced by explicit ones where slack(self-dm-silent) is defined as NOT escalated. The ambiguity that allowed the bug to hide was removed, rather than the bug being patched.
Read that last stat honestly: a 126:2 pass ratio looks superb, but a heartbeat only records tasks that ran. A task that never fires writes no FAIL row — it writes nothing. Which is exactly the failure mode that hid the biz-inbox drain being dead for four days.
Three sources of truth for one fleet, and they disagree.
roster-liveness.json tracksThese cannot all be right, and no process reconciles them. This matters more than it looks: the ownership injection is one of the four blocks inlined into every turn — it is the thing agents consult before building something. A registry that overcounts by 2× is still a registry, but its authority is not what we think it is.
31 July, 02:15. Chrome had accumulated 2,168 unreaped child processes, filling the per-user process limit of 2,666. Nothing on the machine could fork — not Claude's shell, not the deploy runner's children. Every remedy (kill, killall, launchctl) itself needs a fork, so no session could repair it. It needed your hands. Roughly four hours lost.
Three standing rules came out of that, and they are why I declined the 60-agent swarm yesterday: automated browser work drives a separate testing binary, never your primary Chrome; a rising zombie count is treated as a health signal; and parallel build lanes stay capped while the process table is unproven.
.bak / .pre-*) — ~517 MBThe snapshot habit is correct and is eating the disk. "Snapshot before you edit" has saved real work. But nothing ever prunes them, so there are now more backup copies of daemon jobs than daemon jobs. This is a five-minute fix that has never been anyone's job — which is itself the finding.
Two rules, both born from specific failures.
Why org-owned matters: a Shared Drive survives anyone leaving, and membership is access. Personal-Drive ownership would have made every file a liability the day someone left.
| Repo | Remote | Commits | Tracked files |
|---|---|---|---|
| workspace root | none | 23 | 4,169 |
business-app (deck-business) | github.com/nick-deck/deck-business | 66 | 961 |
"Move everything off my Mac" is not a git push, and that is the good news. Roughly fourteen of the fifteen deployable codebases are folders inside one local-only repo — a repo whose 4,169 tracked files also contain your labs, the kids' profiles, Chantelle's material, the vault index and household finance. So the unit of work is split, then push. Once the split is decided each piece is small.
🔴 Two live credential files are tracked — a Google OAuth client secret and a working refresh token — since the initial snapshot commit. They are harmless only because there is no remote. The migration is precisely the event that converts them into a real exposure. They are flagged and deliberately not rotated, because rotating a credential is one of the three things that needs your say-so and a surprise rotation breaks a live upload path with nobody knowing why.
The sequencing that follows: secret-scan and untrack before the first push, not after. gitleaks is free and local; a pre-commit hook costs nothing and closes the class permanently.
"Ruled out" was abolished as a concept. Your proof: creatine, refused in December, daily now. Prior trials are cited as dated evidence and then re-asked — never used as a permanent exclusion. Only genuine safety flags stay hard.
Be sceptical of the left-hand column. Everything in it has, at some point, been dropped by a long thread. That is the entire reason the right-hand column exists — and why yesterday's rule became today's hook.
| Dimension | Verdict | Evidence |
|---|---|---|
| Answer consistency | Much better | Guards inlined; same question returns the same number. Was three answers for one marker. |
| Main answer quality | Better | 59% → 82% on chat-battery M |
| Privacy leaks | Much better | 60% → 93%; firewall throws in code, 12/12 M |
| Hard/nuanced questions | Poor | 35% on the nuance battery, stale 6 days M |
| Per-component cost | Much better | Profiler −93%; 2,320 cheap calls = $3.61 M |
| Overall cost discipline | Worse than it looks | 11.4% cache; a $204 day from a known, documented bug M |
| Reliability of delivery | Mixed | Fixed properly after 7 silent days; heartbeat can't see a task that never ran |
| Continuity across threads | Much better | Generated handoff can't rot; checkpoint discipline survives a cutoff |
| Duplicate systems | Better, not solved | Registry inlined every turn — but its own counts disagree 2× |
| Time saved | Unknown | Zero instrumentation. Any figure would be invented. |
Stated because you asked not to be sold to.
What I would say, defensibly: the architecture is sound and unusually well-evidenced for something built this fast. Its failures are almost entirely adherence failures, not design failures — the rules were right and drifted. That is a much better problem to have than the reverse, and it is fixable with mechanisms rather than resolve.
Zero-token check on the daily cache-hit ratio; alert under a threshold. We have proven 95% achievable and average 11.4%. This is the biggest unclaimed saving and the bug has now recurred twice at increasing scale.
byTransportThe field CLAUDE.md orders every agent to check doesn't exist. Either add it to the spend log or delete the instruction. A rule pointing at nothing is worse than none.
Six days stale, through heavy change. Re-run, then make it recurring — quality has no watchdog at all right now.
profiler-consolidateStill emitting 32–34k output tokens per call. Same bug, sibling job. Applying the fix to the class costs an hour.
76+14 vs 35+57 vs 146. Pick one source, generate the rest, and let the injection carry a number that's true.
gitleaks + a pre-commit hook. The two tracked credential files are safe only while there's no remote — and the remote is the plan.
Deliberately not on this list: the nuance battery's 35%. It's the most important number in the deck, but it's an answer-quality research problem, not a maintenance task — it deserves its own session, not a bullet at the bottom of a punch list.
spend-log.json in full — all 23 days, every source, every model.lane-log.jsonl files line by line.git, find, wc directly on the working tree today.qa-battery result file and counted pass/fail.The obvious next step, if you want it: point a fresh verifier at MEASURED-2026-08-03.md and have it independently reproduce every figure from the same files. That is the standard this deck argues for, and it has not been applied to the deck itself.
projects/ops/infra-report/MEASURED-2026-08-03.md · deck source: projects/ops/infra-report/index.html