<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom">
<channel><title>BLACK BOX — Engineering</title><link>https://blog.acarr.org/engineering/</link><atom:link href="https://blog.acarr.org/engineering/feed.xml" rel="self" type="application/rss+xml" /><description>Engineering notes and the systems I run. Every claim demoable.</description><language>en-us</language><lastBuildDate>Sun, 20 Sep 2026 12:00:00 +0000</lastBuildDate><item><title>Credentials Never Reach the Model</title><link>https://blog.acarr.org/posts/credentials-never-reach-the-model/</link><guid isPermaLink="true">https://blog.acarr.org/posts/credentials-never-reach-the-model/</guid><pubDate>Sun, 20 Sep 2026 12:00:00 +0000</pubDate><category>engineering</category><category>mcp</category><category>claude</category><category>workpulse</category><category>oauth</category><category>python</category><description>Publishing the WorkPulse MCP server: fourteen tools over a real invoicing API, two credential domains that never mix, batch rollback for every bulk write, and Streamable HTTP behind OAuth.</description><content:encoded><![CDATA[<p>WorkPulse is the business-management app I run my own consulting through: clients, projects, time, invoices, expenses, pay periods. It has a REST API. For most of this year I've had a small MCP server sitting on top of that API so Claude Code and Claude Desktop can do the boring parts — "create an expense for $127.50 at Office Depot on the 20th, deductible" — without me tabbing into a form. Today I split it out of the private monorepo and published it.</p>
<p>The interesting parts aren't the tools. They're the two boundaries around them.</p>
<h2>Fourteen tools, one API</h2>
<table>
<thead>
<tr>
<th>group</th>
<th>tools</th>
</tr>
</thead>
<tbody>
<tr>
<td>expenses</td>
<td><code>create_expense</code>, <code>list_expenses</code></td>
</tr>
<tr>
<td>invoices</td>
<td><code>create_invoice</code>, <code>generate_invoice</code> (from billable time entries), <code>list_invoices</code></td>
</tr>
<tr>
<td>contracts</td>
<td><code>create_contract</code>, <code>list_contracts</code></td>
</tr>
<tr>
<td>lookups</td>
<td><code>list_clients</code>, <code>list_projects</code></td>
</tr>
<tr>
<td>batches</td>
<td><code>start_batch</code>, <code>list_batches</code>, <code>rollback_batch</code></td>
</tr>
<tr>
<td>receipts</td>
<td><code>process_document</code>, <code>process_documents_folder</code></td>
</tr>
</tbody>
</table>
<p>Every tool is a thin, typed wrapper over one WorkPulse endpoint. The receipts tools call a separate OCR pipeline (DocTR plus a local Ollama model) that extracts transactions from PDFs and images; the server maps them to expenses and supports <code>dry_run</code> so you can see what it <em>would</em> create first.</p>
<h2>Boundary one: the model never holds a credential</h2>
<p>There are two completely separate credential domains, and the code is organized so they cannot mix.</p>
<p><strong>Outbound.</strong> The server reads a WorkPulse username and password from environment variables at startup, logs in, and gets a JWT. It refreshes that token thirty seconds before expiry and falls back to a full re-login if the refresh fails. Tool calls carry only the bearer token, internally. Nothing about this ever appears in a tool description, a tool result, or a prompt. The model can create an invoice; it cannot learn how the server is allowed to.</p>
<p><strong>Inbound.</strong> Over stdio (the default, for Claude Desktop and Claude Code on the same machine) there's no network surface, so there's nothing to authenticate. Over Streamable HTTP or SSE, the MCP endpoint itself is protected by a bearer token verified against an OAuth 2.1 authorization server. The <code>mcp</code> SDK's own auth middleware enforces it and publishes RFC 9728 protected-resource metadata, so a client that gets a 401 is told exactly where to go get a token. Three verifiers plug in by environment variable:</p>
<ul>
<li><strong>jwt</strong> — validates a signed access token locally against the issuer's JWKS: signature, issuer, audience, expiry.</li>
<li><strong>introspection</strong> — RFC 7662 for opaque tokens.</li>
<li><strong>static</strong> — one pre-shared token, constant-time compared, for development.</li>
</ul>
<p>And one rule that I think matters more than the verifiers: <strong>the HTTP transports fail closed.</strong> If no auth mode is configured, the server refuses to start. You have to write <code>MCP_AUTH_MODE=none</code> yourself to get an unauthenticated network endpoint, and the example config tells you not to.</p>
<h2>Boundary two: every bulk write can be undone</h2>
<p>An MCP server that can create fifty expenses from a folder of receipts is also a server that can create fifty <em>wrong</em> expenses from a folder of receipts. So every create goes into a batch. Batches persist to disk, survive restarts, and <code>rollback_batch</code> deletes everything the batch created, newest first. The folder-processing tool opens a batch automatically. You can hand the model a directory and still have an undo button.</p>
<p>This is the same instinct as the <a href="/posts/what-an-ai-facade-has-to-actually-do/">AIFacade</a> work: don't try to make the model never wrong. Make the system unable to silently absorb it when it is.</p>
<h2>What publishing actually took</h2>
<p>The code had been fine for months. Publishing it took a day, and almost all of that was the boring, load-bearing part:</p>
<ul>
<li>Fresh repository, fresh history. The private monorepo's history has real credentials in it, so nothing was carried over — copied, not moved, then scrubbed and grepped to zero hits. The old password is being rotated regardless; once it's in a history, it's burned.</li>
<li>Migration to the <code>mcp</code> 2.x SDK, since that's what a fresh <code>pip install</code> gets today.</li>
<li>Forty-five tests: token expiry and refresh fallback with a mocked API, all three verifiers including rejection of wrong audience, wrong issuer, expired, and <code>alg=none</code>; the middleware returning 401 with no token, 401 with a bad one, 403 without the required scope, and 200 with a good one; and an in-process end-to-end run from <code>initialize</code> through <code>tools/call</code>.</li>
<li>Example configs for Claude Code, Claude Desktop, and the HTTP transport, all with placeholders.</li>
</ul>
<p>Repo: <code>github.com/aliasdhacker/workpulse-mcp</code>. MIT.</p>
<p><em>If you run a small business on your own API, the pattern is the reusable part: env-only credentials on the server side, OAuth on the endpoint, fail closed, and an undo for anything that writes in bulk. — A.C.</em></p>]]></content:encoded></item><item><title>What an AI Facade Has to Actually Do</title><link>https://blog.acarr.org/posts/what-an-ai-facade-has-to-actually-do/</link><guid isPermaLink="true">https://blog.acarr.org/posts/what-an-ai-facade-has-to-actually-do/</guid><pubDate>Sat, 19 Sep 2026 12:00:00 +0000</pubDate><category>engineering</category><category>aifacade</category><category>llm</category><category>governance</category><category>go</category><category>ollama</category><category>claude</category><description>You do not govern generated code by reading it. Four things a real AI facade has to do — schema boundary, deterministic repair with receipts, fixtures that referee, an audit trail with no exceptions — each demoable against a Go binary in my closet.</description><content:encoded><![CDATA[<p><em>I promised myself this post couldn't be written until it was true. It's true now — every claim below is a curl command against a Go binary running on a box in my closet.</em></p>
<hr />
<p>A few weeks ago, a governance specialist asked me two very fair questions about a financial tool I'd built in six days with AI doing most of the typing:</p>
<blockquote>
<p><em>Did you review every line it generated? And if models cross-check each other, what stops one model's drift from being ratified by the other?</em></p>
</blockquote>
<p>I had a tempting answer ready — "my control plane handles that." It would have been a lie. The control plane existed, but it didn't yet do the things the claim implied. And he was the worst possible audience for an inflated claim: a person whose literal follow-up would have been <em>show me</em>.</p>
<p>So I gave the true answer instead: no, I did not review every line. I reviewed every <em>behavior</em>, and I built the system so line-review was unnecessary where it mattered. Then I made a deal with myself: the tooling claim gets written down the day it's demoable, and not one day earlier.</p>
<p>This is that post.</p>
<h2>The thesis</h2>
<p>You do not govern generated code by reading it. At the speed AI produces code and output, review-by-eyeball is theater — you will skim, you will approve, and the one value with a dropped decimal point will sail through, because a plausible-looking number is the single hardest thing for a tired human to catch.</p>
<p>You govern it with structure: build the system so it is <strong>unable to silently absorb the model's mistakes</strong>. I call the pattern an <em>AI facade</em>, in the plain Gang-of-Four sense — one surface through which AI enters the system, constructed so nothing consequential passes through unchecked. Not a wrapper around the model. A boundary around everything the model touches.</p>
<p>A real facade has to do four things. Here's each one, with the receipts.</p>
<h2>1. A schema boundary the model cannot talk through</h2>
<p>Every consequential completion declares an <code>output_schema</code> — a strict JSON Schema. That schema is enforced <strong>twice</strong>, on purpose.</p>
<p>First, at the model boundary: generation itself is constrained (local models via grammar-constrained decoding, cloud models via structured outputs). Second — and this is the part people skip — the returned output is validated <em>again</em> in deterministic code, because the belt should never trust the suspenders. Parse, then a bounded repair pass (strip markdown fences and prose preambles — code fixing wrappers, never a model fixing a model), then validate against the schema.</p>
<p>If the output can't be made valid, the request returns <strong>HTTP 422 with no <code>output</code> field at all</strong>. A rejected value isn't flagged-but-available. It is structurally absent. Downstream code cannot consume what the response does not contain.</p>
<h2>2. Deterministic repair, with receipts</h2>
<p>Here's the failure that started all of this. During the six-day build, a model read a credit-card statement's APR of <strong>33.61%</strong> and returned <strong>3361</strong>. Grammar-valid. Type-correct. Catastrophically wrong — and exactly the kind of plausible number that survives human review.</p>
<p>The schema's range bound (<code>0 ≤ apr_percent ≤ 100</code>) catches it. The first version of my facade stopped there: rejected, 422, done. But the original incident taught a better lesson — the error class is <em>known</em>. A percentage that is exactly 100× too large is a dropped decimal point or a basis-points confusion, and the fix is arithmetic, not judgment.</p>
<p>So the facade carries a <strong>normalization rule registry</strong>: deterministic transforms for known failure classes, under four hard rules —</p>
<ol>
<li>A rule fires <strong>only</strong> on a value that already violates its schema bounds. Valid output is untouchable.</li>
<li>A repair must land the value in full schema validity, or it doesn't happen.</li>
<li>If two rules propose different fixes, <strong>nothing is normalized</strong>. Ambiguity is rejection, never a guess.</li>
<li>Every applied repair is recorded — rule, field, before, after — in the response and the audit log.</li>
</ol>
<p>Today the 3361 replay comes back like this:</p>
<pre><code class="language-json">{
  &quot;validation&quot;: { &quot;status&quot;: &quot;normalized&quot;,
    &quot;repairs&quot;: [{ &quot;rule&quot;: &quot;scale-down-100&quot;, &quot;field&quot;: &quot;apr_percent&quot;,
                  &quot;from&quot;: 3361, &quot;to&quot;: 33.61 }] },
  &quot;output&quot;: { &quot;apr_percent&quot;: 33.61 }
}
</code></pre>
<p>Caught <em>and</em> healed, with provenance. And one more discipline, enforced by a unit test that fails otherwise: <strong>every rule in the registry must cite the fixture that discovered its failure class.</strong> No provenance, no registration. Rules are scar tissue; scar tissue has a story.</p>
<h2>3. Fixtures referee. Models never do.</h2>
<p>His second question — drift amplification — deserves a structural answer, not a reassuring one.</p>
<p>The facade runs two model families side by side (qwen and llama locally, with a Claude branch wired the same way). Any schema-constrained request can ask for cross-model validation: the same request runs against the <em>other</em> family, and deterministic code diffs the two parsed outputs field by field. When they agree, you get provenance. When they disagree, you get the diff:</p>
<pre><code>outcome: DISAGREE
diff: number: expected 42687, got 854321
</code></pre>
<p>And that's <em>all</em> you get — a flag. No third model votes. No "smarter" model arbitrates. Disagreement is resolved by the other half of the system: a <strong>fixture suite</strong> — known-correct input→expected-output pairs, versioned with the repo, run through the full facade by an eval harness whose comparator is the same deterministic differ. Exact fields are exact; numeric fields declare their tolerance.</p>
<p>That's the drift answer in one sentence: <strong>model output is never promoted to ground truth, so there is no channel through which one model's drift can be ratified by another.</strong> Ground truth lives in fixtures, and fixtures only grow — every caught model failure becomes one before its fix counts as done.</p>
<h2>4. An audit trail with no exceptions</h2>
<p>Every completion the system executes — user chat, background task execution, the summarizer, the eval harness, the cross-model validators themselves — lands in one ledger: model, route, tokens, actual cost, what the cloud <em>would</em> have cost, latency, validation outcome, applied repairs, cross-check outcome. Failed completions too — a request that produced nothing still produces a row, with the error on it.</p>
<p>The ledger has a UI, but the principle matters more than the page: <strong>if it isn't in the ledger, it didn't happen.</strong> When the cloud branch is unreachable, requests degrade to the local model — and the routing decision records exactly why: <code>force_cloud hint (cloud failed: API key unset; served locally)</code>. Degradation is fine. <em>Silent</em> degradation is the disease.</p>
<h2>The supporting rules</h2>
<p>Three smaller disciplines hold the four pillars up.</p>
<p><strong>Models write prose; code writes facts.</strong> The system has an AI scribe that summarizes work sessions and maintains project briefs — narrative only. Every consequential number (counts, durations, costs, statuses) is computed in SQL. When session summaries are distilled into durable key points, even that extraction runs through the same schema pipeline as everything else, tagged with its source.</p>
<p><strong>A model must never be able to mark work done.</strong> My task dispatcher can auto-execute simple tasks through a local model. The day I added "promote this tech-debt item to a task," I realized the dispatcher would happily hand <em>"refactor the tilemap loader"</em> to a 7B model, accept three paragraphs of prose, and mark the task <strong>completed</strong>. Fake work, marked done — the most dangerous output class there is. Promoted tasks now carry a tag the dispatcher is forbidden to touch. Structure again, not vigilance.</p>
<p><strong>Honest claims only.</strong> While building this, I caught myself writing "Phase complete — all 8 items" into the plan when item 3 wasn't done. I'd studied a dead project that marked epics Complete for code that didn't exist; that autopsy is why this rule exists. The plan now says 7 of 8, with the open item named. If your status documents lie, your containment story is already over.</p>
<h2>What this runs on</h2>
<p>None of this needed a platform team. One Go binary, SQLite, an embedded web UI, a consumer GPU box running 7-billion-parameter models, and a cloud fallback that costs cents when it's used at all. That's the part I find genuinely encouraging: <strong>containment is what makes small local models usable for consequential work.</strong> A 7B model inside this facade is more trustworthy than a frontier model on the honor system — because the trust was never in the model. It's in the structure.</p>
<h2>The point</h2>
<p>The question was never "did you read what the AI wrote." The question is: <em>what happens when it's wrong?</em> If the answer involves a human noticing, you don't have governance — you have hope.</p>
<p>Build the surface it enters through. Bound what it can say. Repair only what you can repair deterministically, and write down every repair. Let fixtures referee, never models. Log everything, including the failures. Then you can give the governance specialist the answer I can finally give:</p>
<p>No, I didn't review every line. The system did — and unlike me, it never skims.</p>]]></content:encoded></item><item><title>APR 3361</title><link>https://blog.acarr.org/posts/apr-3361/</link><guid isPermaLink="true">https://blog.acarr.org/posts/apr-3361/</guid><pubDate>Sat, 19 Sep 2026 12:00:00 +0000</pubDate><category>engineering</category><category>aifacade</category><category>llm</category><category>containment</category><category>demo</category><description>The number that started the whole containment project — and what it looks like today when the same mistake hits the facade.</description><content:encoded><![CDATA[<p>Last month a local model read a credit-card statement and returned this:</p>
<pre><code class="language-json">{ &quot;apr_percent&quot;: 3361 }
</code></pre>
<p>Grammar-valid. Type-correct. Thirty-three hundred percent APR. It's the single most dangerous class of model output there is — a <em>plausible-looking number</em> — because a tired human reviewing a page of extracted fields will skim right past it. I know, because I did.</p>
<p>It came out of PayDown's statement import; the build-log entry from that day is <a href="/posts/the-import-was-all-messed-up/">The 3,361% APR</a>. That one value is why <a href="/posts/what-an-ai-facade-has-to-actually-do/">AIFacade</a> exists. This post is the short version, with the receipts, because I recorded a demo against the live instance today and every number below is copied from the responses.</p>
<h2>The fixture passes</h2>
<p>The founding case lives in the repo as <code>fixtures/02-apr-statement.json</code>: the statement excerpt, a strict schema (<code>apr_percent</code> is a number in <code>[0, 100]</code>), the expected value <code>33.61</code>, and a tolerance of half a basis point. Run through the facade on a 7B Qwen model:</p>
<pre><code class="language-json">&quot;validation&quot;: { &quot;status&quot;: &quot;passed&quot; },
&quot;output&quot;: { &quot;apr_percent&quot;: 33.61 },
&quot;route&quot;: &quot;local&quot;, &quot;latency_ms&quot;: 2519, &quot;cost_cents&quot;: 0
</code></pre>
<p>Two and a half seconds, zero cents, correct.</p>
<h2>The replay gets caught — and healed</h2>
<p>Now the original failure. The OCR pass dropped the decimal, so the statement text literally says <code>APR: 3361</code>, and the model faithfully reports it. The schema's range bound rejects the value. Version one of the facade stopped there: HTTP 422, no <code>output</code> field at all, so nothing downstream could consume it.</p>
<p>But this failure class is <em>known</em>. A percentage that is exactly 100× too large is a dropped decimal point or a basis-points confusion. The fix is arithmetic, not judgment. So today the same replay returns:</p>
<pre><code class="language-json">&quot;validation&quot;: {
  &quot;status&quot;: &quot;normalized&quot;,
  &quot;repairs&quot;: [
    { &quot;rule&quot;: &quot;scale-down-100&quot;, &quot;field&quot;: &quot;apr_percent&quot;, &quot;from&quot;: 3361, &quot;to&quot;: 33.61 }
  ]
},
&quot;output&quot;: { &quot;apr_percent&quot;: 33.61 }
</code></pre>
<p>The normalization registry runs under four hard rules:</p>
<ol>
<li>A rule fires <strong>only</strong> on a value that already violates its schema bounds. Valid output is untouchable.</li>
<li>A repair must land the value in full schema validity, or it doesn't happen.</li>
<li>If two rules propose different fixes, <strong>nothing is normalized</strong>. Ambiguity is rejection, never a guess.</li>
<li>Every applied repair is recorded — rule, field, before, after — in the response and in the audit log.</li>
</ol>
<p>And one more, enforced by a unit test: every rule must cite the fixture that discovered it.</p>
<pre><code class="language-json">{ &quot;name&quot;: &quot;scale-down-100&quot;,
  &quot;description&quot;: &quot;A percentage read with its decimal point dropped (or given in basis points): value is exactly 100× too large for its bounds. Divides by 100.&quot;,
  &quot;fixture&quot;: &quot;fixtures/02-apr-statement.json&quot; }
</code></pre>
<h2>A second model checks — code decides</h2>
<p>Add <code>routing_hints.validate</code> and the same schema-constrained request runs against the other model family. Deterministic code diffs the two parsed outputs field by field:</p>
<pre><code class="language-json">&quot;cross_check&quot;: { &quot;outcome&quot;: &quot;agree&quot;, &quot;validator_model&quot;: &quot;llama3.1:8b&quot;,
                 &quot;validator_route&quot;: &quot;local&quot;, &quot;validator_latency_ms&quot;: 5435 }
</code></pre>
<p>When they disagree you get the field-level diff and a flag — and that's all. No third model votes. Ground truth lives in fixtures, and fixtures only grow.</p>
<h2>The ledger, today</h2>
<p>Every completion — chat, background tasks, the scribe, the eval harness, the validators themselves — lands in one routing ledger. As of this morning:</p>
<table>
<thead>
<tr>
<th>requests</th>
<th>served locally</th>
<th>cloud-equivalent cost avoided</th>
<th>normalized</th>
<th>rejected</th>
<th>cross-checks</th>
</tr>
</thead>
<tbody>
<tr>
<td>1,885</td>
<td>100%</td>
<td>$8.11</td>
<td>2</td>
<td>1</td>
<td>4 agree · 2 disagree · 1 validator failed</td>
</tr>
</tbody>
</table>
<p>The two normalizations and the one rejection are in there with their reasons. The <code>validator_failed</code> is honest too: the cloud branch is wired (official Go SDK, structured outputs), but the deployed box doesn't have an API key set yet, so the decision records <em>why</em> it fell back to local instead of pretending.</p>
<p>That's the whole idea. The question was never whether I read what the model wrote. The question is what happens when it's wrong — and the answer has to be structure, not vigilance.</p>
<p><em>Demo script and runner: <code>docs/demo/</code> in the AIFacade repo. — A.C.</em></p>]]></content:encoded></item></channel></rss>
