Software Is NothingAbout · how I work
How I work, with the receipts
I use AI coding tools heavily, every day, and I say so rather than hiding it. What follows is how I make sure the thing that ships is understood, reviewed and owned by me: the rules I hold to and where each one came from, eight refactors of a production platform with dates and commit hashes out of git history, the checks that run every morning on that platform, and a list of what I got wrong. Claims here trace to a file, a commit, a plan document or a production query. Where a number could not be verified, it was left out rather than estimated.
Spoken walkthrough
Listen: a 30-minute spoken walkthrough of this page, in my voice. Recorded with a local text-to-speech model from a script I wrote.
Read the transcript
This is Matthew Perry. What follows is a spoken walkthrough of how I work: eight refactors of my own codebase with dates, and the checks that keep the platform behind them honest. You are about to hear eight refactors of the platform I built and run alone, most recent first, each one with its date, what I actually changed, how I checked it, what I learned, and what I got wrong. After that comes the engineering behind all of it, how the system keeps thousands of scrapers honest, what fails closed, how personal data is handled, and what I check every morning. Every fact traces to git history and the plan and postmortem documents kept alongside the code, nothing synthetic. This runs about thirty minutes.
Let's start with the most recent one. The entry at the top of the page is dated the twelfth to the fourteenth of August, twenty twenty six: merging fifteen separate translation models into one. Here's what happened. A rewrite I'd just finished had failed its own quality gate, a different entry in this recording. As a stopgap I fine tuned one model per language instead. By the twelfth of August that stopgap had grown into fifteen separate models, all sharing a single graphics card. What I changed: I replaced all fifteen with a single model, trained on every locale at once, in both translation directions, gated by a corpus check that had to pass before anything could launch.
How I checked it: that corpus gate ran before every launch, and instead of one blended number, it reported recall broken out by direction. The page notes there's no plan document for this one. What I learned is that my commit messages that week put the root cause in the commit body. Now, what I got wrong. Within forty eight hours I fixed four separate bugs: a gradient checkpointing bug, a quote check that only recognized German punctuation, a gate check that never actually resolved, and a drift tool built to guard this exact migration that instead reported everything in sync while production kept running the old code underneath it.
None of those four were caught before I merged, all four surfaced within a day or two of real use. A later audit found the recall metric I'd used to justify more training was partly just the model memorizing its own repair rules. Seven commits went into this entry, about one thousand nine hundred lines net, and the hashes are there on the page if you want to check any of them yourself. The next section on the page is titled bringing the live translation pipeline back into version control, covering the ninth to the thirteenth of August, twenty twenty six. Here's what happened.
Three production fixes I had shipped on the eighth and ninth of August existed only on the serving host. They were never in git. My plan document says this exact risk had already nearly bitten me twice that same week. What I changed. I vendored the production drainer, the verifier, and the LoRA trainer into the repository, and I added a read-only host-versus-git drift checker, one that never auto-overwrites either side. How I checked it: a plan document, in docs slash plans, that I kept alongside the repository but never actually committed.
It specifies hash comparison, using SHA two fifty six, between the host and the repo, and it keeps the verifier in git as a strict superset of what is actually running in production. What I learned, in my own words from that decision record: the repository becomes the source of truth, and the host becomes a deploy target. Now, what I got wrong. The drift checker I built specifically to catch silent divergence between production and git had two bugs of its own. It reported everything as in sync while the host was actually missing thirty three safety rules.
One bug: an ssh call inside a loop consumed the rest of the loop's input, so only the first of many rows was ever actually checked. The other, a same-day duplicate-row guard hard-failed the whole check instead. Both bugs failed open, which is exactly the direction the tool existed to stop. The commit table on the page has the details for all of this, if you want to check them yourself. Next on the page is the twenty-eighth to the thirtieth of July, twenty twenty six: a from-scratch rewrite of the translation pipeline, and the decision not to ship it. Here's what happened.
The first version of the translation pipeline had a known quality ceiling, so I built a second pipeline from scratch, one that masked entities, chunked sentences, and audited the training corpus, with a blinded panel of language models deciding at the end whether it was good enough to replace the first version. What I changed was fine-tuning that new pipeline on our own corpus for English to German and English to Spanish, then running it through the ship gate against the first version. The verdict came back: does not ship. How I checked it was a code-based gate with three parts: benchmarks run on the actual serving path, safety recall checked against a floor, and a blind panel scoring translations against a bar.
There's no plan document behind this one, just a write-up afterward. What I learned is the harder part. My first full training run finished successfully, with no errors anywhere, and produced garbage checkpoints. A chunker had packed longer chunks than the trainer's old length limit allowed, and nothing in the pipeline caught the mismatch. What I got wrong: that mismatch wasted the entire first training run silently, so I archived those checkpoints with a note never to benchmark them. Even after I fixed the chunking, the corrected pipeline still failed its own ship gate, on both languages. I count that as the healthy kind of wrong.
The gate did exactly what it was built for, and it kept a worse pipeline out of production. The page notes this row runs to about thirty one thousand seven hundred lines net, but that count is inflated by training and evaluation fixtures bundled into the same commits, not by the pipeline code itself, which was a small share of the diff. The next one on the page, the one you're looking at now, is dated the tenth of June, twenty twenty six: stopping the site from silently delisting animals that are still alive. Here's what happened.
An audit found a single clock in the system, fourteen days of an animal not being seen on its source page, that was meant to catch a shelter whose feed had gone dead. The trouble was that the same clock could just as easily flip a real, living animal over to adopted, and once that happened it pushed a permanent signal to the search engine that the animal was gone. What I changed was to add non-destructive guards, along with an autopilot that revives disabled adapters before that fourteen-day clock ever fires, and I fixed the site's four-ten responses across the board. All told, the change came to about four thousand three hundred fifty net lines.
For checking it, I wrote a plan document, in docs slash plans, that laid out SQL and log checks for each part of the fix, backed by a health dashboard, and a companion write-up describes moving my scraper agents onto launchd. What I learned came directly out of that move. The scheduler I had been using before had quietly lapsed for nine days, because it only ran while an application happened to be open. Now here's what I want you to sit with, because it's the honest part. The guard worked exactly as designed. Two months later, on the twenty-sixth of August, that became its own problem.
An adapter was found re-ingesting years of closed shelter pages as if they were live listings, about six thousand and three animals, four point four percent of the entire site's inventory. I fixed the adapter itself, but the same guard, built to block a cleanup whenever an adapter's yield collapses, could not tell a regression from a correction. It blocked the legitimate cleanup indefinitely. Moving to the fourth to the seventh of June, twenty twenty six: rebuilding the animal card as a full-photo overlay, then tuning it live for three more days. A visual-drift audit found the live site had diverged from its own design system, and the product owner had rejected the card we were running.
A full-overlay card became the priority. What I changed: I rebuilt the shared card as a full overlay, with the text sitting over the photo, and rolled it out to twenty-six places across the site where that card renders, while keeping the smart-crop system underneath intact. How I checked it: the plan document required a design audit script and a type check to both pass. My own same-day review also caught a real anchoring bug. What I learned: three earlier reskins like this one had gotten clobbered by branch merges, so this time I committed page by page inside the branch. What I got wrong: the layout was not right on the first ship.
Three days later I made four more commits, reworking where the location and gender pills sat on the card, then softening both because the shipped version read too harsh against real photos. None of this was a logic bug. It was the same visual call, made again, this time against real photos. Let's move to the section labeled nineteenth of May to first of June, twenty twenty six, the one the page calls locale registry, for making one file the source of truth for every supported language. Here's what happened.
Adding a new language to the site meant manual edits in six separate places, two of them duplicate hand-maintained maps, and none of those steps failed loudly if you skipped one. Norwegian shipped on the nineteenth of May, but it was never submitted for indexing. What I changed was to make the locale registry, one JSON file, the actual source of truth, and add code generation that replaced those two hand-maintained maps. How I checked it: the plan document calls for running the new graduation script against the seeded Norwegian incident itself.
What I learned: I shipped this as one coordinated pull request rather than splitting it up, because the code generation, the rewiring, and the sync check are really one edit. What I got wrong: the new C I gate I added was too strict on its very first real day, and I had to loosen it to a warning that same afternoon. About one thousand five hundred fifty net lines changed, across four commits, and the hashes are on the page for anyone who wants to check them. Moving to the section for the twenty-first to the twenty-second of May, twenty twenty six: deleting a dead translation stack, and fixing the real emergency hiding under it.
Checking actual production state, not just the repository, turned up two problems at once. The translator worker had been silently broken for about fourteen hours. And separately, the repository still carried a whole dead translation stack, two Docker services, already removed from production but never deleted from git. What I changed: I made the worker reconnect automatically when its database connection died, and I deleted the dead stack, about three thousand four hundred lines gone. Across the whole two days, the net change came to about six thousand three hundred lines removed.
How I checked it: the plan document, the one in docs slash plans, specifies grepping for zero hits on the old flags, plus a live smoke test watching the system for thirty minutes straight. What I learned: restarting the worker does not fix this. The next idle window just hits the same wall again. What I got wrong: my first reconnect fix was not complete. The very next day I needed a second fix, because the same connection drop also happened partway through the cycle, at the write-back call. And the identical failure showed up again three weeks later, against a different long-held connection, this time in an SEO crawl job instead of the translator worker.
Now to the eighth refactor, the oldest one on the table: switching to a package manager workspace to make room for a mobile app, dated the twenty-sixth to the twenty-seventh of April, twenty twenty six. The repo was still a flat npm layout. That same day I'd decided to build a full-parity mobile app in Expo and React Native, sharing types with the existing web app, and a flat layout couldn't support that. So I switched to pnpm plus Turborepo, replacing a lockfile that ran fifteen thousand two hundred and eighty-five lines, with a net change of about two thousand four hundred and fifty lines.
I kept web, api, and scraper at the root of the repository rather than moving everything into a packages folder. My commit message says all four build tasks were green at the end of that session. The plan I was working from is the mobile v one plan document. What I learned: the original plan called for moving everything into one folder. I reverted that and wrote down why. What I got wrong: the new layout broke the mobile build service's monorepo detection.
It took four follow-up commits the next day: excluding two hundred and thirty gigabytes of local tool state, discovering the exclusion file had to live inside the mobile package itself, discovering that file replaces the standard ignore file instead of merging with it, and re-anchoring the patterns after my first attempt stripped a real source folder out of the build. Four fixes in twenty hours before it worked. Now, the pattern across all eight. Six of the eight start from a written plan document. Most ship a gate that can fail on its own: a corpus gate, a ship gate, a drift checker, a C I check.
Two produced a written postmortem, and several others fold that same habit into the commit message itself. In every one of the eight, something needed a follow-up fix, most within about two weeks, sometimes within a day, once two months later. That's what you find when you look closely enough to write it down. Moving into the second half of this page, titled the engineering behind the record. Those eight refactors are the answer to one question, and what follows is the system they were made in: how the platform keeps its scrapers honest, what fails closed, how personal data is handled, and what I check every morning. Every section cites the repository path it was checked against.
First, how TailHarbor keeps four thousand and three scrapers honest. TailHarbor pulls adoptable animals from shelters across Europe into one searchable site. The hard part was never fetching pages, it's guaranteeing nothing adopted, dead, invented, or private ever reaches a user, while thousands of independent sources drift every day. The scale behind that claim: four thousand and three shelter adapters, forty-three country prefixes, three thousand three hundred eighty-nine commits since the thirty-first of March twenty twenty six, one thousand two hundred fifty-six merged pull requests, seventy schema migrations, and twenty-seven scheduled job modules, all checked against one exact commit the page names directly, as of the third of September twenty twenty six.
Next, one animal record, twelve stages, from a shelter's HTML to the search index. Every night, about four thousand adapters run, each returning a list of animals found on one shelter's site, and something has to decide what's real, new, changed, gone, and what must never be shown. That path lives mostly in the orchestrator file, two thousand five hundred seventy-two lines, twelve stages, and six of them exist only to say no. A slot stage spreads the four thousand three jobs across an eighteen-hour window so they never start together. A sanitize stage strips scripts and bank details from descriptions; a filter stage drops implausible ages and animals missing basic fields.
A contract check fails the whole batch if more than half the records violate the schema, rather than half-loading bad data. And a health stage trips a forty-eight-hour breaker after three hard failures. Last for this segment, five layers between a shelter's HTML and a user's screen. The rules that matter most, no aggregators, no adopted or dead animals, no invented fields, no bank details, are each enforced in more than one place, by code running at a different time than the code it protects. A single gate can be bypassed by accident; five gates on different clocks cannot.
The five: a pre-flight probe before any code runs, a static contract check at commit time through local git hooks, a smoke test against the live site before anything can merge, a sanitize pass inside the orchestrator on every scrape, and integrity sweeps after ingest. The page gives the incident that proves why the redundancy matters: on the third of May twenty twenty six, an aggregator adapter already deleted from git kept scraping every day, because the compiled file had survived on the production host through a sync gap. The lint gate had done its job at build time, and it didn't matter.
The fix was a second check inside the orchestrator, immediately before the scrape call, reading the same blocklist independently. That is the whole argument for five layers instead of one. Moving into the section called one shared filter, forty-one country keys, tested at import time. The rule that matters most: never list an animal already adopted, reserved, or dead. Two hundred and seventy adapters used to hand-write that check on their own, each one drifting differently. Now it's one file, one marker table keyed by country: forty-one country codes across roughly eighteen language families, Austria and Switzerland sharing the German set, self-testing on import so a broken pattern is caught before production. What it proves is defense in depth.
On the third of May twenty twenty six, an aggregator already deleted from git kept scraping daily, because the compiled file had survived on the host. The build gate had done its job and it still didn't matter, so a second, independent check now runs inside the orchestrator itself, right before the scrape call. Two exceptions are deliberately not fail-closed, and both are written down: an unknown country falls back to English markers, and the aggregator check degrades only if its file is missing, which can't happen on a machine that already passed the lint gate or the deploy gate. Next, bank details out of descriptions, and G D P R articles fifteen and seventeen as endpoints.
Shelters paste their own donation account numbers into animal descriptions. An audit on the fifteenth of May twenty twenty six found at least three hundred and thirty-two descriptions across fourteen adapters carrying real bank details into production. The fix is a scrubber that strips IBANs by country pattern and removes the whole clause around them, since a leftover bank name still identifies the account holder. Access under article fifteen streams an export, rate-limited to once per user per day; erasure under article seventeen requires an exact confirmation sentence, then soft-deletes with a seven-day grace period. That deletion's own audit log keys on a free-text identifier on purpose, so it survives the deletion it records.
And a volunteer count of exactly one is reset to zero, because saying one person passed through would itself re-identify them. Third, removal is an explicit act, never a side effect of a broken scraper. On the twentieth of April twenty twenty six, the site lost seven thousand four hundred and twenty one animals in one morning, because a scraper silently returned a quarter of a shelter's listings, which looks identical to mass adoption; the only alarm was the homepage counter. So removal now runs through a guarded status machine: nothing is marked unavailable unless today's yield beats a threshold against that adapter's own thirty-day average, and a failing adapter is skipped by the nightly sweep entirely.
The invariants from that June redesign say a hard failure never opens this gate, and photos and translations are always kept, so delisting stays reversible. But the guard is also sometimes wrong, by its own logic: on the twenty-fifth of August it blocked a real cleanup, because a seventy-seven percent drop in one shelter's count is exactly the signature it exists to catch. The page's conclusion is blunt: a threshold on a delta cannot tell a regression from a correction, since both move the number the same way. Last here, a day in U T C, with the morning cluster ordered on purpose.
Repair runs before disable, disable before the graveyard sweep, and the health snapshot before the canary that reads it: oh three hundred, oh three thirty, oh three thirty-five, oh six thirty, oh six thirty-five. An adapter that could still be revived at three in the morning must not be disabled thirty minutes later. What it proves is that I watch the watchers too: the maintenance agent's own heartbeat check runs on a separate hosted service, not on the machine running the agent, after the agent itself silently stopped firing for three weeks and the gap read as flaky cadence.
One more chapter here is called translation is a pipeline with a gate that can say no, and has. TailHarbor takes descriptions in twenty six source languages and serves them in fifteen. A mistranslated safety line, something like must wear a muzzle, is a real hazard, so the pipeline runs like a regulated process: translation claims are atomic, so concurrent workers on different machines can never grab the same row twice, and there's a separate interlock that catches a different kind of failure, one where nothing crashes at all. Here's why the interlock exists.
On the twelfth of August, twenty twenty six, the serving host restarted running the German model while the drainer's language setting still said Spanish, and for eighty five minutes the German model translated Spanish requests. Nothing crashed, the output read fluently, and must wear a muzzle came back as the Spanish for must wear a thigh. My first estimate was five hundred and ninety eight corrupted rows; a control group brought that down to two hundred and eighteen, all re-queued and re-checked. The fix was an interlock: the drainer now asks the model server what it is actually serving and refuses on a mismatch, every batch, no bypass.
What that proves: a translation can be completely wrong and still look fine, so the check has to sit at the point where the harm happens, not just once at startup. Next, silence is the failure mode I watch for. A job that crashes gets fixed the same day. A job that runs, does nothing, and reports success survives until something expensive happens. On the twentieth of April, twenty twenty six, the site lost seven thousand four hundred and twenty one animals in one morning, and the only alarm was the homepage counter, because nothing compared today's count with yesterday's. That gap is now a daily snapshot and a canary five minutes behind it.
The same blind spot showed up again differently: four hundred and ninety nine adapters were storing nothing at all, even though the found count was over eight thousand animals a day, because the health metric counted what the scraper found, not what survived the filter and got kept. What every case like this proves: success has to be measured by what stuck, never by whether the job finished without an error. Last for you here, every incident becomes a document with a header you can grep.
The repository keeps its own memory in three layers: sixty one solution documents with dated headers, a pattern library of forty seven numbered defects, and a recipe database of seventy nine documents where a fix only counts as proven once it has recurred three times. Pattern number seventeen shows why the numbering matters: a word-boundary bug first seen in a Slovak adapter turned up again three months later inside a shared utility, and because it already had a number, it was recognized on sight instead of re-diagnosed from scratch. What it proves: the next fix starts from the last one, not from zero. Let's move to the next section, called six rules I now build to.
Each one sits next to the incident that forced it, not just an opinion of mine. Rule one: one fact should live in one place, and the point of harm should check it. On the twelfth of August, the target language was recorded in three places, and only one got updated. A German model translated Spanish text for a while, fluently and wrong, must wear a muzzle came out as must wear a thigh. Two hundred and eighteen rows shipped that way. Now the translator asks the model what it is serving and refuses on any mismatch, every batch. Rule two: ask what a green result actually computed.
A production database turned out to be an empty four kilobyte file with no tables, because the driver creates a missing file rather than raising an error. Forty seven scripts had been opening that empty file and reporting success for weeks. Rule three: a health number must count what was kept, not what was found. Four hundred and ninety nine scrapers were discarding over eight thousand animals a day, and every dashboard still read green. Rule four: deletion needs a real death signal, never the absence of success. An automated cleanup proposed removing seventeen scrapers as dead. I checked four sites myself before merging. None were dead; two of the sites returned full pages when I checked them.
Rule five: a score built from the same rules that repaired the training data measures memorization, not quality. A held out test using different wording showed real recall far lower, on the more trained version it fell from about eighty percent to under forty. Rule six: a count of missing pages is not a count of adoptions. A search report once flagged over twenty five thousand pages as not found. Sorted properly, real adoptions were closer to five thousand. Next, a section called the same problems in a manufacturer's vocabulary takes the problems a manufacturer described to me and renames them. Two systems with no clean join becomes identity resolution in a fixed order.
A dashboard that cannot say why something is failing quality control becomes classifying every failure before counting it. Cloud platform consolidation is a real, dated change too, one reviewed pull request in late August. The worked, data-side examples live on the business case page, not this one. Last section, nothing here should be taken on trust. Every count on this page came from a command run against one specific commit, on the third of September this year, and every mechanism points to the file behind it. Writing this, I found two stale references in my own documentation and did not use them, and said so on the page. Where my notes disagreed with the actual files, the files won.
The repository is private, but I am happy to walk through any of these files on a call, or to share read access. The file paths on the page are there for anything you want to check yourself.
The working rules ↑ Contents
What I review myself, and what I let a tool do
A tool can draft a change, propose a repair, sort a backlog or write a first version of a test. It does not get to decide that the change is correct, and it does not get to make anything irreversible. These nine rules are how that line is drawn on TailHarbor, the platform I have built and run alone since March 2026. Each one exists because something went wrong first.
- Nothing is done until it has been run or rendered against the real thing. Reading the diff is not verification. Mechanism: a new source adapter must return at least one animal with photos from the live site before it may be committed (scraper/smoke-test.ts), and the same gate CI runs is runnable locally (scripts/ci-local.sh). The June card rebuild could not ship until a design-drift audit script and a type check both passed.
- Gates fail closed, and they are allowed to say no to me. A gate that can be waived when it is inconvenient is documentation, not a gate. Mechanism: if more than half a batch violates the record contract the whole job fails rather than half-loading (scraper/src/orchestrator.ts:95). In the translation promotion gate, a missing bench file, an under-sampled bench or an unreadable manifest are all BLOCKED, never a soft pass. A whole rewrite failed its own gate in July 2026 and did not ship.
- The rules that matter are enforced twice, by code on different clocks. One enforcement point is one accident away from nothing. Mechanism: on 2026-05-03 an aggregator adapter deleted from git kept scraping daily, because the compiled file survived on the production host through a sync gap. The build-time check had done its job and it did not matter. The fix was a second check inside the orchestrator immediately before scrape(), reading the same blocklist and refusing with reason BLOCKED_AGGREGATOR.
- Checking tools are read-only. They report drift, they never reconcile it themselves. Mechanism: the host-versus-git drift checker compares a 42-row manifest by SHA-256 and exits non-zero on any difference; it never auto-overwrites either side. In the recipe database the markdown is the source of truth, the SQLite index is a cache, and the query tool never writes.
- Nothing destructive happens on the absence of a signal, and the irreversible step keeps a human in it. Mechanism: on 2026-08-22 an automated triage run proposed removing 17 adapters, 4,078 lines, as dead. I fetched four of the sites before merging. Zero were dead and two returned full pages. The pull request was left as a draft and the criterion was changed to require a positive death signal: DNS failure, a sustained 404 or 410, or a parked page. The daily maintenance agent opens fix pull requests rather than pushing, and refuses to act at all when more than 50 adapters break at once. Search-engine URL removals are capped at 180 of the 200 daily quota, leaving 20 for manual work.
- Quality judgement that a machine cannot be trusted to grade goes to a blinded panel, and it is labelled as a panel. Mechanism: the July 2026 pipeline rewrite had a three-part gate, none of it waivable: pollution no worse than the incumbent, safety recall at 97 percent, and a blinded pairwise review of 30 rows per language needing 27 of 30. The blind review scored the candidates 20 of 30 and 18 of 30, a third attempt scored 19 and 20, and the candidates stayed behind the gate.
- A health metric counts what was kept, never what was produced. An empty result from a check that did not run is indistinguishable from an empty result from a check that ran and found nothing. Mechanism: animals_found was assigned before the pre-ingest filter, so a run that found 127 animals and stored none logged SUCCESS. 499 adapters were discarding 8,242 animals a day with every dashboard green. Stored counts now sit beside found counts and a ghost-ingest canary compares the two at 06:50 UTC. The same rule caught a translation metric: recall was scored by the rules that had repaired the training data, and a held-out paraphrase probe dropped it from 82.5 to 66.7 percent on one generation and 80.4 to 38.9 on the next.
- The checker gets checked, from somewhere it cannot take down with it. Mechanism: the maintenance agent's heartbeat check runs on GitHub, not on the machine that runs the agent, after the agent silently stopped firing for three weeks and the gap read as a flaky cadence. A separate dead-man switch in host cron alerts if the health table has no new row for 90 minutes. Two missed heartbeats, not one, count as a fault.
- Every incident becomes a written document, and the plan is written before a big build. The next fix starts from the last one rather than from zero. Mechanism: 61 solution documents with machine-readable front matter, 47 numbered failure patterns cited by id in adapter repairs, 79 recipe documents with a confidence state promoted to proven at three occurrences, 52 plan documents and 19 handoffs. Six of the eight refactors below start from a written plan document.
Counts in this section were produced against repository commit 71324a9e8 on 3 September 2026; production figures were read from the live database on 29 August 2026.
The refactoring record ↑ Contents
Eight refactors, dated
Eight refactors of TailHarbor between April and August 2026, most recent first, drawn from git history and from the plan and postmortem documents I keep alongside the repository. The last column, and the last part of every entry, is what I got wrong.
| Date | What | Size | Verified by | Went wrong |
|---|---|---|---|---|
| 12 to 14 Aug 2026 | Merged 15 per-language translation models into one | ~1,900 net | Corpus gate, per-direction recall | 4 bugs in 48 hours, plus a metric flaw found later |
| 9 to 13 Aug 2026 | Brought the live translation pipeline back into git | ~5,000 net | SHA-256 host-vs-git drift check | The drift checker itself had 2 bugs |
| 28 to 30 Jul 2026 | Rewrote the translation pipeline, then did not ship it | ~31,700 net* | Blind AI panel, safety recall gate | First full training run wasted on a silent bug |
| 10 Jun 2026 | Stopped live animals being falsely marked adopted | ~4,350 net | SQL/log runbook, health dashboard | Same guard blocked a real cleanup 2 months later |
| 4 to 7 Jun 2026 | Rebuilt the animal card as a full-photo overlay | ~3,450 net | Design audit script, tsc, manual check | Layout reworked 4 more times over 3 days |
| 19 May to 1 Jun 2026 | Made one file the source of truth for locales | ~1,550 net | Re-run against the seeded incident | The new CI gate was too strict on day one |
| 21 to 22 May 2026 | Deleted a dead translation stack, fixed a live outage | ~6,300 net removed | Grep for zero hits, 30-min smoke test | First reconnect fix was incomplete |
| 26 to 27 Apr 2026 | Switched to pnpm/Turborepo for a mobile app | ~2,450 net | pnpm turbo run typecheck, all green | Broke EAS Build, 4 fixes in 20 hours |
*The 28 to 30 Jul row is inflated by real training and evaluation fixtures bundled into those commits, JSONL and markdown exports of test fixtures and translated animal descriptions. The pipeline code itself is a small share of that diff. The other rows are close to the actual code change.
12 to 14 August 2026 · Merging fifteen separate translation models into one
What happened
A rewrite had just failed its own quality gate (below), so I fine-tuned one model per language. By 12 August that was 15 models sharing one GPU.
What I changed
Replaced 15 per-language models with one model trained on every locale, both directions, gated by an upfront corpus check.
How I checked it
The corpus gate ran before every launch, with per-direction recall reporting instead of one blended number. No plan document exists for this one.
What I learned
My commit messages that week put the root cause in the commit body.
What I got wrong
Within 48 hours I fixed four bugs in the new code: a gradient-checkpointing bug, a quote check that only understood German punctuation, a gate check that never resolved, and a drift tool meant to guard this exact migration that instead reported everything in sync while production ran old code. None were caught before I merged; all surfaced within a day or two of real use. A later audit found the recall metric I used to justify more training was partly the model memorizing its own repair rules.
Commits
| Hash | Message | Lines |
|---|---|---|
| 38e60aff4 | gate every corpus up front, one orchestrator per language | +333/-4 |
| a03b023d5 | move serving off the shared GPU onto its own card | +15/-2 |
| 4c33b8498 | one language-agnostic model instead of fifteen adapters | +277/-4 |
| 6cc4af13d | ONE model, every language, both directions | +302/-10 |
| 39c7da205 | fix gradient-checkpointing, cover the mixed scorer | +72/-5 |
| c0cfbba76 | quote check rejected non-German quote marks | +126/-7 |
| d49145c82 | report recall per direction, measure reject recovery | +828/-6 |
9 to 13 August 2026 · Bringing the live translation pipeline back into version control
What happened
Three production fixes from 8 and 9 August existed only on the serving host, never in git, and a core verifier there was about 594 lines ahead of the repository. My plan doc says this exact risk had already nearly bitten twice that week.
What I changed
Vendored the production drainer, verifier, and LoRA trainer into the repo, and added a read-only host-vs-git drift checker that never auto-overwrites either side.
How I checked it
A plan document I kept alongside the repository but never committed, docs/plans/2026-08-09-001-fix-petranslate-prod-git-reconciliation-plan.md, specifies SHA-256 hash comparison between host and repo across a 42-row manifest, keeping the verifier in git as a strict superset of production.
What I learned
My own decision record: the repo becomes the source of truth, the host becomes a deploy target.
What I got wrong
The drift checker I built to stop silent prod-and-git divergence had two bugs, and reported everything in sync for four days while the host was actually missing 33 safety rules. An ssh call inside a loop consumed the loop's remaining input, so only the first of many rows was ever checked, and a same-day duplicate-row guard hard-failed the whole check instead. Both bugs failed open, the exact direction the tool existed to stop.
Commits
| Hash | Message | Lines |
|---|---|---|
| b58a3a404 | reconcile git with what production actually runs | +958/-30 |
| 398a17be6 | vendor the petranslate drainer, add replace-mode | +482/-0 |
| e9f2258a3 | vendor the LoRA trainer and its launch/restore scripts | +1648/-0 |
| 415a184e3 | Italian EN-to-target safety rules | +246/-0 |
| 8da48f466 | French EN-to-target safety rules, unblocks FR training | +964/-0 |
| 94ac85b32 | vendor the factory's host-only scripts, add drift checker | +704/-3 |
| 9c40fa13e | drift tooling reported "in sync" while prod ran old code | +6/-4 |
28 to 30 July 2026 · A from-scratch rewrite of the translation pipeline, and the decision not to ship it
What happened
v1's quality had a known ceiling. I built a v2 pipeline that masked entities, chunked sentences, and audited the corpus, with a blind panel deciding whether it could replace v1.
What I changed
Fine-tuned on our own corpus for English to German and Spanish, then ran the ship gate against v1: verdict, does not ship.
How I checked it
A code-based ship gate: benches on the shipped path, safety recall checked against a 97 percent floor, and a blinded pairwise review of 30 rows per language scored against a bar of 27. Write-up: docs/solutions/logic-errors/train-serve-eval-maxlen-chunk-mismatch-2026-07-30.md. No plan document.
What I learned
The first full training run finished "successfully" with no errors and garbage checkpoints. A chunker packed longer chunks than the trainer's old length limit allowed.
What I got wrong
That mismatch wasted my first full training run with no error anywhere; I archived the damaged checkpoints with a note never to bench them. Even after I fixed it, the corrected pipeline still failed its own ship gate on both languages, scoring 20 of 30 and 18 of 30 on the blind panel, then 19 and 20 the next day. I count this as the healthy kind of wrong: the gate did its job and kept a worse pipeline out of production.
Commits
| Hash | Message | Lines |
|---|---|---|
| 5dfb6c5c6 | fine-tune opus-mt on TailHarbor's own corpus (EN-DE, EN-ES) | +11476/-134 |
| eac11e28b | CT2 conversion was broken; stop backfill clobbering rows | +151/-48 |
| c2e89bf9a | integrity verifier, glossary, chunker, frozen-set extension | +3289/-21 |
| cc979e189 | everything before the GPU: masked, chunked, audited data | +9708/-160 |
| 71088b7ff | gate item 2 verdict via blind AI panel, fail both languages | +110/-0 |
| 3ef99fcb5 | gate verdict DOES NOT SHIP, benches, panel, failure dumps | +7349/-41 |
10 June 2026 · Stopping the site from silently delisting animals that are still alive
What happened
An audit found that a single 14-day not-seen clock, meant to catch dead sources, could also flip a real, alive animal to adopted and push a permanent search-engine de-index signal.
What I changed
Added non-destructive guards plus an autopilot that revives disabled adapters before the clock fires, and fixed 410 responses across the site.
How I checked it
My plan doc, docs/plans/2026-06-10-non-destructive-removal-bulletproof-repair-plan.md, specifies SQL and log checks per unit, a sitemap-exclusion check within 30 minutes, and a ten-step removal runbook. A write-up, docs/solutions/scraper-drift/2026-06-10-bulletproof-pipeline.md, documents moving my scraper agents onto launchd.
What I learned
The prior scheduler had lapsed for 9 days because it only ran while an app was open.
What I got wrong
The guard worked as designed. Two months later, on 26 August 2026, that became its own problem: an adapter was found re-ingesting years of closed pages as live listings, 6,003 animals, 4.4 percent of the site's inventory. I fixed the adapter, but the same partial-return-drift guard, built to block cleanup when yield collapses, could not tell a regression from a correction, and blocked the legitimate cleanup indefinitely.
Commits
| Hash | Message | Lines |
|---|---|---|
| 24460f93b | non-destructive graveyard guards, repair-autopilot | +1672/-41 |
| 7460b0e1d | 410 correctness, gone-slugs window/cap, locale + shelter 410 | +150/-10 |
| 64c45f60f | auto_remediate non-destructive guards, URL_DELETED gate | +134/-7 |
| 27e212ad8 | merge: seo-remaining-audit-items | +2522/-76 |
| d0293da5b | clear AnimalTranslation rows on hard-delete, interim | +9/-0 |
4 to 7 June 2026 · Rebuilding the animal card as a full-photo overlay, then tuning it live for three more days
What happened
A visual-drift audit found the live site had diverged from its own design system. The product owner had rejected the current card, and a full-overlay card became the priority.
What I changed
Rebuilt the shared card as a full-overlay, text over the photo, across 26 render sites, keeping the smart-crop system intact.
How I checked it
My plan doc, docs/plans/2026-06-04-001-feat-v4-editorial-web-port-plan.md, required a design audit script and a type check to pass, plus a manual check at 375 and 1280 pixels. My own same-day review caught a real anchoring bug.
What I learned
Three prior reskins had been clobbered by branch merges, so I committed per page inside the branch this time.
What I got wrong
The overlay's layout was not right on first ship. Three days later I made four more commits reworking where the location and gender pills sat, then softening both because the shipped version read too harsh against real photos. None of these were logic bugs, just the same visual call remade against real photos.
Commits
| Hash | Message | Lines |
|---|---|---|
| 94cce4a96 | activate v4-editorial, add drift audit + brief + plan | +3165/-3 |
| 5554b377e | v4 primitives: PhotoBadge, StatsRow, FilterPills, PageHeader | +420/-5 |
| d576197ea | rebuild AnimalCard as full-overlay card | +117/-129 |
| 97e279d9c | re-port Home, Browse, AnimalDetail, ShelterDetail onto v4 | +498/-667 |
| cf49dbd50 | apply code-review autofix feedback | +49/-38 |
| 7ff6eddc2 | location is the map link, gender bottom-right, lower scrim | +81/-33 |
| 579333a3c | soften pills to frosted-glass, low contrast | +18/-14 |
19 May to 1 June 2026 · Making one file the source of truth for every supported language
What happened
Adding a locale meant manual edits in six places, two of them duplicate hand-maintained maps, none of which failed loudly if skipped. Norwegian shipped on 19 May but was never submitted to the search engine.
What I changed
Made the locale registry, one JSON file, the source of truth, and added codegen replacing two hand-maintained maps, with a parity check in CI.
How I checked it
My plan doc, docs/plans/2026-06-01-001-feat-locale-registry-graduation-plan.md, specifies running the new graduation script against the seeded Norwegian incident, a locale that had been live but invisible for weeks.
What I learned
I shipped this as one coordinated pull request, since the codegen, the rewires, and the sync check are one edit.
What I got wrong
The new CI gate itself was too strict on its first real day, and I had to loosen it to a warning that same afternoon.
Commits
| Hash | Message | Lines |
|---|---|---|
| fececb0c7 | graduate Norwegian to web, coverage past DA parity | +592/-35 |
| a88e292ba | registry as single source of truth + graduate-locale.ts | +1184/-210 |
| 2a489e825 | graduate `no` (Norwegian), closes #460 incident | +5/-4 |
| 35ae141d9 | warn instead of fail on errors > 0 for submitted locales | +25/-4 |
21 to 22 May 2026 · Deleting a dead translation stack, and fixing the real emergency hiding under it
What happened
Checking actual production state surfaced two problems at once: the translator worker had been silently broken for about 14 hours because a connection pooler was killing its long-held connection, and the repo still carried a whole dead translation stack already removed from production.
What I changed
Made the worker reconnect automatically on a dead connection, and deleted the dead stack, two Docker services and about 3,400 lines.
How I checked it
My plan doc, docs/plans/2026-05-21-002-refactor-cleanup-dead-translation-infra-plan.md, specifies grepping for zero hits on the old flags, three type-check passes, and a live smoke test over a 30-minute tail.
What I learned
Restarting the worker does not fix this, since the next idle window hits the same wall.
What I got wrong
My first reconnect fix was not complete. The next day I needed a second fix for the write-back call partway through the cycle, where the same drop also happened. The identical failure recurred three weeks later against a different long-held connection, in an SEO crawl job rather than the translator worker.
Commits
| Hash | Message | Lines |
|---|---|---|
| ae1c01775 | remove libretranslate + NLLB zombie services | +27/-3118 |
| 04264b1a3 | reconnect on pgbouncer-induced connection kills | +17/-2 |
| d62ae2c13 | pull prod main.py back into repo, never committed | +261/-115 |
| ace7f165e | mid-cycle reconnect on dead-conn write_back | +40/-7 |
| 4b22f5eed | sync docker-compose.prod.yml, fix two healthchecks | +77/-110 |
| 0db852e2b | delete dead LibreTranslate + NLLB code | +30/-3419 |
26 to 27 April 2026 · Switching to a package-manager workspace to make room for a mobile app
What happened
The repo still used a flat npm layout. That could not support the full-parity mobile app I had decided, the same day, to build in Expo and React Native, sharing types with the existing web app.
What I changed
Switched to pnpm plus Turborepo, replacing a 15,285-line lockfile, and kept web, api, and scraper at the repo root instead of moving them all into packages/.
How I checked it
My commit message states all four build tasks were green at session end, with the verification command recorded in the commit. Plan document: docs/MOBILE_V1_PLAN.md.
What I learned
The original plan said move everything into one folder. I reverted that and wrote down why.
What I got wrong
The new layout broke the mobile build tool's monorepo detection, and it took four follow-up commits the next day to fix: excluding 230GB of local tool state, discovering the exclusion file also had to live inside the mobile package, discovering that the exclusion file completely replaces the standard ignore file instead of merging with it, and re-anchoring the patterns after my first attempt stripped a real source folder out of the build.
Commits
| Hash | Message | Lines |
|---|---|---|
| 6112da5ec | pnpm + Turborepo workspace, packages/shared, packages/mobile | +17634/-15317 |
| 7bb8048db | exclude 230 GB of local tool state and other hidden dirs | +42/-24 |
| a2021eb53 | also place easignore at packages/mobile/ for build detection | +61/-0 |
| ed9289a40 | replicate .gitignore exclusions (.easignore replaces it) | +23/-11 |
| 4eda16f48 | anchor top-level paths, stop stripping mobile/src/api/ | +15/-15 |
| 110ecde63 | unblock build: strip .js extensions, valid config files | +164/-119 |
The pattern
Six of these eight start from a written plan document. Most ship a gate that can fail on its own: a corpus gate, a ship gate, a drift checker, a CI check. Two produced a written postmortem, and several others fold the same habit into the commit message itself. In every one of the eight, something needed a follow-up fix, most within about two weeks, sometimes within a day, and in one case two months later. That is what you find when you look closely enough to write it down.
The commits themselves are available on request; I am preparing a redacted export of the patches and plan documents.
The platform, in mechanisms ↑ Contents
How the platform keeps itself honest
TailHarbor collects animal listings from roughly four thousand independent shelter websites across 40 countries, reconciles about 150,000 records every day and serves 120,000 live listings to the public. Built and operated by one person. The hard part is not fetching pages. It is guaranteeing that nothing adopted, dead, invented or private ever reaches a user while thousands of sources drift every day.
In production, read 29 August 2026
| Measure | Reading | Detail |
|---|---|---|
| Sources | 4,138 | shelters registered in production, 40 countries |
| Polled daily | ~3,500 | one slot each in an 18-hour window |
| Records reconciled daily | 153k | created, updated or retired |
| Live public listings | 120k | tailharbor.eu |
| Images under management | 1.42M | object storage, own lifecycle |
| Job success, trailing week | 99.8% | scrape_jobs, 29 Aug 2026 |
| Shelter adapters in the repo | 4,003 | scraper/src/adapters/*.ts |
| Commits since 2026-03-31 | 3,389 | git rev-list --count HEAD |
| Merged pull requests | 1,256 | gh pr list --state merged |
| Schema migrations | 70 | api/prisma/migrations/ |
| Documented solutions | 61 | docs/solutions/**/*.md |
| Reusable recipes | 79 | scraper/recipes/**/*.md |
| Scheduled-job modules | 27 | scraper/src/cron/*.ts |
Twelve stages, six of which exist only to say no
Every night about four thousand adapters run. Each returns a list of animals it found on one shelter's website. The orchestrator then decides what is real, what is new, what has changed, what has gone, and what must never be shown. The whole path is one file, scraper/src/orchestrator.ts (2,572 lines), with scraper/src/index.ts.
| Stage | Guarantee | Where |
|---|---|---|
| Slot | Each adapter gets a stable minute in an 18-hour window from the hash of its id, so 4,003 jobs never start together and a 6-hour retry still lands inside the window | scraper/src/index.ts:77 |
| runAdapter() | Refuses aggregator domains before any fetch, and refuses shelters marked inactive before creating a job row | scraper/src/orchestrator.ts:417 |
| sanitize() | Strips inline JS, CSS, WordPress shortcodes and form-widget text from descriptions, removes IBANs, and drops a dog breed from any non-dog | scraper/src/orchestrator.ts:664 |
| filter | Rejects ages over 30, call-to-action names, news-like names, names over 80 characters, and any animal with no age, breed, gender or size unless it is already known | scraper/src/orchestrator.ts:845 |
| contract check | Every record is validated against the schema; if more than half of a batch violates it, the whole job fails rather than half-loading | scraper/src/orchestrator.ts:95 |
| upsertDog() | Identity is resolved in a fixed order: shelter plus URL, then the source id when the URL is ambiguous, then a hash of shelter, id and name; the write is an atomic upsert on the composite unique key (sourceUrl, sourceId) | scraper/src/orchestrator.ts:1631, api/prisma/schema.prisma:602 |
| upsertPhotos() | At most 12 photos, deduplicated on a canonical URL, rejected under 200 by 150 pixels, converted to a 1200 px and a 400 px WebP pair in object storage; a partial unique index guarantees exactly one primary photo per animal | scraper/src/orchestrator.ts:2221, scraper/src/utils/imageStorage.ts:77 |
| index | The search document is built from the reconciled record, not the raw one, and shelters that opted out of public listing are purged from the index rather than skipped | scraper/src/orchestrator.ts:2202 |
| yield and stale | Today's count is compared with the adapter's own 30-day average before anything is marked unavailable | scraper/src/orchestrator.ts:1157 |
| health | Three hard failures trip a 48-hour breaker; five empty returns from a previously productive adapter do the same; a run that used over 60% of its time budget grows the budget by 25%, capped at 6 hours | scraper/src/orchestrator.ts:1362 |
| revalidate | A signed request tells the web tier which paths and cache tags to refresh; a failure here is logged, never thrown, so a cache problem cannot fail an ingest | scraper/src/orchestrator.ts:1231 |
- 3 hours per adapter, enforced in the worker with an abort controller because the queue library has no per-job timeout. A janitor runs every 5 minutes and closes anything still marked running past its budget.
- Retries are bounded. A failed run is re-queued once 6 hours later under a job id that includes the day, so an adapter can never retry more than 3 times in a day however many paths lead to a retry.
- Concurrency is 20 workers with per-domain politeness in the HTTP layer rather than a queue-level limiter, so one slow site cannot starve the others.
- Translations and photos are never re-done by accident. A hash of the raw description decides whether the English text is stale; when it is, 15 target-language origin columns are cleared together so every downstream translator re-runs.
- The window is deliberately wider than throughput requires. These are small charities on modest hosting, and a crawler that stays polite stays welcome. About 20 sources are in flight at any moment, at most.
- Extraction is per source, uniform downstream. Roughly three quarters of sources are static HTML, about 800 expose JSON, 584 publish XML or RSS, 279 run on one site builder, 67 need character-encoding conversion, and fewer than 30 need a full browser. Every adapter satisfies the same short interface, so adding a source means writing one file against a known shape rather than touching the pipeline.
- Roughly three sources drift per day out of 3,500 when a site is redesigned and selectors stop matching. That is the background rate of collecting from sites you do not control, a rate to keep pace with rather than a bug to eliminate.
- Normalisation is most of the real work. Forty countries write ages, sizes, sexes and breeds differently. All of it is coerced into fixed vocabularies, 199 canonical breeds and 366 mapped alternative spellings, with locations resolved against a gazetteer of 75,000 cities and a manual override where the automatic result is wrong.
- A representative day is about 900 created, 131,000 updated and 1,000 retired. Runs are incremental, never a truncate and reload, and retirement is by staleness rather than absence in a single run.
Five layers between a shelter's page and a user's screen
The rules that matter most (no aggregators, no adopted or deceased animals, no invented fields, no bank details) are each enforced in more than one place, by code that runs at a different time than the code it protects. A single gate can be bypassed by accident. Five gates on different clocks cannot.
| When | Gate | What it does | Files |
|---|---|---|---|
| Before any code | Probe and pre-flight | Backlog rows are probed for DNS, redirects and content fingerprints before they can be reserved. Pre-flight detects the CMS, aborts on JS-only or firewall-blocked sites, and prints the platform playbook section. Exit 1 means do not build; the target is skipped with a recorded reason | scraper/probe-backlog.ts, scraper/pre-flight.ts |
| At commit | Static contract check | Seven hard rules (class naming, id format, country prefix equals country code, https base URL, no duplicate ids, no aggregator domain) and five warning rules that catch the bugs shipped most often. Pre-commit runs type-checks scoped to the staged workspaces; pre-push refuses direct pushes to main | scraper/lint-adapters.ts (882 lines), .githooks/ |
| Before merge | Smoke test against the live site | A new adapter must return at least one animal with photos from the real site before it may be committed, and the sample is checked for adopted or deceased markers and for CSS, JS or donation-button pollution. CI re-runs the contract check, five type-checks, a design-drift audit, and a real prisma migrate deploy against an empty database | scraper/smoke-test.ts, scripts/ci-local.sh, .github/workflows/ci.yml |
| At runtime | Orchestrator sanitize pass | The aggregator blocklist is checked again before scrape(), so a deleted adapter that survives on disk still cannot fetch. Descriptions lose inline JS, CSS, shortcodes, form-widget chrome and bank details. A dog breed on a non-dog is dropped | scraper/src/orchestrator.ts:550-970 |
| After ingest | Post-hoc integrity sweeps | A description repeated verbatim five or more times inside one adapter's batch is treated as boilerplate and nulled. Animals unseen for 14 days move to adopted, but only if their adapter's yield says the scrape itself was healthy. A 06:35 UTC canary compares today's counts with yesterday's and alerts on a 3 percent drop | scraper/src/cron/ |
Examples of what the static gate rejects: an exported class that does not end in Adapter (auto-registration used to crash the whole scraper); an id or country code mismatch, which silently files a shelter under the wrong nation; a base URL on the aggregator list, including parent-domain suffixes; two files declaring the same id. As warnings: a bare $("h1, h2").first() or $("title") used as the animal name, which shipped 12 or more times and made the site name every animal's name; .text() read without stripping script and style, which put stylesheet fragments in front of users as pet descriptions; a raw fetch() that bypasses the shared rate-limited helper; and a hard-coded constant on gender, size, breed or age, which satisfies the runtime signal gate with data that was never on the page, with 133 single-breed rescues exempted because their constant is true.
One shared adopted filter, 41 country keys, tested at import time
Listing an animal that has already been adopted, reserved or has died is the worst thing this product can do to a shelter and to a family. The rule is written once, in scraper/src/utils/adopted-filters.ts (477 lines), and every adapter calls it. Before that file existed, 270 adapters carried their own hand-written expression, and each one drifted differently.
- One record, keyed by country. The marker table holds 41 country-code keys covering roughly 18 language families; sibling countries share a body.
- Applied to everything, not just the title. The check runs against title, description, categories and status taxonomies together.
- Guards are documented with the incident that caused them. German vermittelt werden ("will be placed") is future tense and must not match; a negative lookahead handles it and the comment cites the issue that found it.
- Self-tests run on import. A fixture set of true and false positives executes when the module loads outside production, so a broken expression is visible on the first local run.
- Archive sections are never crawled at all, rejected by URL path.
- If a fix cannot prove the adopted filter still works, escalate instead of shipping. That is in the repair playbook as a rule. PR 490 exists because a Polish shelter's API has no adoption-status field at all, and the fix had to derive status from a different signal rather than assume "available".
- Two places are deliberately not fail-closed, and both are written down rather than hidden. An unknown country code falls back to the English marker set instead of rejecting everything, and the runtime aggregator check degrades to no protection if its JSON file is missing, because that same file is a hard exit-1 failure at lint time and at deploy time.
Bank details out of descriptions, and data-subject rights as endpoints
Shelters paste their donation IBANs into animal descriptions. An audit on 2026-05-15 found 332 or more descriptions across 14 adapters carrying real account numbers into production. The response was a last-line-of-defense scrubber in the orchestrator, plus a proper data-subject-rights implementation for the accounts side of the product.
- IBAN pattern with a country allowlist. About 70 real ISO country codes, then two check digits, then 12 to 30 characters. The length floor avoids matching 15-character microchip IDs, which are legitimate content.
- BIC and label patterns, plus the clause that follows each.
- Whole clauses go, not just digits, because a dangling bank name still identifies the account holder.
- Sixteen further passes strip legal notices, cookie banners, opening hours, phone numbers, emails, street addresses, URLs and glued form labels before the text is stored.
| Right | Implementation |
|---|---|
| Access (Art. 15) | GET /api/portal/gdpr/my-data, streamed in chunks, rate-limited to one export per user per 24 hours, and it excludes its own export events so the file cannot grow itself |
| Erasure (Art. 17) | POST /api/portal/gdpr/self-erasure requires the literal confirmation sentence, soft-deletes with a 7-day grace period, and scrubs edit attribution to [redacted] in one transaction |
| Audit trail | GdprAuditLog keys on a free-text user id on purpose, so the log row survives the deletion it records |
| Volunteer retention | 24-month retention, then a 7-day hard-delete grace, swept daily in two passes. Erasure runs under an advisory lock and keeps lifetime counters, except that a count of exactly 1 is set to 0 because "one person passed through" would itself re-identify them |
| Queues and monitoring | Job payloads carry only an id, never personal data. The error monitor's beforeSend strips obvious personal data |
Source: api/src/routes/portal/gdpr.ts:22 · api/src/middleware/rate-limit.ts:244-251 · api/prisma/schema.prisma:1939 · api/src/services/volunteerGdpr.ts:15 · api/src/instrument.ts
Removal is an explicit act, never a side effect of a broken scraper
The dangerous transition in this system is not adding an animal. It is removing one. A scraper that silently returns a quarter of a shelter's catalogue looks exactly like a shelter that adopted out three quarters of its animals. On 2026-04-20 the site lost 7,421 animals in one morning that way, and the only alarm was the homepage counter. Status moves AVAILABLE, then UNAVAILABLE after 7 unseen days, then ADOPTED after 14, and both forward transitions are refused while the run or the adapter looks suspect. The return path costs nothing because nothing was deleted.
| Signal | Threshold | Effect | Where |
|---|---|---|---|
| Run yield | Today's count under 70% of the adapter's own 30-day average of successful runs, after at least 7 prior runs | The run is suspect: no animal is marked unavailable | orchestrator.ts:217 |
| Adapter yield history | last_yield_pct under 0.7, or one or more consecutive low-yield returns | The nightly sweep skips that adapter's animals entirely | auto-graveyard-stale-animals.ts:88 |
| Circuit state | Adapter disabled, or 3 or more consecutive failures | Same: skipped by the sweep | same file |
| Stale clock | Unseen 7 days: UNAVAILABLE. Unseen 14 days: ADOPTED | Only when no guard above is active | orchestrator.ts:2525 |
Invariants the June 2026 redesign wrote down: a hard failure never opens the graveyard gate (G1); a disabled adapter never causes its animals to leave the index, so disabling for repair is reversible and non-destructive (G3); translations and stored photos are retained across any delisting, so the graveyard is always reversible by re-upsert with no translation re-cost (G8); photo objects are never deleted inside the window where a database row might still come back (G9).
The guard is correct, and it is also wrong. On 2026-08-25 an adapter fix corrected a German rescue whose adopted filter had only ever checked the page title. A live sample of 210 pages showed about 92 percent already carried an adopted or deceased category; the shelter was showing 6,003 available animals, 4.4 percent of the whole site. The fix brought the scrape from 6,004 to about 1,350 real animals. Twenty-four hours later the count was still 6,006, because a 77 percent yield drop is exactly the signature the guard exists to block. The written conclusion: a guard keyed on a delta cannot distinguish a regression from a correction, because both move the number the same way. The documented options are to carry intent through the fix, to verify against the source by sampling the vanished pages, or to add an operator override with an audit trail. The one option ruled out is lowering the threshold, because the opposite failure has already happened and costs more.
A day in UTC, with the morning cluster ordered on purpose
Repair runs before disable, disable runs before the graveyard, and the health snapshot runs before the canary that reads it. The order is the design: an adapter that can be revived at 03:00 must not be disabled at 03:30 or have its animals retired at 03:35.
Job inventory with thresholds and files (27 scraper modules, plus API and host jobs)
| UTC | Job | Threshold or rule | File |
|---|---|---|---|
| 00:05 | Reset retry budgets | todaysAttempts back to 0 for the new day | cron/reset-daily-attempts.ts |
| 03:00 | Repair autopilot | Revives up to 20 transiently disabled adapters per run, emails a drift digest, writes a prioritised repair queue to Redis (25 h TTL) | cron/repair-autopilot.ts:518 |
| 03:00 | Database ANALYZE | animals, animal_photos, shelters, so planner statistics survive constant ingest | ops/db/README.md:38 |
| 03:30 | Auto-disable stale adapters | 14 empty returns, or 5 empty and unhealthy for 30 days, or 7 failures; disabled for 90 days | cron/auto-disable-stale.ts:10 |
| 03:30 | Database dump to offsite storage | Daily; prune 03:40; secrets bundle Sunday 03:45; restore test Monday 04:15 | ops/backup/cron.d/ |
| 03:35 | Graveyard sweep | AVAILABLE and unseen 14 days becomes ADOPTED, skipping suspect adapters | cron/auto-graveyard-stale-animals.ts:60 |
| 03:40 | Prune orphan photos | Rows with no stored file older than 7 days | cron/prune-orphan-photos.ts:63 |
| 04:00 | Vision debt rollup and storage snapshot | Per-adapter photo-quality score; adapters under 0.7 counted as tripped | cron/vision-debt-rollup.ts:46 |
| 04:15 | Search-engine URL removals | Capped at 180 of the 200 per day quota, leaving 20 for manual work | ops/gsc/cleanup-404s.cron:24 |
| 04:30 | Stale photo cleanup | Modes off, observe, dry-run, enforce; enforce ceiling 2% of photos per day for the first week, then 0.5% | cron/cleanup-stale-photos.ts:40 |
| 05:00 | Retry photos with no stored file | Bounded batch of 5,000, three attempts each | cron/retry-null-stored-photos.ts:36 |
| 05:30 | Data drift detectors | Schema versus live database, table bloat (vacuum at most 2 tables per pass) | cron/data-drift-detectors.ts:52 |
| 06:30 | Health snapshot | One row per day: animals, shelters, fresh in 24 h, stale, broken adapters, stuck jobs | cron/daily-health-snapshot.ts:25 |
| 06:35 | Health canary | Alerts on animals down over 3%, fresh-24h down over 10%, broken up by over 20, or over 100 stuck jobs | cron/daily-health-canary.ts:70 |
| 06:50 | Ghost-ingest canary | Adapters that found animals and stored none; baseline 498, alert at 10% growth | cron/ghost-ingest-canary.ts:49 |
| 07:33 | SEO auto-remediation | Uses the same 180 per day quota cap; 26-hour lookback to survive cron skew | ops/seo/cron.d-tailharbor-seo:42 |
| 08:00 | Daily maintenance agent | Diffs the broken set against yesterday, opens fix PRs, refuses to act when over 50 adapters break at once | scraper/maintenance-agent.md:323 |
| 09:00 | Email dead-letter check, token age, removal pushes | 5 failed emails in 24 h alerts; social token warns 10 days before expiry | cron/email-dlq-check.ts:212 |
| every 5 min | Stuck-job janitor | Closes jobs running past their budget (3 h default, up to 6 h), re-queues within the 3 per day limit | cron/janitor.ts:26 |
| every 30 min | Heartbeat watchdog | Every daily job has a 26-hour SLA; two consecutive misses alert, throttled to one alert per day | cron/heartbeat-watchdog.ts:54 |
| every 30 min | Sitemap precompute | Writes XML to Redis with a 25 h TTL after a deep-offset render caused a 503 cascade | scraper/src/index.ts:300 |
Not in the table: materialised-view refresh every 5 minutes, cache warming every 4 minutes, push notifications every 15 and 30 minutes, and a weekly cycle (SEO debt scoring Sunday 02:00, Core Web Vitals Sunday 04:13, shelter digest Sunday 09:00, backup restore test Monday 04:15).
Who watches the watchers. The maintenance agent's own heartbeat check runs on GitHub, not on the machine that runs the agent, after the agent silently stopped firing for three weeks and the gap read as a flaky cadence. A separate dead-man switch in the host cron alerts if the health-check table has no new row for 90 minutes. Every daily job reports a heartbeat, and the watchdog treats two missed heartbeats, not one, as a fault, so a single slow morning does not page.
Five months in four charts, and the constraint that was not the pipeline
The metrics store went in at the end of August 2026 and was backfilled to the first day the platform ran. The first thing it showed was a story nobody could see while it was happening, because the instrumentation to see it did not exist yet.
| Period | Runs/day | Failures/day | p99 latency | Runs per source |
|---|---|---|---|---|
| 2 to 19 Apr, build-out | 1,798 | 72 | 1,654s | ~1.3 |
| 20 Apr to 13 May, strain | 4,472 | 261 | 2,160s | 1.5 to 4.0 |
| 14 May onward, stable, 108 days | 3,345 | 54 | 493s | 1.01 |
The full investigation: the growth, the two culprits, the two fixes, and the variance proof
The fleet grew from 102 sources to 3,884 in nine weeks. What you would expect: throughput climbing and failures climbing with it. What you would not: the failures were not concentrated in the new sources. Across the strain period, newly-added sources failed on 8.9% of runs and long-established ones on 5.4%. The whole fleet was struggling at once, which points at a shared constraint rather than at any individual source.
The second signal named the mechanism. Each source is polled once a day by design. During the strain period the ratio of runs to distinct sources sat between 1.5 and 4.0. Those extra runs were retries, and retries are both symptom and cause. At worst, on 10 May, the 99th-percentile run took 6,250 seconds. One run in a hundred was taking over an hour and a half.
The constraint was memory, and the pipeline was not the thing consuming it. On 4 May a machine-translation model was deployed into the same stack as a sidecar with a full deep-learning runtime behind it. On 11 May the languages it served were expanded from five to twenty-five. That same day the sidecar ran out of memory mid-translation, died, and left a zombie process behind while the worker went on requesting translations from something that was no longer there. Roughly two hours of zero throughput, and the worst day anywhere in the data: an 18.7% failure rate against a normal 5%. Separately, the public website was rendering its XML sitemaps on demand, and that render ran more than fifty slow queries against the same database at the moment a search engine asked for it.
Two fixes landed two days apart. On 11 May the translation sidecar was put under a supervisor that reaps and respawns it rather than letting a dead process masquerade as a live one. On 13 May the sitemap render moved to a precomputed cache, rebuilt every thirty minutes and served from memory, which took response time from 8 to 30 seconds down to under 15 milliseconds and turned unbounded on-demand load into bounded scheduled load. Failures fell from about 380 a day to 40, a factor of 9.5, and the 99th-percentile run time from 4,338 seconds to 463. Runs per source dropped to 1.01 and has never since exceeded 1.05. Median run time settled around 12 seconds.
The measurement that made the diagnosis convincing was variance, not the average. Before, the failure rate swung between 0.7% and 20.2% depending on the hour, standard deviation 4.7 points. After, it sat between 0.3% and 4.8%, standard deviation 1.3. Hour-to-hour variance collapsed by a factor of 3.7, which is the specific signature of removing contention for a shared resource. Code that is simply badly written fails at a steady rate all day.
The theme did not end there, which is the honest part. The translation model was removed from the production stack entirely on 22 May. In June the same constraint resurfaced: a confirmed out-of-memory kill of the scraper on 11 June, and the discovery that the cache's container memory limit had been set below its own internal limit, so the kernel would kill it before it could evict anything, taking the job queue and the sitemaps down with it. Both ceilings were raised the following day. The real finding is that a platform running many workloads on one machine has memory as its binding constraint, and the component that fails is frequently not the component causing the problem.
Translation is a pipeline with a gate that can say no, and has
Descriptions arrive in 26 source languages and are served in 15. A mistranslated safety sentence ("must wear a muzzle", "not with children") is a real-world hazard, so the translation system is built like a regulated process: atomic claims, an interlock at the point of harm, a promotion gate with numeric floors, and evaluations designed to catch the metric lying.
- Work is claimed atomically. UPDATE … FOR UPDATE SKIP LOCKED stamps an origin column so concurrent workers on two machines never translate the same row twice, and stale claims are released after 15 minutes. Ten target languages are marked bench-passed in the locale registry and are live; sixteen are not and cannot be drained by accident, because the registry is the only source of that fact.
- The interlock, and the incident that built it. On 2026-08-12 the serving host was restarted with the German model while the drainer's language setting still said Spanish. The German model translated Spanish requests for 85 minutes. Nothing threw. The output was fluent. "Must wear a muzzle" came back as "debe llevar un muslo", which is a thigh. The first estimate was 598 corrupted rows; a control group corrected it to 218, all re-queued and re-checked. The drainer now asks the model server what it is serving and refuses on mismatch, re-checked every batch rather than only at startup. There is no bypass flag, and a test proves the one escape hatch cannot silence a real mismatch.
- The promotion gate has floors, not opinions. Safety recall 97.0%; a would-fail ceiling of 5.0% against a live Spanish rate of 2.2%; 200 rows minimum, after a 200-row bench once picked the wrong model and was reversed; 100% of the reference locale's fact categories, so a language with no safety rules cannot pass. A missing bench file, an under-sampled bench or an unreadable manifest are all BLOCKED, never "probably fine". A pass writes status=gated and nothing else; promotion is a separate, deliberate script. The manifest records Italian's first attempt as FAIL at 89.3 percent.
- Safety rules as code. Two deterministic rule engines patch what the models miss: 25 source-language rule sets for the source-to-English leg (2,702 lines) and 15 target languages times four rule families for English-to-target (2,590 lines). When four languages first shipped with no rules, a measured 6 to 40 percent of safety facts were dropped; thirteen rules per language brought that to 0.8 to 2.3 percent.
- Production matches source control, checked by hash. A 42-row manifest of git path and host path pairs, compared by SHA-256, exiting non-zero on any drift.
- Training discipline. Checkpoints are split into resume checkpoints (unconditional, never promotable) and the quality-gated best checkpoint (the only promotable artefact), after a run died 22 GPU-hours in with nothing banked. Evaluation history is written beside the best checkpoint, not inside it, because the best directory is overwritten wholesale on every save. Training refuses to start when more than one GPU is visible, naming the intended card, after a smaller card on the same machine silently absorbed a job and ran out of memory three minutes in. A long run is refused until the PCIe link has been proven under load, not merely idle.
Catching the metric lying. Training data was repaired by rule R, and recall was scored by rule R. A paraphrase probe with the same facts in different wording collapsed recall from 82.5 to 66.7 percent on generation 2 and from 80.4 to 38.9 on the more-trained generation 3, and that gap is now a promotion-gating number. Fabrication was free: the model appended canned safety sentences for facts the source never stated and they shipped with a passing score, so fabricated fallbacks are now rejected. One null field disabled 40 percent of the rules: every row in the frozen evaluation set had a null animal type, so 13 of 30 German and 12 of 30 Spanish rules, including muzzle and bite history, could never fire, for months, while recall read 85 percent. And the bench must run the shipping prompt: an ad-hoc prompt scored 93 percent where the real template scored 71, so a CI parity test now asserts the benchmark's copied functions are identical to the drainer's. On its first run it found 6 of 8 had already drifted.
Silence is the failure mode I watch for
A job that crashes gets fixed the same day. A job that runs, does nothing, and reports success survives until something expensive happens. Most of the instrumentation in this repository exists because of a specific case of the second kind, and each case is written down.
How the largest one was found
In August 2026 records marked stale jumped from 9,355 to 14,418 in one day, after a week flat around 9,000. No run had failed anywhere. The path from the number to the mechanism is the transferable part:
- Attribute it before explaining it. Grouping the newly stale records by source showed one source holding 4,661 of them, 92% of the entire movement. A fleet-wide worry became a question about one source.
- Read its history rather than its status. That source had returned 7,963 records on 10 August, about 5,950 mid-month, and 1,326 by the 25th. Every one of those runs reported success.
- Establish what the alarms were actually watching. Consecutive failures: zero. Consecutive empty responses: zero. The source was answering politely with a fifth of its inventory, and no failure-based check can see that.
- Check whether it was one source or a class. It was a class: 336 sources had returned under half their own 30-day average on their most recent run.
- Find the mechanism. The count of records found was recorded before the validation step discarding them. A source that extracted 127 records and stored none recorded "success, 127 found".
- Size it honestly. About 500 sources were in that state, discarding roughly 8,200 records a day, and none appeared in any broken-source count anywhere in the system.
What changed, in order: first a count of what was actually stored, recorded next to the count of what was found, so the discard became visible and any later fix could be proven. Second, the cleanup job was stopped from retiring records belonging to sources in that state, because their data was being dropped by our own validation rather than removed by the shelter. The recovery is being done in batches rather than all at once, because a subset of those sources also publish animals that have already been adopted and there is no filter for that yet.
The change that was deliberately not made. The obvious next step is to feed the new count into the existing health system. That would have marked all 500 sources unhealthy, and a separate job disables anything unhealthy for two weeks running, so 500 sources would have gone dark within a fortnight and a monitoring improvement would have become a much larger outage. The arithmetic is recorded at that branch in the code so the improvement is not made later by someone without the context.
The catalogue
| What went silent | Why nothing noticed | How it is seen now |
|---|---|---|
| 7,421 animals gone in one morning (2026-04-20) | No metric compared today with yesterday | Daily snapshot at 06:30 and a canary at 06:35 with four thresholds; alerts by email, an audit-log row, and a Redis key the dashboard reads |
| Thirty-one shelters frozen the day a new gate landed (2026-05-02) | The count drop was under 3% and every run reported success | A per-shelter detector: 10 or more animals with 80% unseen for 5 days |
| The canary under-reported its own blast radius for 5 days | One query with LIMIT 50 served both the sample and the total | Split into an unbounded aggregate and a bounded sample (fixed 2026-07-12) |
| 499 adapters storing nothing, 8,242 animals a day | The health field counted animals found, before the filter that dropped them | Ghost-ingest canary at 06:50 comparing found with stored; a worklist report; a heartbeat for the agent, hosted off the machine |
| 443 adapters healthy on every metric with zero animals ever stored | Yield measured what the adapter returned, not what was kept | Classified into duplicate coverage (32), no signal in code (296), has signal but drops (113), orphan row (1); two adapters found faking constants were flagged, and a lint rule now catches that |
| The scrape-target database on production was a 4,096-byte file with no tables | The driver creates a missing file; 47 scripts computed 4 different paths | One path resolver, an open helper that requires the file and asserts the table, unified deploy excludes, and a real test runner (25 files, 357 tests) |
| The local CI mirror ran zero checks on some pushes | Inside a command substitution, set -e only inspects the last command, so a failed diff was masked | The substitution was split, and the quick mode, which had run zero tests, now always runs the fast unit suite |
| The dashboard said 109 adapters were broken | "Broken" was a symptom bucket, not a diagnosis | A triage wave classified them: about 40% were code bugs, the rest dead sites, firewall blocks, a hacked site and stale rows. Triage is now classification-first |
| Metrics endpoint reachable from the internet | A catch-all proxy location would have exposed it behind only an app token | An explicit deny-all block for the internal path, flagged for human review because it is infrastructure |
The dashboard is a worklist, not a verdict. The admin surface classifies every adapter from its last ten job rows into broken, degraded, healthy or not scheduled, in one SQL round trip, and shows service-level buckets with targets: hard-failed (target 3), selector drift (target 10), stale-zero (target 5), photoless share (target 1 percent). Administratively disabled adapters are excluded from the broken count so an open circuit breaker cannot inflate it, and an adapter that timed out but succeeded within the week is counted as slow rather than broken, because a live probe on 2026-05-11 found 116 of 162 "failed" adapters were exactly that.
Health is now one question: did the source change the database in the last 24 hours? The system previously carried seven overlapping definitions of a failing source. Queried together they read anywhere from 12 to 500, because each answered a different question and nothing on screen said which. One of them counted only sources still being scheduled, so its value changed on its own as sources entered and left cooldown. The single output-side question subsumes every earlier definition, because a hard failure, an empty response and a silently discarded batch all produce the same observable result, and it cannot be defeated by a counter recorded at the wrong point in the pipeline. The reading on 29 August 2026: 2,964 of 3,533 sources producing data, 569 producing nothing, reported as a count rather than a percentage because a ratio flatters a growing denominator, against a target of under 100 set at what a healthy fleet looks like rather than at last week's average.
Every incident becomes a document with a grep-able header
The repository keeps its own memory in three machine-readable layers, so the next fix starts from the last one instead of from zero.
- 61 solution documents with YAML front matter (module, date, tags, problem type, and for scraper drift also component, severity, symptoms, root cause and resolution type), so a question like "what do we know about the translation engine" is one grep. By category: ops 18, seo 10, mobile 9, logic-errors 8, scraper-drift 8, root 3, best-practices 2, and one each for design patterns, integration issues and performance. The ten most recent titles read like a list of ways a check can pass without checking: a two-part gate that reported PASS after one part finished; a null field that disabled 40 percent of the rules; a metric built from the rules that repaired the corpus; a checker that shared an assumption with the thing it checked.
- 47 numbered patterns cataloguing how shelter websites break scrapers, each with a detection signature, the fix shape, and the adapters where it was seen. Numbering is the point: a word-boundary bug on accented characters first seen in a Slovak adapter recurred in a shared utility three months later, and the fix was recognised by id, applied centrally, and appended to the same entry.
- 79 recipe documents: 11 adopted-filter recipes (one per language), 57 build recipes across 25 platform and country folders, 9 triage recipes, 1 description-sanitisation recipe, and an index. Each has an id, a type, a platform or symptom fingerprint, an occurrence count, and a confidence state that is either provisional or proven, promoted to proven at three occurrences in code rather than by opinion. Drift is reported in three classes: recipes no adapter uses, repeated code in three or more adapters with no recipe, and recipes whose selector no longer appears in any of their reference adapters.
- 52 plan documents and 19 handoffs. Plans follow a dated naming convention (43 of them strictly), handoffs record the state at the end of a working session, and the maintenance agent appends one row per run to its own log. A plan is reviewed before a large build starts.
Where it is fragile
| Area | Position |
|---|---|
| Discarded records | Measured and contained, not yet recovered. Recovery is staged in batches for the reason described above. |
| Blocked reads as empty | When a site serves an anti-bot challenge, some sources record it as "nothing listed here" rather than "we were blocked". Those two need to look different and currently do not everywhere. |
| Single points of failure | One server, one database, no standby. Email is the only alerting channel. Conscious tradeoffs for a one-person system, and the first things to change with a team. |
| Metrics store is new | It went in at the end of August 2026, backfilled to day one. Before that, timing statistics lived in process memory and reset on restart. The store itself has only days of operational history. |
A second pipeline, in Python. Alongside the collection platform there is a warehouse pipeline: roughly 16,000 lines of Python with test coverage, mirroring BigQuery datasets into PostgreSQL daily under a service account, feeding twenty reporting tables.
Its failure worth describing: a region default emptied a table, and thirteen analyses reported on it anyway
A daily report announced a traffic collapse that had not happened. BigQuery resolves queries against a region and defaults to the United States when not told otherwise. The dataset is resident in Europe, so every query looked on the wrong continent, found nothing, and errored. The copying job recorded the error and continued, leaving the destination table empty. The complication is what makes it instructive: the same pipeline also reads a public dataset that genuinely is resident in the United States, so a single connection-level setting could not serve both, and the region had to be specified per query. The larger problem was downstream. Thirteen separate analyses depended on that table and none checked whether it contained data. They ran against nothing and produced confident, plausible, entirely fictional results. Fixing the region fixed the data. It did not fix the fact that thirteen analyses would happily report on an empty table, so each now refuses to run when its own source is stale.
Nothing here should be taken on trust
Every count above was produced by a command against commit 71324a9e8 on 3 September 2026, and every mechanism cites the file that implements it. Production figures were read from the live database on 29 August 2026. The repository is private, but I am happy to walk through any of these files on a call, or to share read access.
Two figures in the repository's own documentation are stale and were not used: the project guide still says 486 adapters, and a solutions document referenced elsewhere does not exist in the tree. Where a number changed between my notes and the files, the file won. Two cited documents are kept beside the repository but are not committed on main; they are marked as such where cited. The platform is a Next.js web app, an Express and Prisma API on PostgreSQL, a BullMQ scraping fleet, Meilisearch, and a translation engine with locally trained models, deployed with Docker Compose on a single Hetzner host, with a BigQuery metrics store added in August 2026.
The other column ↑ Contents
What I got wrong
Collected from the refactors and the incident record above, in one place, unsoftened.
- Four bugs in 48 hours in the model consolidation, none caught before I merged: a gradient-checkpointing bug, a quote check that only understood German punctuation, a gate check that never resolved, and a drift tool meant to guard that exact migration that instead reported everything in sync while production ran old code.
- The recall metric I used to justify more training was partly the model memorizing its own repair rules. A paraphrase probe dropped it from 82.5 to 66.7 percent, and from 80.4 to 38.9 on the more-trained generation.
- The drift checker I built to stop silent divergence had two bugs and both failed open, the exact direction the tool existed to stop. It reported everything in sync for four days while the host was missing 33 safety rules.
- A silent chunk-length mismatch wasted my first full training run, which finished with no errors and garbage checkpoints. I archived the damaged checkpoints with a note never to bench them.
- The corrected rewrite still failed its own ship gate on both languages, scoring 20 and 18 of 30 on a blind panel, then 19 and 20 the next day, and missing recall at 96.7 and 93.2.
- The removal guard blocked a legitimate cleanup for months. It could not tell a regression from a correction, because both move the number the same way.
- The overlay card was not right on first ship, and took four more commits over three days to settle.
- The new locale CI gate was too strict on its first real day and had to be loosened to a warning that same afternoon.
- My first reconnect fix was incomplete. A second fix followed the next day for a mid-cycle write, and the identical failure recurred three weeks later on a different long-held connection.
- The workspace switch broke the mobile build's monorepo detection, taking four follow-up commits in 20 hours.
- I let a translation drainer run for 85 minutes against the wrong model. Nothing threw, the output was fluent, and 218 rows reached production fluently wrong.
- I shipped a health metric that counted what was found rather than what was kept, so 499 adapters discarded 8,242 animals a day with every dashboard green.
- Production's scrape-target database was a 4,096-byte file with no tables for weeks, because the driver creates a missing file and 47 scripts computed four different paths. Two defects hid each other.
- An automated triage run proposed deleting 17 adapters as dead and none of them were. I only found out because I fetched four of the sites before merging.
- A frozen evaluation set had a null field that silently disabled 40 percent of the safety rules for months, including muzzle and bite history, while recall read 85 percent.
- The canary under-reported its own blast radius for five days, because one query with LIMIT 50 served both the sample and the total.
- My local CI mirror ran zero checks on some pushes, because inside a command substitution set -e only inspects the last command.
- I lost 7,421 animals in one morning and the only alarm was the homepage counter.
- I treated 25,254 "not found" URLs as a problem of scale rather than of classification at first. Bucketing showed real adoptions were about 5,000 and the rest were ghosts from a partial-return bug, with one adapter returning 26 percent of a catalogue with zero failures recorded.
- I let a single machine's memory become the binding constraint and did not see it until the metrics existed, including a container memory limit set below the service's own internal limit so the kernel killed it before it could evict anything.