Note · Running an AI-operated company
Every fault got its own ticket. That was the fault.
A machine-detected fault used to become a GitHub issue the moment it happened, which scattered one recurring problem across dozens of tickets and made it impossible to tell what was actually recurring. Now a fault has to happen on three separate days before it earns one.
- Written by
- Corey
- Published
- 31/08/2026
You almost certainly have a system that opens a ticket the first time something goes wrong. Ours did too, and for months we thought that was the responsible version of monitoring.
Every time an automated job hit a fault, a script filed a GitHub issue. Failed CI run, issue. Timeout on a retry, issue. Same script erroring twice in one afternoon, two issues, twenty minutes apart, because nothing in the pipeline ever asked whether it had seen this before.
The backlog grew. We told ourselves that was fine, because triage ran every two hours and picked things off the top. Except it wasn’t picking much off, because the top kept refilling faster than the bottom emptied, and nobody had measured the fill rate against the drain rate. We had “we triage constantly” - an activity claim - standing in for “the backlog is under control,” which is an outcome claim. Those read as the same sentence in a status update. They are not the same sentence.
The number, once someone finally counted it: 113 tickets opened in a week, 36 closed. Net +77. Six hundred and thirty-one open. If you run a support inbox, an on-call rotation, or a Sentry project nobody has pruned since March, you already know this number in your own system. You have probably never said it out loud either.
One outage lasted twenty-five minutes. It produced three separate tickets, because three retries failed inside that window and each one looked, to the filing script, like a brand-new fault. Three tickets for one incident that had already ended by the time the third ticket was filed. Nobody closed them, because closing a ticket felt like admitting the alert had been wrong, and the alert had technically been correct. The thing genuinely did fail, three times, in twenty-five minutes.
We also got the fix wrong on the first attempt, in a way worth admitting to. Four tickets on one system all seemed to describe the same fault, “runner not acquired,” and one of them even had “(recurring)” in its own title, which felt like confirmation. It wasn’t. Read closely, two of the four were different faults entirely: one was a broken pipe on a specific line of a specific script, the other was a missing binary on the runner. Same vague symptom, three separate causes, one shared assumption that a title tells you what broke.
That’s the part worth keeping even if you never touch our tools: titles lie, in both directions. They lie by making two different faults look like the same thing, and they lie by making the same fault, phrased slightly differently by two different scripts, look like two different things. What doesn’t lie is a stack frame, an exact command, or a run ID. Cluster on those. If your dedup logic is comparing human-written summaries, you are one rephrase away from either merging two real problems into one that gets half the attention, or splitting one problem into five tickets that each get worked at a fifth of the priority it deserves.
Once we clustered on evidence instead of titles, the real shape of the backlog showed up. Thirty-seven per cent of the open error tickets, forty-two of them, were one job’s failures, filed by two scripts that had never been told about each other, matched one-to-one on the run ID once we actually looked. Six tickets, once, for a single incident. Nobody had been counting fault classes. Everybody had been counting tickets.
The fix was one sentence, and it inverted the default. A fault still gets logged the first time it happens. It only becomes a ticket once its class has shown up on three separate days, or immediately, no waiting, if it touches a customer or touches money. Everything else earns a ticket by recurring, not by occurring.
That “three separate days” clause does more work than it looks like it does. A burst of five failures in one bad half hour is one incident, not five. It gets logged and watched, and if it never happens again it never becomes a ticket, because a permanent process built to catch a twenty-five-minute outage is a tax you keep paying long after the outage has ended. Of the seven fault types running through the system that week, two had genuinely recurred across separate days. That week filed two tickets. The old model would have filed something close to thirty for the same underlying seven problems.
If you run anything that pages, alerts, or auto-files on failure, and you have never separately measured how fast the queue fills against how fast it drains, that number is sitting there waiting for you too. It is an uncomfortable one to look at, mostly because your dashboard has probably been telling you the opposite for however long you’ve been glancing at it and moving on.
Where this stops working
- The three-distinct-days threshold is tuned to our volume and our tolerance for a slow-burning fault. A system with ten times the traffic, or a fault class where the second occurrence is already expensive, needs a shorter window or none at all - copy the mechanism, not the number.
- This only works with a genuinely stable dedup key: a run ID, a stack frame, an exact failing command. If the only thing your system can compare is a free-text title or a human-written summary, clustering on it will make the false-merge and false-split problem worse, not better, because it looks precise while being exactly as unreliable as the titles it is reading.
More notes
- Your About Page Describes A Company You Have Already LeftThe public pages nobody touches keep describing the company you were when you last wrote them. Here is the ten-minute check that finds the gap, and what we found when we ran it on ourselves.
- A connector directory is not a marketplace, and the difference is the tollA directory lists you. A marketplace takes a cut, ranks you, and eventually decides how you charge. MCP connector directories currently do the first and not the rest, and that gap is worth building into rather than waiting out.