T1 triage bot

Bug triage across the digital org's scrum teams ran on a human rotation: someone watches the channel, someone eyeballs severity, someone chases the SLA. Manual, slow, and inconsistent — the same ticket got a different priority depending on who was on duty that week. The cost wasn't just time; severity, the thing the whole SLA hinges on, was a coin flip weighted by whoever was on shift.

It looked like a staffing problem. It was an information problem. Everything needed for a severity call already lives in the ticket system — the bot just has to read it the way the system intends: programmatically, through the API, with the ticket system as the source of truth. No pasting ticket text into a chat window.

The tempting first build was a chat assistant: copy the ticket, paste it, ask for a severity. It would have demoed well and rotted fast — pasted blobs lose structure, drift out of sync with the ticket, and can't be a system of record. The route taken instead: structured API data is the only input the model ever sees.

The second choice gets missed in most AI-tool conversations: the model is used only where the judgment actually lives. Severity classification — reading a thread plus live ticket data against the SLA rubric and citing which criteria fired — is the LLM call. Everything else (the rotation schedule, notifications, user-group updates, channel topics) is a deterministic engine on cron. No model where a rule will do.

Mention the bot in a thread and it pulls the full conversation, extracts the ticket keys, fetches each ticket from Jira in parallel, and sends thread + structured tickets + the four-level SLA rubric to the model — which returns a severity level with the specific objective criteria cited, posted back in-thread. Conversation memory makes follow-ups work: ask “what if it only affects one region?” and it re-assesses in context. Postgres holds rotation state and, since v2, every verdict; a fifteen-minute cron tick drives the rotation engines.

The runtime model sits behind a provider adapter (today a fast Gemini model, chosen by an eval harness — see v2 below); Claude's role in this system is the build chain — the repo is specced and maintained through Claude Code, spec-first, like everything that came after it.

A year in, a read-only review of the assessor turned up something no incident had: the ticket system's v3 API returns descriptions as a rich-text document object, and the prompt had been string-interpolating it. For its entire prior life the model had seen [object Object] where the description should be. Every severity call had been made from the summary, the Slack thread, and metadata. Comments — where repro steps and business input actually live — were never fetched. A custom field that was supposed to carry “affected users” carried nothing on this instance. And the hand-maintained SLA rubric had drifted from the three canonical wiki pages.

The rebuild was ordered by leverage, inputs first: rich text → plain text for description, environment, and the latest ten comments; the thread transcript with the triage team's own reaction tags (their severity and environment emoji are explicit human signals that had been discarded). Then the engine became two steps at temperature zero — a schema-constrained fact extraction (surface, environment, scope, reproducibility, workaround, legal and data-loss flags, verbatim evidence), then a verdict against the active guideline version that cites matched criteria verbatim, states confidence (high only when environment and scope are both known), lists what's missing, and says what evidence would change the call. Verdicts render as Block Kit and persist as rows — facts, criteria, guideline version, model, latency, conversation — so follow-ups survive restarts and drift can be measured later instead of remembered.

The SLA itself became data with a version and an approval step. A weekly job fetches the three canonical wiki pages, transforms them into the structured rubric with a schema-constrained model call (criteria near-verbatim, never invented), diffs against the active version, and — only if something changed — inserts a pending version with the diff and asks an admin to approve it in the portal. Assessments keep using the last approved version; nothing changes silently. The model is a config value behind a provider adapter, gated by an eval harness that pulls real resolved tickets and scores the engine against the business's final call: first live run, six of six within one level, about fifteen seconds each. Small sample, stated as one.

Two things learned the hard way, now in the runbook: the provider's structured-output API rejects JSON-Schema $ref, which silently broke the sync for five days until the schemas were inlined; and a tiny intent-classification call plus a forgiving default replaced the keyword allow-list, so “should we be worried about this one?” now gets an assessment instead of silence.

The bot also moved from consumer hosting to the agency's internal platform — after first discovering it had been “deploying” there for two months without ever starting. Two independent causes: a cron-wrapper syntax in the manifest that the platform stores verbatim and Kubernetes rejects (copied from our own internal doc, since corrected), and a serverless database that auto-pauses and resumes in ~15 seconds against an ORM that gives up after 5 — about 1,400 restarts, found by reading the platform's own source.

A rotation bot that posts to a whole team is a bad thing to cut over noisily, so the cutover was designed to be silent: import production data through a token-gated endpoint and verify the current and next sprint's assignments match; record a pre-aligned notification snapshot on the new side so its first run diffs to “no changes” and sends no mass DMs; flip the Slack event URLs atomically; disable the old cron by rotating its secret to a sentinel; exactly one owner of the daily notification window; rollback is flipping the pointers back. Zero bot posts during the flip. Along the way it gained a self-serve portal — schedules, rosters, and multi-rotation management behind Slack sign-in, so any team can create its own rotation with its own cadence and timezone — and a post-cutover crash that was invisible in pod listings (every in-window tick failed on a UUID column) taught the cron to report its own failures to the admin channel.

About 90% of manual routing eliminated. Severity recommendations land within ~5 minutes — against up to a full day before — and get accepted roughly 99% of the time. The bot handles hundreds of tickets a month and prevents dozens of SLA breaches monthly. The best recent anecdote: on a production failure in a secondary workflow the bot returned a strict-rubric Sev 2 but explicitly offered Sev 3 if the workflow was secondary; the team chose Sev 3, and analytics later showed the feature was under 1% of traffic. That “what would change this verdict” field is the point.

And it became the canonical internal reference — “see: how the t1 triage bot pulls ticket information” is how we describe the pattern to every tool that came after. The v2 lesson joins it: before you change the model, check what the model can actually see.