rotation-proof

The platform abstracts AWS away — nobody on the team has console access, which is the point. Underneath, the managed database's master password rotates roughly weekly, and the platform injects connection credentials into each pod at deploy time. Frozen. So any pod that restarted after a rotation came up with last week's password, failed its startup migration, and crash-looped until a human redeployed it. Apps nobody had touched went down on a schedule. Through late May and early June it hit five of them; the operational answer, visible in CI history, was a human cron job — manual redeploys on the 21st, the 28th, the 5th.

The one that made it a platform story: a revenue-pipeline tracker that had already shipped rotation hardening in June — runtime resolver, self-healing pool, watchdog, two replicas — went 503 anyway in August. About 1,150 restarts. The runtime logs said the live fetch worked. So why dead?

Credentials are a runtime concern, not a deploy-time one. The connection pool is disposable; the live secret is the source of truth. And — the part that took three outages to fully learn — it has to be true for every pool and every phase: the one-shot startup commands (migrate, seed) that run before the app's pool exists, the long-lived runtime pool, and every secondary pool nobody thinks of. A correct runtime layer is defeated by a stale startup step; a correct primary pool is defeated by a frozen session store.

Reading the startup script answered the August question: the resolver only ran if DATABASE_URL is empty. The platform — contrary to its own documentation, which said it injected only the component variables — had started injecting a composite connection URL. The guard made the resolver dead code; migrations always ran with the deploy-time password; any post-rotation restart crash-looped before the runtime layer got a chance. The same dead-code bug took down the telemetry service and a brand-intelligence app two weeks later, on the same day. Once the precedence was inverted — prefer live, fall back to frozen, log which path fired — the fix ported in one commit each.

Hand-bouncing works — a redeploy re-freezes the current password and the app is back in four minutes — and it is a human cron job with a weekly deadline. Rejected as a steady state. Runtime-only resolution (the June fix) was necessary and insufficient: validated against a local Postgres, never against a real rotation, and blind to the startup step. Replicas help — a single replica is the whole app down the instant its pool dies — but aren't universal: one app is pinned to one replica because in-process schedulers would duplicate, so the fix has to live at the app layer, not the manifest.

Two siblings to the same lesson surfaced along the way. A guest sign-in kiosk where every valid login returned 500 and every bogus login returned 401 — the asymmetry was the tell: the ORM pool self-healed, but the session store had its own pool with the password frozen at startup. Fix: an async password callback so each new connection re-resolves. And a Slack rotation bot with ~1,400 restarts that wasn't rotation at all — a serverless database that auto-pauses at zero capacity and resumes in ~15 seconds against an ORM that gives up after 5, each backoff retry arriving after the database had paused again. Plus a cron-schedule syntax copied from our own doc that the platform stores verbatim and Kubernetes rejects — two months of a “failed deploy” that had never once started, found by reading the platform's own source. Verify against reality, not docs — including ours.

Seven pieces, now the reference implementation across the fleet (implementations vary by stack; not every app carries all seven): a startup resolver that fetches the live secret before migrate or seed and says on stderr which source it used; a runtime resolver returning a URL plus a fingerprint (a hash prefix, never the secret); a swappable client holder with single-flight rebuild — auth-class errors re-resolve, rebuild, retry once, and the old client drains; a watchdog on a 60-second cadence comparing fingerprints so the pool swaps before a query fails; a real readiness probe that pings the database, with liveness kept light so a transient blip doesn't kill the pod; secondary pools on an async password callback; and replicas ≥ 2 where the app tolerates it.

A static “ok” health check hides a dead pool forever — and defeats the platform's own restart backstops, which can only act on a pod that admits it's sick. That rule, with the pg driver quirk that treats a URL's sslmode=require as full certificate verification, the serverless auto-pause timeouts, and the cron-syntax fix, went into the shared deployment guide as a rotation-proof chapter so no future app rediscovers any of it.

The cleanest proof on record: four days after the rotation bot was hardened, the weekly rotation landed and the logs showed a new credential fingerprint, a swapped pool, and health that never left 200 — no restart, no human. The tracker, the telemetry service, and the brand-intelligence app all boot with “using live credentials from Secrets Manager” before their migrations now, and the runtime pools report source=secret; the next natural rotation is their live test. Two August apps shipped the rotation-aware layer on day one of their migration, the reference implementation for every Node/pg app that follows.

The honest footnote: the shared guide still carried the stale “the platform doesn't inject a connection URL” claim that caused the August outages when I last checked, with the correction in review. Docs rot exactly like credentials do — which is why the pattern logs the truth on every boot.