Case Study

Anonymised and generalised from real operational work on self-hosted gateway deployments. Identifying details, hostnames, exact figures and vendor mix are altered; the timeline structure and the failure modes are the real lesson. All numbers below are illustrative of the pattern, not billing records.

Starting point

A small product team ran three services against two AI providers. Each service held its own provider key in its own env file; one key had been shared over chat during an incident and never rotated. There was no per-team usage view — the invoice was one number.

Phase 1 · Consolidation (week 1)

Deployed the gateway (Docker Compose, non-root, read-only rootfs) behind Caddy with HTTPS and HSTS on an internal subdomain.
Moved both provider keys into the vault file; issued one virtual key per service with a monthly budget and a model allow-list.
Services switched by changing base_url and key — no code changes, because the gateway speaks the same API dialect.
Rotated the chat-leaked provider key after cutover; zero client changes needed, which is the point of the indirection.

Phase 2 · Routing policy (week 2)

Cheap/fast model class for internal tooling, premium class for the customer-facing feature — enforced by key, not by convention.
Per-key rate limits sized from two weeks of observed traffic.
Nightly encrypted backups plus an external health probe with transition-based alerting.

Phase 3 · The outage that paid for it

Some weeks later, the primary provider had a partial outage — elevated errors and latency (the pattern any status page archive will show for every major provider). Timeline as the gateway saw it, reproduced in the demo with synthetic data:

T+0m upstream A error rate exceeds threshold; health check marks A degraded T+0m router begins retrying failed requests against upstream B (same class) T+2m probe alerts once: "upstream-a: FAIL" — one message, no storm T+31m provider recovers; health check passes twice; A back in rotation T+31m probe alerts once: "upstream-a: RECOVERED" Client-visible impact: elevated p95 latency for ~2 minutes; zero 5xx bursts

What made it boring (the goal)

Failover was configured and tested with a forced failure drill during handover — the incident was the second time it ran, not the first.
Alert deduplication meant two messages, not two hundred.
The retrospective was one paragraph, because the audit log already had the timeline.

Handover

The team received the runbook, the restore drill recording, and admin access. My involvement ended by design; the system did not need me — that is the deliverable.