Architecture
The reference layout I deploy, and — more important — where the trust boundaries sit.
Data flow
Clients (your services, staff tools)
│ virtual keys (vk-...), never provider keys
▼
┌───────────────────────────────────────────────┐
│ Reverse proxy (Caddy / nginx) │ TLS, HSTS, rate limit,
│ - terminates HTTPS │ security headers
└───────────────┬───────────────────────────────┘
▼
┌───────────────────────────────────────────────┐
│ Gateway (LiteLLM / one-api class) │
│ - auth: virtual key → team, quota, models │
│ - routing: model name → upstream pool │
│ - usage metering, request logging (redacted) │
│ - failover: retry next healthy upstream │
└──────┬──────────────────────────┬─────────────┘
▼ ▼
┌──────────────┐ ┌──────────────┐
│ Key vault │ │ SQLite/Postgres│
│ (env/file, │ │ usage, keys, │
│ 0600, no │ │ audit log │
│ web access) │ └──────────────┘
└──────┬───────┘
▼ provider keys (sk-..., owned by YOU)
Upstream providers (OpenAI, Anthropic, Google, ... official APIs)
Trust boundaries
| Boundary | Rule |
|---|---|
| Internet → proxy | Only 443 exposed. Admin UI is not on the public interface — SSH tunnel or VPN only. |
| Proxy → gateway | Loopback or private network. The gateway container never binds a public port. |
| Gateway → vault | Provider keys are readable by the gateway process user only (0600, non-root). They never appear in logs, error messages, or client responses. |
| Client → gateway | Clients hold revocable virtual keys with quotas. Compromise of a client burns one virtual key, not the provider account. |
Failure domains
- Upstream outage: health checks mark the upstream unhealthy; router retries the request against the next configured provider offering the same model class. The case study shows a real failover timeline (with synthetic data).
- Gateway crash: systemd/compose restart policy plus an external health probe (the kind shown in my sentinel project) alerting on state change.
- Host loss: nightly encrypted backup of the database and config; restore drill documented and rehearsed — restore time depends on image pull and DNS, measured per deployment during handover.