Architecture

The reference layout I deploy, and — more important — where the trust boundaries sit.

Data flow

Clients (your services, staff tools) │ virtual keys (vk-...), never provider keys ▼ ┌───────────────────────────────────────────────┐ │ Reverse proxy (Caddy / nginx) │ TLS, HSTS, rate limit, │ - terminates HTTPS │ security headers └───────────────┬───────────────────────────────┘ ▼ ┌───────────────────────────────────────────────┐ │ Gateway (LiteLLM / one-api class) │ │ - auth: virtual key → team, quota, models │ │ - routing: model name → upstream pool │ │ - usage metering, request logging (redacted) │ │ - failover: retry next healthy upstream │ └──────┬──────────────────────────┬─────────────┘ ▼ ▼ ┌──────────────┐ ┌──────────────┐ │ Key vault │ │ SQLite/Postgres│ │ (env/file, │ │ usage, keys, │ │ 0600, no │ │ audit log │ │ web access) │ └──────────────┘ └──────┬───────┘ ▼ provider keys (sk-..., owned by YOU) Upstream providers (OpenAI, Anthropic, Google, ... official APIs)

Trust boundaries

BoundaryRule
Internet → proxyOnly 443 exposed. Admin UI is not on the public interface — SSH tunnel or VPN only.
Proxy → gatewayLoopback or private network. The gateway container never binds a public port.
Gateway → vaultProvider keys are readable by the gateway process user only (0600, non-root). They never appear in logs, error messages, or client responses.
Client → gatewayClients hold revocable virtual keys with quotas. Compromise of a client burns one virtual key, not the provider account.

Failure domains