Security

The parts of gateway operations that are invisible in a happy demo and decisive in an incident.

Key custody

Secret	Where it lives	Where it must never appear
Provider keys (sk-…)	env file 0600 on the host, or a secrets manager; readable only by the gateway process user	logs, error bodies, client responses, git, container images, backups in plaintext
Virtual keys (vk-…)	hashed at rest in the gateway DB	anywhere in full after creation — the UI shows a one-time reveal, logs show a prefix
Admin credentials	password manager; TOTP enabled	shared chats, .env files in repos

Containers run as a dedicated non-root UID with a read-only root filesystem and a tmpfs for scratch space.
The admin plane binds to localhost; reaching it requires SSH tunnel or VPN membership. The public interface serves inference traffic only.
Database user has no DDL rights at runtime; migrations run as a separate step.
Each client team gets its own virtual keys with model allow-lists and budgets — revoking one client never touches another.

Gateways sit on the request path, so their logs are a honeypot by default. The deployment redacts:

Authorization headers → vk-****last4
Prompt and completion bodies → logged only as token counts unless the client explicitly opts into content logging for debugging, with a TTL
Upstream error bodies → provider keys stripped before the error reaches client or log

The demo request log shows exactly this shape: enough to operate, nothing worth stealing.

Incident	Response
Virtual key leaked	Revoke the key (requests fail closed), review its usage window in the audit log, issue a replacement with the same policy. Blast radius: that key’s quota.
Provider key suspected	Rotate at the provider, update the vault entry, restart the gateway (seconds of downtime behind a retrying proxy), verify with a canary request.
Host compromise suspected	Freeze: revoke provider keys at the provider first — they are the crown jewels. Rebuild from a clean image plus the encrypted backup; never restore executables from the compromised host.
Data loss	Restore last night’s encrypted dump; usage rows since the dump are gone and acknowledged as gone — billing reconciles against provider dashboards.