Deployment Docs

The condensed, public version of the runbook every client receives. Commands are real and tested against the open-source gateway stack in the demo; hostnames and paths are examples.

1 · Docker deployment

# docker-compose.yml (excerpt — full file in the GitHub repo)
services:
  gateway:
    image: ghcr.io/berriai/litellm:main-stable   # or your gateway of choice
    user: "10001:10001"            # non-root
    read_only: true                # read-only rootfs
    tmpfs: [/tmp]
    env_file: /srv/gateway/secrets.env   # 0600, root:root
    expose: ["4000"]               # internal only — no public bind
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://127.0.0.1:4000/health/liveliness"]
      interval: 30s
      retries: 3

Principles: the gateway container never publishes a host port; only the reverse proxy does. Secrets live in an env file owned by root with mode 0600, mounted read-only. The container runs as a fixed non-root UID with a read-only filesystem.

2 · HTTPS with automatic renewal

# Caddyfile — HTTPS, HSTS and headers in 8 lines
example.com {{
  encode gzip
  header {{
    Strict-Transport-Security "max-age=31536000; includeSubDomains"
    X-Content-Type-Options nosniff
    X-Frame-Options DENY
    Referrer-Policy strict-origin-when-cross-origin
  }}
  reverse_proxy 127.0.0.1:4000
}}

Caddy provisions and renews certificates automatically. With nginx the same is done with certbot and a systemd timer; both variants are in the repo.

3 · Backups that restore

# nightly: dump DB + config, encrypt, keep 14 days
sqlite3 /srv/gateway/data/gateway.db ".backup /tmp/gw.db"
tar czf - -C /srv/gateway config data | age -r "$BACKUP_PUBKEY"   > /backup/gateway-$(date +%F).tar.gz.age
find /backup -name "gateway-*.age" -mtime +14 -delete

A backup is not a backup until it has been restored. The runbook includes a quarterly restore drill: restore to a scratch directory, start a second gateway instance on a private port, run the smoke tests, tear down.

4 · Upgrades without surprises

  1. Read the release notes; check for schema migrations.
  2. Snapshot: backup as above, note current image digest.
  3. Pull the new image, restart, watch the health endpoint and error rate.
  4. Regression: run the smoke suite (key create → chat request → usage row).
  5. Rollback path: previous image digest + restored DB — rehearsed, so the timeline is known per deployment, not guessed.

5 · Monitoring