LiteLLM - Hardened Self-Hosted LLM Gateway

What is LiteLLM

LiteLLM is a lightweight, open-source LLM gateway written in Python. It exposes a single OpenAI-compatible API for 100+ Large Language Model providers, so applications written against the OpenAI SDK can talk to any backend without code changes. The gateway adds virtual API keys, per-key spend tracking and budgets, automatic fallback between providers, rate limiting, request and response logging, and a built-in admin UI to manage models, teams, and keys.

Why self-host LiteLLM

Self-hosting keeps every prompt, completion, and provider API key inside your own infrastructure - no third-party visibility into your data, no per-seat SaaS fees on top of the underlying model costs. Ideal for teams with data residency requirements, organisations operating under regulated frameworks (HIPAA, GDPR, ISO 27001), defence and government work, and platform teams that need a single chokepoint for LLM cost, audit, and access control across many internal applications.

What this VM image adds

Security hardening:

• Unique master key generated per instance - sk- prefixed bearer token created at first boot, used to manage virtual keys via the REST API
• Unique salt key generated per instance - AES key that encrypts stored virtual keys and provider credentials inside PostgreSQL, never the same on two deployments
• Admin UI password equal to the per-VM Instance ID - operator changes it after first web login
• PostgreSQL 16 listening on localhost only - 127.0.0.1:5432, per-instance database password, no external exposure
• LiteLLM proxy bound to 127.0.0.1:4000 - all external traffic terminates at Nginx, the gateway itself is never directly exposed
• Nginx reverse proxy with TLS - HTTP to HTTPS redirect, hardened cipher suite, security headers (X-Frame-Options, X-Content-Type-Options, Referrer-Policy), startup splash page served while the proxy is warming up
• LiteLLM runs as non-root - dedicated litellm system user, systemd UMask=0027, ProtectSystem=full, NoNewPrivileges
• CVE scan - every image is scanned for vulnerabilities with Trivy before release
• UFW firewall - only ports 22 (system SSH), 80, and 443 open
• fail2ban - SSH brute-force protection
• AppArmor - mandatory access control
• Certbot pre-installed - one command issues a Let's Encrypt certificate after you point a domain at the VM

OS hardening (CIS Level 1):

• CIS Level 1 hardened - CIS Ubuntu 24.04 LTS Level 1 Benchmark via ansible-lockdown
• auditd - system call auditing for critical paths
• SSH hardening - PasswordAuthentication disabled, key-only access
• Kernel hardening - SYN cookies, ASLR, rp_filter, TCP BBR
• /tmp as tmpfs - nosuid, nodev, noexec
• Azure IMDS endpoints - egress rules pre-configured (169.254.169.254, 168.63.129.16)

Compliance artifacts (inside the VM):

• SBOM - CycloneDX 1.6 at /etc/lynxroute/sbom.json
• CIS Conformance Report - OpenSCAP HTML at /etc/lynxroute/cis-report.html
• Tailored CIS profile - /usr/share/doc/lynxroute/CIS_TAILORED_PROFILE.md
• Server credentials file - /root/litellm-credentials.txt with public IP, admin UI URL, the per-instance master key, the salt key, and the per-instance PostgreSQL password

Quick Start

1. Deploy VM from Azure Marketplace (Standard_D2s_v3 or larger recommended)
2. Open NSG: TCP 80 and 443 from your client networks, TCP 22 from your management IPs only
3. SSH: ssh -i key.pem <username>@<PUBLIC_IP> (username set during VM creation, default: azureuser)
4. Read connection details: sudo cat /root/litellm-credentials.txt - contains the admin UI URL, the master key, and the PostgreSQL password
5. Open https://<PUBLIC_IP>/ui in your browser, accept the self-signed certificate, log in as admin with the password from the credentials file, and change it after first login
6. Add your provider API keys to /etc/litellm/providers.env and run sudo systemctl restart litellm
7. Issue a public TLS certificate (recommended before sharing with users): sudo certbot --nginx -d your.domain.com

The image ships with a self-signed certificate so the web UI works on first boot - replace it with a CA-signed certificate before exposing the gateway to end users.