Skip to content

LiteLLM

πŸ” Quick overview

LiteLLM is an open-source proxy/gateway (MIT-licensed core, enterprise/ features under a commercial license) that exposes an OpenAI-compatible interface to 100+ providers in Q4 2026: Ollama, vLLM, OpenAI, Anthropic, Azure, Bedrock, Vertex AI, Hugging Face, and others1.

πŸ’‘ Why this project interests us

For a custodian agent, LiteLLM serves as a model abstraction layer. The agent speaks OpenAI-compatible; the operator decides whether the request goes to local Ollama, vLLM, or a cloud provider.

This avoids rewriting the agent on every model change.

βœ… Strengths

  • Unified API for local and cloud models1.
  • Documented Ollama/vLLM support12.
  • Central proxy with virtual keys, routing, costs, logs, and guardrails.
  • Useful for gradual cloud β†’ local migration.

⚠️ Limitations and risks

  • Not a model: it routes to backends.
  • Bad config = leak to cloud.
  • Logging/observability can capture prompts/responses if enabled without care3.
  • Adds a critical layer to secure: in 2026 LiteLLM suffered a series of exploited vulnerabilities, including three listed in CISA’s KEV catalog (CVE-2026-42208, CVE-2026-42271, CVE-2026-59822, all fixed from 1.84.0 onwards) and a critical privilege escalation (GHSA-7hp6-4w63-5g45, CVSS 9.9, an internal_user becomes proxy_admin and then executes commands on the host; fixed in 1.100.4 / 1.101.3 / 1.102.2 / 1.103.1). As of Q4 2026, never deploy a version earlier than 1.84.0. Version 1.100.4 fixes the September escalation, but the critical advisory of 2026-10-27 only patches the 1.103 to 1.106 lines (1.103.10 / 1.104.8 / 1.105.6 / 1.106.5), and the advisory of 2026-11-12 (GHSA-2rwg-2wh4-7c4f, high severity, CVSS 7.7: on a proxy serving at least one azure/ or azure_ai/ model without AZURE_API_KEY or AZURE_OPENAI_API_KEY set, an authenticated user reads the proxy’s environment variables, including LITELLM_MASTER_KEY) is only fixed in 1.106.9 / 1.108.6 / 1.109.1, with the 1.107 line having no fix: as of Q4 2026, target at least one of these three versions, or in the meantime block the Entra ID parameters (tenant_id, client_id, client_secret, azure_username, azure_password, azure_scope) at the reverse proxy4. This is imperative with an Enterprise license (GHSA-w4c7-34wj-hqm8, CVSS 10.0: an unauthenticated request makes the proxy read a local file, for example /proc/self/environ, and exposes LITELLM_MASTER_KEY and the providers’ keys; open-source deployments without a license are not affected). In all cases, restrict the proxy’s outbound network access to the providers’ endpoints and load the master key from a secrets manager rather than from the process environment567.
  • Version discipline: since 2026-06-29, only the four most recent stable minor lines receive patches (one minor per week, so roughly one month of coverage per line) β€” plan at least a monthly update; a frozen version is a vulnerable version8.
  • Supply chain: in March 2026, two PyPI releases (1.82.7 and 1.82.8) published by an attacker stole the host machine’s secrets; the official Docker image was not affected9. Install from the pinned official image or with hash-pinned dependencies, and rotate every secret held by the proxy if a compromised version was installed.
  • MCP surface: the proxy also embeds an MCP gateway; two of its vulnerabilities were exploited in 2026 (CVE-2026-42271, CVE-2026-59822). If the custodian agent does not use MCP through LiteLLM, do not expose these endpoints5.

πŸ”’ Sovereignty and privacy

  • Data: transit through the proxy; destination per backend.
  • Model: local if local Ollama/vLLM backend; cloud if cloud provider.
  • Memory: not application memory, except logs/observability.
  • Telemetry/logging: configurable; disable message logging for sensitive data3.
  • 100% offline mode: yes with local backends.
  • Verdict: βœ… sovereign if configured local-only; ⚠️ otherwise.

πŸ”— Possible integration in this vault

LiteLLM is the target layer between:

  • OpenHands / Agent Canvas (or Aider, whose development has been frozen since May 2026, see its sheet);
  • Ollama/vLLM;
  • routing policies;
  • local logs;
  • optional cloud fallback.

πŸ“Š Project maturity

Very widely used as a gateway (over 60,000 GitHub stars as of Q4 20261), LiteLLM publishes one minor line per week (1.104.2 on 2026-10-0810) and, since 2026-06-29, only patches the four most recent stable minor lines: each line receives roughly one month of fixes8. Freezing a version is therefore not a viable strategy: plan at least a monthly update, follow the GitHub advisories feed, and version/audit configuration, secrets, logs, and routing rules.

πŸ”— See also

πŸ“š Sources

Footnotes

  1. LiteLLM GitHub README (β€œ100+ LLM providers”) and LICENSE (MIT, except enterprise/ under enterprise/LICENSE), read on 2026-10-09. https://github.com/BerriAI/litellm ↩ ↩2 ↩3 ↩4

  2. LiteLLM Proxy docs β€” local proxy, Ollama, vLLM. https://docs.litellm.ai/docs/proxy_server ↩

  3. LiteLLM Docs, Logging β€” callbacks, OpenTelemetry, turn_off_message_logging. https://docs.litellm.ai/docs/proxy/logging ↩ ↩2

  4. BerriAI, GHSA-2rwg-2wh4-7c4f (LiteLLM advisory, high severity: secret exfiltration by an authenticated user via the Entra ID parameters of an azure/ or azure_ai/ model, when AZURE_API_KEY and AZURE_OPENAI_API_KEY are not set; fixed in 1.106.9 / 1.108.6 / 1.109.1; workaround: block the six parameters at the reverse proxy), 2026-11-12. https://github.com/BerriAI/litellm/security/advisories/GHSA-2rwg-2wh4-7c4f ↩

  5. CISA, Known Exploited Vulnerabilities Catalog (JSON feed; LiteLLM entries CVE-2026-42208 added 2026-05-08, CVE-2026-42271 on 2026-06-08, CVE-2026-59822 on 2026-09-02), catalog dated 2026-10-08. https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json ↩ ↩2

  6. BerriAI, GHSA-7hp6-4w63-5g45 (LiteLLM advisory, internal_user β†’ proxy_admin β†’ command execution on the host, CVSS 9.9, versions 1.91.0 β†’ < 1.100.4, fixed in 1.100.4 / 1.101.3 / 1.102.2 / 1.103.1), 2026-09-30. https://github.com/BerriAI/litellm/security/advisories/GHSA-7hp6-4w63-5g45 ↩

  7. BerriAI, GHSA-w4c7-34wj-hqm8 (critical LiteLLM advisory, CVSS 10.0: unauthenticated file read on a proxy under an Enterprise license, exposing LITELLM_MASTER_KEY and the providers’ keys; affected versions < 1.103.10 / < 1.104.8 / < 1.105.6 / < 1.106.5, fixed in 1.103.10 / 1.104.8 / 1.105.6 / 1.106.5; open source without a license not affected; recommendations: master key in a secrets manager, distinct LITELLM_SALT_KEY, outbound network access limited to the providers), 2026-10-27. https://github.com/BerriAI/litellm/security/advisories/GHSA-w4c7-34wj-hqm8 ↩

  8. LiteLLM, Version Support Policy (blog: from 2026-06-29, only the four most recent stable minor lines receive patches), 2026-06-20. https://docs.litellm.ai/blog/version-support ↩ ↩2

  9. BerriAI, Issue #24518 (PyPI releases 1.82.7 and 1.82.8 published outside CI by an attacker, host secret theft; proxy Docker image not affected), 2026-03-24. https://github.com/BerriAI/litellm/issues/24518 ↩

  10. BerriAI, LiteLLM Releases (v1.104.2 published on 2026-10-08, cadence of one minor line per week), accessed 2026-10-09. https://github.com/BerriAI/litellm/releases ↩

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.