Skip to content

LiteLLM

πŸ“ Short definition

Proxy/gateway exposing an OpenAI-compatible API and routing requests to Ollama, vLLM, OpenAI, Anthropic, Azure, Bedrock, or other providers.

πŸ“– Detailed definition

LiteLLM is an abstraction layer between an agentic app and model engines. Instead of coding one connector per provider, the agent speaks one interface; the operator then chooses local models, a vLLM cluster, or a cloud provider.

It can also centralize keys, routing, quotas, logs, observability and, since 2026, an MCP gateway to the agents’ tools1.

πŸ’‘ Why it matters for on-prem AI

In a sovereign setup, LiteLLM can enforce local-only routing to Ollama or vLLM. In a hybrid architecture, it keeps a stable API while migrating gradually from cloud to local.

⚠️ Common pitfalls

  • Assuming LiteLLM makes a stack sovereign automatically: everything depends on configured backends.
  • Enabling prompt/response logs without a retention policy.
  • Leaving a silent cloud fallback in a supposed on-prem configuration.
  • Deploying the proxy and then forgetting it: since June 2026, LiteLLM only maintains its last four minor lines (about one month of fixes each)2, and several of its 2026 flaws are actively exploited (CISA KEV catalog). A gateway exposed to users must keep up with releases β€” see the solution sheet.

πŸ“š Go deeper

  1. LiteLLM solution sheet
  2. Target custodian agent architecture

πŸ”— See also

Footnotes

  1. LiteLLM Docs, MCP Overview (MCP Gateway: single entry point to MCP servers, per-key and per-team access control; A2A β€œAgent Gateway” in the menu), accessed 2026-10-10. https://docs.litellm.ai/docs/mcp ↩

  2. LiteLLM, Version Support Policy (four stable minor lines maintained, rolling window of about one month per line, effective 2026-06-29), 2026-06-20. https://docs.litellm.ai/blog/version-support ↩

Give my opinion

Everything is optional: one answer is enough. This site sends nothing: you send it, the way you choose.

Your noteNo note
Your opinion is about
About what? (several choices possible)
What will be sent
Send by e-mail

Private, no account: opens your mail program with the message ready, for ia-on-prem@damien.becherini.fr.

Open a GitHub issue β†—

Public, with a GitHub account. You read the issue again before you publish it.

To paste it wherever you want.

Give at least a note, a subject or a message to be able to send.