LiteLLM
π Short definition
Proxy/gateway exposing an OpenAI-compatible API and routing requests to Ollama, vLLM, OpenAI, Anthropic, Azure, Bedrock, or other providers.
π Detailed definition
LiteLLM is an abstraction layer between an agentic app and model engines. Instead of coding one connector per provider, the agent speaks one interface; the operator then chooses local models, a vLLM cluster, or a cloud provider.
It can also centralize keys, routing, quotas, logs, observability and, since 2026, an MCP gateway to the agentsβ tools1.
π‘ Why it matters for on-prem AI
In a sovereign setup, LiteLLM can enforce local-only routing to Ollama or vLLM. In a hybrid architecture, it keeps a stable API while migrating gradually from cloud to local.
β οΈ Common pitfalls
- Assuming LiteLLM makes a stack sovereign automatically: everything depends on configured backends.
- Enabling prompt/response logs without a retention policy.
- Leaving a silent cloud fallback in a supposed on-prem configuration.
- Deploying the proxy and then forgetting it: since June 2026, LiteLLM only maintains its last four minor lines (about one month of fixes each)2, and several of its 2026 flaws are actively exploited (CISA KEV catalog). A gateway exposed to users must keep up with releases β see the solution sheet.
π Go deeper
π See also
Footnotes
-
LiteLLM Docs, MCP Overview (MCP Gateway: single entry point to MCP servers, per-key and per-team access control; A2A βAgent Gatewayβ in the menu), accessed 2026-10-10. https://docs.litellm.ai/docs/mcp β©
-
LiteLLM, Version Support Policy (four stable minor lines maintained, rolling window of about one month per line, effective 2026-06-29), 2026-06-20. https://docs.litellm.ai/blog/version-support β©