What is
LiteLLM
?
Most platform initiatives fail the same way: they start too big, get funded like projects instead of products, build for too long without testing with users, and confuse renaming an infrastructure team with building a platform team. Instead, start small, learn fast and scale deliberately.
At a glance
Attribute | Details |
|---|---|
Primary use case | Centralize AI model, agent, and MCP access with request routing, spend controls, and governance |
Deployment type | Self-hosted AI gateway/proxy server and Python SDK; preconfigured Selectel cloud virtual machine also available |
Pricing model | Free/open-source core; paid Enterprise licensing available through sales; self-hosting infrastructure costs separate |
API compatibility | OpenAI-compatible API and input/output format; Model Context Protocol (MCP) |
Supported integrations | OpenAI, Anthropic, Gemini, Amazon Bedrock, Azure, Vertex AI, Ollama, Langfuse, MLflow, Helicone, MCP servers |
Source type | Open |
Cloud provider | Cloud Agnostic |
Ideal team size | Small teams to enterprise platform teams |
Onboarding time | Minutes for initial self-hosted setup |
Required skills | API integration, Python for SDK usage, API key and secret management, infrastructure administration for self-hosting |
AEP branch | Agent Infrastructure |
AEP category | Harness |
AEP subcategory | Execution |
AEP detail | Gateways / Routers |
Profile
Gateways / Routers are tools for routing requests to AI services through a centralized interface. LiteLLM is an open-source AI gateway and proxy server, also available as a Python SDK, that provides an OpenAI-compatible interface to 140+ providers and 1,800+ models.
The gateway can centralize access to models, agents, and MCP servers. It provides request routing, spend controls, and governance, and supports connections to internal, fine-tuned, and self-hosted models.
Focus
LiteLLM is intended for small teams through enterprise platform teams that need to centralize AI access. It addresses the problem of connecting applications and users to multiple model providers, agents, and MCP servers through a shared API while managing keys, requests, and spending.
Background
The available research identifies BerriAI as the maintainer of the LiteLLM open-source project and links to its public GitHub repository. It describes LiteLLM as an open-source AI gateway, but does not provide details about the company’s founding or funding.
Features
Unified model access
Teams can connect applications to models from multiple providers through one consistent interface. LiteLLM provides an OpenAI-compatible API for 140+ providers and 1,800+ models, and supports bringing internal, fine-tuned, and self-hosted models behind the same key.
Request routing and reliability
Teams can direct requests to suitable model deployments and manage availability across them. The gateway supports routing, load balancing, retries, and fallback routing across deployments.
Spend controls and governance
Platform teams can manage AI access and spending centrally. LiteLLM provides virtual keys, team- and project-scoped access, spend budgets, and rate limits; its website also describes integration with existing secret managers.
Agent and MCP access
Teams can access agents and MCP servers through the same gateway used for models. LiteLLM supports routing requests to agents and MCP servers as well as to LLM providers.
Observability integrations
Teams can connect gateway activity to observability and tracking tools. Supported integrations include Langfuse, MLflow, and Helicone.
Pros and cons
Pros
Provides one OpenAI-compatible API for 140+ providers and 1,800+ models, allowing teams to change models without changing application code.
Centralizes access to models, agents, and MCP servers, with virtual keys, scoped access, spend budgets, and rate limiting.
Supports routing, load balancing, retries, and fallbacks, and integrates with Langfuse, MLflow, and Helicone.
Offers an open-source core and multiple deployment options, including self-hosting and a preconfigured Selectel cloud virtual machine.
Cons
Self-hosting requires infrastructure administration, and infrastructure costs are separate from the free/open-source core.
SSO is free for up to 5 users from v1.76.0; larger SSO deployments require an Enterprise license, available through sales.
FAQs
LiteLLM FAQ
How does LiteLLM route requests across model deployments?
LiteLLM routes requests across model deployments and supports load balancing, retries, and fallback routing.
Which providers integrate with LiteLLM?
LiteLLM supports connections to OpenAI, Anthropic, Gemini, Amazon Bedrock, Azure, Vertex AI, Ollama, and MCP servers; observability integrations include Langfuse, MLflow, and Helicone.
Can LiteLLM route requests to agents and MCP servers?
Yes. LiteLLM routes requests to agents and MCP servers through the same gateway it uses for models.
What access and spending controls does LiteLLM provide?
LiteLLM provides virtual keys, team- and project-scoped access, spend budgets, and rate limits.
