LiteLLM (US) is an open-source proxy that exposes an OpenAI-compatible API for more than a hundred model providers. It can run as a central service (gateway) or as a Python library inside the application.
What it does
- Access different providers with the same API contract
- Routing with retries and fallback between deployments of the same model
- Virtual keys and budgets per project, team or user
- Centralized spend tracking and usage limits
- Send logs to observability tools through callbacks
How it works
- In proxy mode, the application calls a single OpenAI-compatible endpoint
- In library mode, the SDK translates the call to the chosen provider
- The router picks a deployment based on availability and limits
- It runs on your own infrastructure, with optional database and cache
Models and providers
- More than a hundred providers, including OpenAI, Anthropic, Google, Azure, Bedrock and open providers
- Text, image, audio, embeddings and responses models
- Providers added with your own key (BYOK)
Public pricing
- Open-source core free for self-hosting
- Enterprise edition under quote, with SSO, guardrails and access controls
- No mandatory managed service
Strengths
- A de facto standard for self-hosted model gateways
- Easy to start as a library and grow into a central proxy
- Broad community and adapters for new providers
Points of attention
- Operation, scaling and security are on your team
- Full observability requires integrating external tools
- Advanced governance features are in the paid edition