LiteLLM

LiteLLM is an LLM gateway built for platform teams that need to give developers access to many different AI providers without managing the complexity of each one individually. It sits between your applications and providers like OpenAI, Anthropic, Azure, Gemini, and Bedrock, exposing a single OpenAI-compatible API regardless of which model is actually handling the request.

The core appeal is normalization. Every provider has its own API shape, authentication scheme, and error format. LiteLLM abstracts all of that away, so your developers write code once and can swap or add models without touching their integration.

Key capabilities include:

• Spend tracking and budgets: Assign virtual keys to teams or users, set hard spending limits, and get accurate per-team cost breakdowns.
• Rate limiting: Enforce RPM and TPM caps per key, team, or model to prevent runaway usage.
• Load balancing and fallbacks: Route requests across multiple deployments of the same model, and automatically fall back to a secondary provider when one fails.
• LLM observability: Native integrations with Langfuse, Arize Phoenix, LangSmith, and OpenTelemetry for logging and tracing.
• Guardrails: Apply input/output filtering before requests reach the model.
• Pass-through endpoints and S3 logging: For teams that need raw request capture or custom routing.

The open source version covers the full feature set for most teams. The enterprise tier adds SSO, JWT auth, audit logs, and custom SLAs for larger organizations.

LiteLLM is self-hostable via Docker and has seen over 240 million pulls. Netflix uses it to give developers access to new models within a day of release, citing the elimination of per-provider input/output transformation as the main time saver.