LiteLLM

LiteLLM is an LLM gateway built for platform teams that need to give developers access to many different AI providers without managing the complexity of each one individually. It sits between your applications and providers like OpenAI, Anthropic, Azure, Gemini, and Bedrock, exposing a single OpenAI-compatible API regardless of which model is actually handling the request.

The core appeal is normalization. Every provider has its own API shape, authentication scheme, and error format. LiteLLM abstracts all of that away, so your developers write code once and can swap or add models without touching their integration.

Key capabilities include:

• Spend tracking and budgets: Assign virtual keys to teams or users, set hard spending limits, and get accurate per-team cost breakdowns.
• Rate limiting: Enforce RPM and TPM caps per key, team, or model to prevent runaway usage.
• Load balancing and fallbacks: Route requests across multiple deployments of the same model, and automatically fall back to a secondary provider when one fails.
• LLM observability: Native integrations with Langfuse, Arize Phoenix, LangSmith, and OpenTelemetry for logging and tracing.
• Guardrails: Apply input/output filtering before requests reach the model.
• Pass-through endpoints and S3 logging: For teams that need raw request capture or custom routing.