Open Source Projects tagged "Inference"
A curated collection of the best open source projects tagged "Inference". Each listing includes a website screenshot along with a detailed review of its features.
High-throughput, memory-efficient serving engine for LLMs
Stars
92,916Last commit
3 hours agoLicense
Apache-2.0
Inference and serving engine for large language models, built for speed and hardware efficiency with an OpenAI-compatible API and support for a wide range of open models.
Open Source Alternative to:
Route GenAI traffic with confidence using open source
Stars
2,154Last commit
20 hours agoLicense
Apache-2.0
Open source gateway built on Envoy for routing application traffic to GenAI services. Supports 16+ LLM providers including OpenAI, Anthropic, AWS Bedrock.
Open Source Alternative to:
GPU orchestration across clouds, Kubernetes, and on-prem
Stars
2,263Last commit
4 hours agoLicense
MPL-2.0
Unified control plane for AI compute that provisions and schedules GPU workloads across any cloud, Kubernetes cluster, or bare-metal server.
Open Source Alternative to:
Unified API for all LLM providers with analytics
Stars
1,665Last commit
4 hours agoLicense
MIT
Route, manage, and analyze LLM requests across multiple providers with one API. Compatible with OpenAI format, includes usage analytics and performance monitoring.
Open Source Alternative to:
Sent – Messaging Infrastructure for Developers. One API for SMS, WhatsApp, and RCS across 190+ countries.
Sent