Open Source Projects tagged "Inference"
A curated collection of the best open source projects tagged "Inference". Each listing includes a website screenshot along with a detailed review of its features.
High-throughput, memory-efficient serving engine for LLMs
Stars
Stars growing over the last 30 days96,539Last commit
9 hours agoLicense
Apache-2.0
Inference and serving engine for large language models, built for speed and hardware efficiency with an OpenAI-compatible API and support for a wide range of open models.
Open Source Alternative to:
Route GenAI traffic with confidence using open source
Stars
Stars growing over the last 30 days2,367Last commit
2 days agoLicense
Apache-2.0
Open source gateway built on Envoy for routing application traffic to GenAI services. Supports 16+ LLM providers including OpenAI, Anthropic, AWS Bedrock.
Open Source Alternative to:
GPU orchestration across clouds, Kubernetes, and on-prem
Stars
2,307Last commit
2 days agoLicense
MPL-2.0
Unified control plane for AI compute that provisions and schedules GPU workloads across any cloud, Kubernetes cluster, or bare-metal server.
Open Source Alternative to:
Unified API for all LLM providers with analytics
Stars
Stars growing over the last 30 days1,731Last commit
13 hours agoLicense
MIT
Route, manage, and analyze LLM requests across multiple providers with one API. Compatible with OpenAI format, includes usage analytics and performance monitoring.
Open Source Alternative to:
Bright Data – Web data for AI agents and apps. Built-in unblocking, proxies, and ready-made APIs.
Sent