Open Source Projects tagged "Llm Inference"
A curated collection of the best open source projects tagged "Llm Inference". Each listing includes a website screenshot along with a detailed review of its features.
Run LLMs locally with minimal setup, maximum hardware support
Stars
134,454Last commit
4 hours agoLicense
MIT
C/C++ inference engine for large language models, supporting quantization, multi-GPU, Apple Silicon, and an OpenAI-compatible server across a wide range of hardware.
Open Source Alternative to:
High-throughput, memory-efficient serving engine for LLMs
Stars
Stars growing over the last 30 days96,830Last commit
4 hours agoLicense
Apache-2.0
Inference and serving engine for large language models, built for speed and hardware efficiency with an OpenAI-compatible API and support for a wide range of open models.
Open Source Alternative to:
Run open-source AI models privately on your own device
Stars
77,356Last commit
1 year agoLicense
MIT
GPT4All runs open-source language models locally on Windows, macOS, and Linux with no cloud dependency, keeping your data on your machine.
Open Source Alternative to:
Serverless GPU compute with sub-second cold starts
Stars
1,848Last commit
6 hours agoLicense
AGPL-3.0
Run GPU inference, task queues, and sandboxes on serverless infrastructure with sub-second cold starts, autoscaling, and support for your own AWS, GCP, or bare metal.
Open Source Alternative to:
InfluxDB – The purpose-built time series platform for high-velocity ingestion and real-time queries at scale.
Docmost