vLLM

vLLM is a serving engine for large language models, built for teams and developers who need to run LLMs at scale without burning through GPU budgets. It's designed around two core problems: throughput and memory. Most inference setups waste GPU memory and process requests inefficiently. vLLM addresses both.

The engine's standout technique is PagedAttention, which manages the KV cache the way an operating system manages virtual memory. This dramatically reduces memory waste and allows more requests to run concurrently on the same hardware. Paired with continuous batching, it keeps GPU utilization high even under variable load, rather than waiting to fill a fixed batch before processing.

Key capabilities include:

• OpenAI-compatible API so existing apps built against OpenAI's endpoints can switch to self-hosted models with minimal code changes
• Broad model support covering the latest open models, production-ready out of the box
• Multi-hardware support across NVIDIA, AMD, Intel, and CPU-only environments through a unified API
• Flexible deployment via Python package or Docker, with CUDA and ROCm builds available

For teams building on top of LLMs, vLLM fits naturally into LLM application frameworks and works alongside routing layers like LiteLLM or an LLM gateway for multi-provider setups. It's a common self-hosted alternative to managed inference services like Together AI.

Compared to tools like Ollama or llama.cpp, which prioritize ease of use on consumer hardware, vLLM targets production deployments where throughput per GPU matters. It's backed by compute resources from AWS, Google Cloud, NVIDIA, AMD, and others, and maintained by an active open-source community with support channels for both newcomers and teams running complex deployments.