llama.cpp

llama.cpp is a C/C++ inference engine for running large language models locally or in the cloud, with no external dependencies. It targets developers, researchers, and anyone who wants to run open-weight models on their own hardware without relying on cloud APIs.

The project's defining strength is hardware breadth. It runs on Apple Silicon via Metal and ARM NEON, NVIDIA GPUs via custom CUDA kernels, AMD GPUs via HIP, Intel hardware via SYCL, and a long list of other backends including Vulkan, Ascend NPU, and Snapdragon. CPU-only inference works too, and a hybrid CPU+GPU mode lets you run models larger than your available VRAM by splitting the load.

Quantization is a core feature. Models can be stored and run at 1.5-bit through 8-bit integer precision, dramatically reducing memory requirements while keeping inference fast. The GGUF format is the standard file format for these quantized models, and Hugging Face hosts a large library of compatible weights.

Key capabilities include:

• OpenAI-compatible HTTP server (llama-server) with multi-user parallel decoding, speculative decoding, embedding endpoints, and reranking support
• Grammar-constrained output for structured generation, including JSON, via custom GBNF grammars
• Multimodal support for vision-language models alongside dozens of text-only architectures including LLaMA 3, Mistral, Qwen, Gemma, Phi, DeepSeek, and many more
• for downloading models directly by name, with models stored in the standard HF cache so they're shareable with other tools