LLM inference in C/C++

High-speed Large Language Model Serving for Local Deployment

Run GGUF models easily with a KoboldAI UI. One File. Zero Install.

Lightweight inference library for ONNX files, written in C++. It can run Stable Diffusion XL 1.0 on a RPI Zero 2 (or in 298MB of RAM) but also Mistral 7B on desktops and servers. ARM, x86, WASM, RISC-V supported. Accelerated by XNNPACK.

A highly optimized LLM inference acceleration engine for Llama and its variants.

Fast Multimodal LLM on Mobile Devices

🤘 TT-NN operator library, and TT-Metalium low level kernel programming model.

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

React Native binding of llama.cpp

A high-performance inference system for large language models, designed for production environments.

CPU inference for the DeepSeek family of large language models in pure C++

Yet Another Language Model: LLM inference in C++/CUDA, no libraries except for I/O

Swift library to work with llama and other large language models.

Simple chat program for LLaMa, GPT-J, and MPT models.

llama.cpp (GGUF LLMs) and llava.cpp (GGUF VLMs) for ROS 2

Kolosal AI is an OpenSource and Lightweight alternative to LM Studio to run LLMs 100% offline on your device.

LLaVA server (llama.cpp).

Booster - open accelerator for LLM models. Better inference and debugging for AI hackers

UnrealMCP is here!! Automatic blueprint and scene generation from AI!! An Unreal Engine plugin for LLM/GenAI models & MCP UE5 server. Supports Claude Desktop App, Windsurf & Cursor, also includes OpenAI's GPT4o, DeepseekR1, Claude Sonnet 3.7 APIs and Grok 3, with plans to add Gemini, audio & realtime APIs soon.