koboldcpp

KoboldCpp is a single-file executable for running GGUF-quantized language models locally with a web UI optimized for creative writing and roleplay — combining the llama.cpp inference engine with a purpose-built interface featuring context shifting (smart memory management for long conversations), World Info (automatic lore injection when keywords appear), and an immersive chat mode, all packaged into a zero-installation binary that runs on any platform.

What Is KoboldCpp?

• Definition: A self-contained local LLM inference application that bundles llama.cpp, a web server, and a creative-writing-focused UI into a single executable file — no Python, no Docker, no installation required. Download the binary, download a GGUF model, double-click, and start writing.
• Creative Writing Focus: While Ollama and LM Studio target general chat, KoboldCpp is specifically designed for interactive fiction, roleplay, and creative writing — with features like World Info, Author's Note, and Memory that are essential for maintaining narrative coherence in long stories.
• Single Binary: The entire application (inference engine, web server, UI) is compiled into one file — the simplest possible deployment for non-technical users who want to run AI locally.
• KoboldAI Ecosystem: KoboldCpp is the llama.cpp-based successor to KoboldAI (which used Transformers) — maintaining compatibility with the KoboldAI API and the large community of creative writing tools built around it.

Key Features

• Context Shifting: When the conversation exceeds the model's context window, KoboldCpp intelligently shifts context by removing older messages while preserving the system prompt, World Info, and recent conversation — maintaining coherence without abruptly truncating.
• World Info: Define lore entries with trigger keywords — when a keyword appears in the conversation, the associated lore text is automatically injected into the context. Essential for maintaining consistent world-building in interactive fiction.
• Author's Note: A persistent instruction injected near the end of the context — guides the model's writing style, tone, or plot direction without being visible in the conversation.
• Memory: A persistent text block always included at the top of the context — used for character descriptions, setting details, and story summaries that should always be available to the model.
• Multiple Backends: Supports GGUF (llama.cpp), with optional CUDA GPU acceleration, Vulkan GPU support, and CLBlast for OpenCL devices.

KoboldCpp vs Alternatives

KoboldCpp is the zero-installation creative writing AI tool — packaging llama.cpp inference with narrative-focused features like World Info, context shifting, and immersive mode into a single executable that makes local AI storytelling accessible to anyone who can download two files.