Barrel Inference Server¶

OpenAI-, Anthropic-, and Ollama-compatible HTTP server on top of Barrel Inference. One Erlang/OTP node; real GGUF inference via llama.cpp under the hood. Drop-in for SDKs that already speak any of those three APIs.

Three API families, one process tree¶

┌───────────── barrel_inference_server (Erlang/OTP) ─────────────┐
│                                                       │
│  OpenAI     /v1/chat/completions  /v1/completions     │
│             /v1/embeddings        /v1/models[/:id]    │
│                                                       │
│  Anthropic  /v1/messages                              │
│                                                       │
│  Ollama     /api/generate  /api/chat  /api/embed      │
│             /api/pull  /api/tags  /api/show  /api/copy │
│             /api/delete  /api/create  /api/ps         │
│             /api/version  /api/search                 │
│                                                       │
│  health     /health  /health/ready  /metrics          │
└───────────────────────────┬───────────────────────────┘
                            ▼
              barrel_inference (NIF over llama.cpp)

Pick your reading path¶

New to it? Start with Quickstart.
Want every endpoint as a curl one-liner? HTTP API reference.
Hooking up an SDK? Client examples (Python OpenAI / Anthropic / ollama, JavaScript fetch, LangChain, LiteLLM, Claude Code).
Running tools server-side (web_search)? Server-side tools.
Picking a model that fits your laptop? Sizing guide.
Pulling and managing models? Registry guide.
Downloading internals? Fetching guide.
Machine-readable? OpenAPI 3.1 spec.

Highlights¶

Three API families side by side with one supervised process tree.
Tool calling via grammar-constrained sampling on the OpenAI and Anthropic paths.
Structured output (response_format on OpenAI, format on Ollama). "json" and JSON Schema both compile to GBNF.
Model registry with content-addressed blob cache. Pull from HuggingFace, Ollama, plain HTTPS, or file://. Resumable, sha256-verified.
Modelfile-compatible /api/create with FROM, PARAMETER, SYSTEM, TEMPLATE directives.
Keep-alive eviction counted by active requests so a long generation never trips the unload timer.
Cancel-on-disconnect propagates back to llama.cpp.
Prometheus /metrics, CORS preflight, X-Request-ID echo.

Project links¶

Source on GitHub: https://github.com/barrel-platform/barrel_inference
These docs: https://barrel-platform.github.io/barrel_inference/
Issues: https://github.com/barrel-platform/barrel_inference/issues
Releases: https://github.com/barrel-platform/barrel_inference/releases