Use with llama.cpp

Short answer. llama-server is llama.cpp’s HTTP server. Start it with a GGUF model and an explicit port:

llama-server -hf ggml-org/gemma-3-4b-it-GGUF --port 8080 -c 32768

Then check it:

curl http://127.0.0.1:8080/health

{"status":"ok"} means the model is loaded and the server is ready; a 503 with Loading model means wait. Atomic Agent connects to llama.cpp in two ways: it can run llama.cpp for you (managed mode, which you turn on by choosing Local models on first run or with atomic-agent models use), or it can use a llama-server you already run (External in the terminal, Custom server in the desktop app).

The default port changed in October 2026

llama.cpp builds from 9 October 2026 on listen on port 9931 by default (PR #30159); older builds used 8080. Atomic Agent’s default External URL is still http://127.0.0.1:8080. Either pass --port 8080 as above, or set the agent’s URL to the port your build prints at startup. The official llama.cpp Docker images still listen on 8080 inside the container (Docker docs).

llama-server cheat sheet

From the llama.cpp server README:

Flag	What it does
-m path/to/model.gguf	Loads a local GGUF file
-hf user/repo[:quant]	Downloads a GGUF from Hugging Face (Q4_K_M by default)
--port N / --host ADDR	Port and bind address (host defaults to 127.0.0.1)
-c N	Context size in tokens (0 = take it from the model)
-ngl N	Layers to put on the GPU (auto by default)
--api-key KEY	Requires a key on requests (/health stays open)

Install options (install script, prebuilt binaries, Docker, building from source) are in the llama.cpp README. Recent builds also ship a llama serve command that starts the same OpenAI-compatible server.

Common llama-server errors

• Connection refused on 8080. The server runs on a different port (9931 on recent builds) or is not running. Read the port from its startup log, or restart it with --port 8080.
• 503 Loading model. The server is up but still loading weights. Wait and check /health again.
• 401 on chat requests while /health answers. The server was started with --api-key. llama.cpp leaves /health (and /v1/health) open without a key, so a health check passes and real requests fail.
• Out of memory at load. The model plus context do not fit. Lower -c, choose a smaller quant, or reduce -ngl.

Option 1: let Atomic Agent run llama.cpp (managed mode)

This is the path that needs no separate server. It is not on by default: a fresh config points at an external server on http://127.0.0.1:8080, and managed mode starts when you pick Local models on first run or run atomic-agent models use <id>. The agent downloads GGUF models from Hugging Face, installs a llama.cpp backend and starts it on 127.0.0.1:19091. The engine is Atomic Core by default; atomic-agent models engine llama-server switches to the agent’s own llama-server engine, which downloads AtomicBot’s TurboQuant build of llama.cpp.

• Desktop: pick Local models on the first-run screen, or Settings › Models › Local models, and download a model.

• Terminal:

  atomic-agent models list
  atomic-agent models pull qwen-3.5-4b
  atomic-agent models use qwen-3.5-4b
  atomic-agent models status

  models use switches the config to managed mode. Inside the TUI, /model opens the same model picker.

Option 2: connect a llama-server you run yourself

Use this when you already run llama-server with your own flags, on this machine or another one.

Desktop app

1. Start llama-server and confirm /health returns {"status":"ok"}.

2. On the first-run screen choose Custom endpoint (“A llama.cpp server you already run”), or later open Settings › Models › Custom server.

3. Enter the base URL, for example http://127.0.0.1:8080. The app probes the server before it saves anything.

Terminal

1. Start llama-server and check /health.

2. In the TUI, run /model http://127.0.0.1:8080, or open /llm, go to the External pane and enter the URL.

3. Check the route with atomic-agent models status, which prints mode: external and the URL.

Both surfaces write localModels.mode = "external", localModels.url and the url of the built-in local-llama provider together. If you edit config.json by hand, change both URLs; atomic-agent config set localModels.url ... updates only the first one.

If your server uses --api-key

Put the same key in <stateDir>/.env (default ~/.atomic-agent/.env) as ATOMIC_AGENT_LLAMA_API_KEY=... and restart the agent. For a server that Atomic Agent started itself, the key is in <models dir>/llama-server.key.

Ollama, LM Studio and vLLM do not go here

The External route uses llama.cpp’s own endpoints (/health, /props, /slots, /apply-template) on top of the OpenAI-style ones. When the URL answers like a different OpenAI-compatible server, the agent refuses it and points you to the provider list instead: Ollama, LM Studio, vLLM.

What you get on the llama.cpp route

On the llama.cpp route (managed or External), Atomic Agent constrains tool calls with a GBNF grammar, so the model’s output is held to the tool-call format while it is generated. On OpenAI-compatible providers such as Ollama, LM Studio and vLLM, tools go through the API’s tools field and the server and model decide the format. More on the grammar approach: GBNF grammar-constrained tool calling.

FAQ

What is the default llama.cpp server port?

9931 for builds from 9 October 2026 on, 8080 before that. The official Docker images still listen on 8080 inside the container. Set it explicitly with --port.

How do I check that llama-server is running?

curl http://127.0.0.1:<port>/health. {"status":"ok"} means ready; a 503 with Loading model means it is still loading.

Do I need to run llama-server myself to use Atomic Agent locally?

No. In managed mode the agent installs llama.cpp, downloads the model and starts the server itself. Run your own only when you want full control over its flags.

Can I point Atomic Agent at llama-server through the OpenAI-compatible provider?

You can add any /v1 server that way, but for llama.cpp use the External route: it is the one that uses llama.cpp’s own endpoints and grammar-constrained tool calls.

Where are the agent’s downloaded models?

In ~/.atomic-agent/models (the models folder of the state dir), unless you set localModels.managed.dataDirOverride. atomic-agent models list shows which models are on disk and which one is active.

Related

• Use with Ollama, Use with LM Studio, Use with vLLM
• Ollama vs llama.cpp
• Models