Local models

Atomic Agent runs language models on your own machine. There are no per-token fees, your prompts never leave your computer by default, and you can swap models whenever you like. Under the hood it talks to llama.cpp’s llama-server over a small HTTP API.

This page explains the two ways to get a model running — managed (Atomic Agent downloads and runs llama.cpp for you) and external (you point it at a llama-server you run yourself) — plus GPU acceleration, device selection, and the other surfaces that share the same model.

Developer Preview

Atomic Agent is a Developer Preview. Model IDs, CLI flags, and config keys in this page are accurate to the current build but may shift between releases. Pin a release if you need stability.

Two modes at a glance

Managed

Atomic Agent downloads the llama.cpp binary and a GGUF model, picks a GPU, starts the daemon, and watches its health. One command and you’re running. Set localModels.mode: "managed".

External

You already run llama-server (or any OpenAI-compatible llama.cpp build) somewhere. Atomic Agent just connects to its URL. Set localModels.mode: "external".

The mode lives in your config file at <stateDir>/config.json:

{
  "localModels": {
    "mode": "managed",            // or "external"
    "url": "http://localhost:8000",
    "managed": {
      "modelId": "qwen-3.5-4b",
      "port": 8000
    }
  }
}

Note

localModels.url is the connection string Atomic Agent uses to reach the chat server. In managed mode it points at the daemon Atomic Agent spawns; in external mode it points at your own server. You can also override it with the ATOMIC_AGENT_LLAMA_URL environment variable.

Managed mode (recommended)

In managed mode, Atomic Agent owns the full lifecycle of llama.cpp. The atomic-agent models commands drive everything.

Get a model running

# See which models are available and what's installed
atomic-agent models list

# Download a model's GGUF weights (idempotent — safe to re-run)
atomic-agent models pull qwen-3.5-4b

# Make it the active model
atomic-agent models use qwen-3.5-4b

# Start the daemon (downloads the llama.cpp backend if missing)
atomic-agent models start

# Check it's alive
atomic-agent models status

The default model for a fresh install is qwen-3.5-4b. The catalog ships 8 chat models (Qwen 3.5 / 3.6 and Gemma 4 variants); run atomic-agent models list to see the current set.

What models start actually does

When you start a managed daemon, Atomic Agent walks through a fixed sequence:

1. Verify prerequisites — checks the llama.cpp backend binary and the GGUF model are downloaded; fetches them if not.
2. Resolve a GPU device — enumerates GPUs and picks one (see GPU acceleration).
3. Spawn the daemon — launches llama-server detached, writes a PID file, and polls http://localhost:<port>/health until it returns ok (30s timeout).
4. Optionally start an embedding daemon — a second llama-server on a separate port for hybrid memory recall.

flowchart TD
    Start["atomic-agent models start"] --> Verify["Verify backend + GGUF<br/>(download if missing)"]
    Verify --> GPU["Resolve GPU device<br/>(Vulkan / nvidia-smi)"]
    GPU --> Spawn["Spawn llama-server detached<br/>write PID file"]
    Spawn --> Health["Poll /health<br/>until ok or 30s timeout"]
    Health -->|ok| ChatUp["Chat daemon up<br/>(primary)"]
    Health -->|timeout| Fail["Fail + print log tail"]
    ChatUp --> Embed{"Embeddings<br/>enabled?"}
    Embed -->|yes| EmbedDaemon["Spawn embedding daemon<br/>on separate port"]
    Embed -->|no| Done["Ready"]
    EmbedDaemon -->|ok| Done
    EmbedDaemon -->|fail| Degrade["Chat continues<br/>FTS5-only memory fallback"]
    Degrade --> Done

Chat is primary, embeddings are optional

The chat daemon is the primary process — if it fails to start, that’s fatal. The embedding daemon is a graceful add-on: if it fails, your chat agent keeps working and memory recall falls back to keyword (FTS5) search. You won’t be blocked.

Manage the daemon and models

atomic-agent models stop          # SIGTERM, wait 3s, then SIGKILL; remove PID file
atomic-agent models update        # update the llama.cpp backend binary
atomic-agent models remove <id>   # delete a model's weights from disk
atomic-agent models devices       # list detected GPU devices
atomic-agent models use-device auto   # or: cpu | Vulkan0 | Vulkan1 ...

Embedding models have their own subcommands:

atomic-agent models list-embeddings
atomic-agent models pull-embedding nomic-embed-text-v1.5
atomic-agent models use-embedding nomic-embed-text-v1.5
atomic-agent models use-embedding --disable

The default embedding model is nomic-embed-text-v1.5; the catalog has 5 (Nomic, BGE, Jina variants).

Caution

The models subcommands only work in managed mode. In external mode, list, pull, use, start, stop, update, and remove expect you to be running llama-server yourself elsewhere.

External mode

If you already run llama-server (for example with a custom build, a shared GPU box, or a model Atomic Agent doesn’t bundle), use external mode and just point at it.

{
  "localModels": {
    "mode": "external",
    "url": "http://localhost:8080"
  }
}

Atomic Agent will probe that URL’s /health and /props endpoints at startup to detect the model profile and plan KV-cache slots. Everything else — tool grammar, hot-swap detection, prompting — works the same as managed mode.

Hot-swapping models behind llama-server

Atomic Agent watches the modelId field on each completion response. If you swap the model behind your llama-server mid-session (e.g. from Qwen to Gemma 4), it detects the mismatch, re-probes /props, rebuilds the tool-call grammar, and adapts — no restart needed. This relies on the modelId field, which older llama.cpp builds may not send.

GPU acceleration and device selection

Atomic Agent uses Vulkan for GPU acceleration in managed mode, which covers all major vendors (NVIDIA, AMD, Intel) through a single path. It enumerates devices by running llama-server --list-devices and parsing the Vulkan table, with an nvidia-smi fallback for pre-download VRAM checks.

atomic-agent models devices

How a device is chosen

• auto (default) — Atomic Agent picks the best GPU by VRAM, preferring a discrete GPU over an integrated one even if the integrated GPU reports more memory.
• A specific device — e.g. Vulkan0, Vulkan1.
• cpu — skip the GPU entirely.

atomic-agent models use-device auto
atomic-agent models use-device Vulkan0
atomic-agent models use-device cpu

The setting persists as localModels.managed.device in config.

VRAM budgeting

Atomic Agent computes a VRAM budget before loading:

• macOS uses 75% of system RAM, because Apple Silicon has unified memory. The conservative fraction avoids false “insufficient VRAM” warnings.
• Linux / Windows use the selected device’s reported VRAM.
• CPU mode has no GPU budget.

Vision models need more headroom

Vision-capable models load extra CLIP projector weights and run with special batch flags (--image-min-tokens 560 and friends) so image embeddings aren’t near-noise. Expect higher memory use when you pull a vision model and its mmproj projector.

Reaching the same model from other surfaces

Whatever model you’re running — managed or external — is shared across every way you talk to Atomic Agent. They all connect to the same llama-server.

OpenAI-compatible HTTP

Run an OpenAI-compatible HTTP server and point any OpenAI SDK at it:

atomic-agent serve --host 127.0.0.1 --port 8787 --api-key <key>

Then call the chat completions endpoint:

curl http://127.0.0.1:8787/v1/chat/completions \
  -H "Authorization: Bearer <key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "atomic-agent",
    "messages": [{ "role": "user", "content": "List the files here." }],
    "stream": false
  }'

Notes:

• POST /v1/chat/completions supports both streaming (stream: true, SSE) and synchronous responses.
• GET /v1/models returns a single atomic-agent entry and needs no auth.
• GET /health is a no-auth liveness probe (also checks the underlying llama-server).
• Session IDs are derived deterministically from the prompt, or set explicitly with the X-Atomic-Session-Id header.
• The API key can also come from the ATOMIC_AGENT_API_KEY environment variable.

Tauri sidecar

For desktop apps, Atomic Agent ships a sidecar process that speaks newline-delimited JSON (NDJSON) over stdin/stdout. A Tauri (or any) host spawns it, pipes the streams, sends typed requests, and reads back streamed events.

atomic-agent-sidecar

• One active session per sidecar process. Start a new session and the previous one is disposed.
• Requests: start_session, send_message, cancel, approval_response, get_session, skill_*, ping, shutdown.
• Events stream back as you go: step_started, tool_call_started, assistant_delta, assistant_reply, approval_request, and more.
• The llama.cpp server URL defaults to localhost:8000 and is overridable with ATOMIC_AGENT_LLAMA_URL.

The sidecar uses the same managed/external model under the hood — it’s just another front end for the runtime.

Telegram

You can drive the same agent — and the same local model — from your phone via a single-user Telegram channel.

{
  "telegram": { "enabled": true }
}

Put your bot token in <stateDir>/.env as TELEGRAM_BOT_TOKEN (never in config.json — the token lives in the secrets file, mode 0600). Then pair as the owner and chat. Dangerous tool calls arrive as inline approve/deny buttons.

/help     show commands and setup
/status   active session, turn count, last error
/new      start a fresh session
/cancel   abort the running turn

Telegram is intentionally single-user: once paired, only the owner’s messages are accepted; everyone else is dropped.

Configuration reference

Key	Meaning
localModels.mode	managed (Atomic Agent runs llama.cpp) or external (you do)
localModels.url	HTTP URL of the chat server
localModels.managed.modelId	Active model in managed mode (e.g. qwen-3.5-4b)
localModels.managed.port	Chat daemon port
localModels.managed.device	auto, cpu, or a device id like Vulkan0
localModels.managed.autoUpdate	Auto-update the backend binary
localModels.embeddings.enabled	Run the secondary embedding daemon
localModels.embeddings.modelId	Embedding model (e.g. nomic-embed-text-v1.5)
localModels.embeddings.port	Embedding daemon port

Useful environment variables:

Variable	Effect
ATOMIC_AGENT_LLAMA_URL	Override the llama-server connection string
ATOMIC_AGENT_LLAMA_API_KEY	Bearer token for the llama-server (optional)
ATOMIC_AGENT_LLAMA_MAX_TOKENS	Max new tokens per completion (default 4096, clamped 64–131072)
ATOMIC_AGENT_LLAMA_TEMPERATURE / _TOP_P / _TOP_K / _SEED	Sampling overrides (parsed at startup)
ATOMIC_AGENT_STATE_DIR	Root state directory holding config.json, .env, and model files

Troubleshooting

• Daemon won’t come up. atomic-agent models start polls /health for 30 seconds; on timeout it prints the tail of the llama-server log. Re-run atomic-agent models status and check the log for load errors.
• Foreign daemon error. If a daemon was started by another user (e.g. with sudo) and you try to stop it as yourself, Atomic Agent raises a ForeignDaemonError rather than orphaning the process. Stop it as the original user.
• Embeddings failed but chat works. That’s by design — memory recall silently falls back to keyword (FTS5) search. Re-pull the embedding model or check its daemon port.
• Garbage memory recall. An embedding model’s pooling strategy must match how it was trained; the catalog encodes this. Use the catalog model IDs rather than hand-pointing at an arbitrary GGUF.
• GitHub rate limits during download. Backend downloads hit the GitHub API (unauthenticated ~60 req/h). Set GITHUB_TOKEN for heavy use or CI.

Related

CLI reference

Full Atomic Agent models, run, and serve command surface.

Configuration

Every config.json block and ATOMIC_AGENT_* environment variable.

Memory

How the embedding daemon powers hybrid recall over your notes.

HTTP API

OpenAI-compatible endpoints, headers, and streaming details.