Local models

Atomic Agent runs language models on your own machine by default. There are no per-token fees, your prompts never leave your computer, and you can swap models whenever you like. Under the hood it talks to llama.cpp’s llama-server over a small HTTP API.

This page explains the two ways to get a local model running — managed (Atomic Agent downloads and runs llama.cpp for you) and external (you point it at a llama-server you run yourself) — plus GPU acceleration, device selection, and the other surfaces that share the same model. If you’d rather call a hosted model, see cloud providers at the end.

Developer Preview

Atomic Agent is a Developer Preview. Model IDs, CLI flags, and config keys in this page are accurate to the current build but may shift between releases. Pin a release if you need stability.

Two modes at a glance

Managed

Atomic Agent downloads the llama.cpp binary and a GGUF model, picks a GPU, starts the daemon, and watches its health. One command and you’re running. Set localModels.mode: "managed".

External

You already run llama-server (or any OpenAI-compatible llama.cpp build) somewhere. Atomic Agent just connects to its URL. Set localModels.mode: "external".

The mode lives in your config file at <stateDir>/config.json:

{
  "localModels": {
    "mode": "managed",            // or "external"
    "url": "http://127.0.0.1:8080",
    "managed": {
      "modelId": "qwen-3.5-4b",
      "port": 19091
    }
  }
}

Note

localModels.url is the connection string Atomic Agent uses to reach the chat server. In managed mode it points at the daemon Atomic Agent spawns; in external mode it points at your own server. The defaults are 19091 for the managed chat daemon, 19092 for the embedding daemon, and http://127.0.0.1:8080 for localModels.url (llama.cpp’s own default, which is what an external llama-server listens on out of the box).

Managed mode (recommended)

In managed mode, Atomic Agent owns the full lifecycle of llama.cpp. The atomic-agent models commands drive everything.

Get a model running

# See which models are available and what's installed
atomic-agent models list

# Download a model's GGUF weights (idempotent — safe to re-run)
atomic-agent models pull qwen-3.5-4b

# Make it the active model
atomic-agent models use qwen-3.5-4b

# Start the daemon (downloads the llama.cpp backend if missing)
atomic-agent models start

# Check it's alive
atomic-agent models status

The default model for a fresh install is qwen-3.5-4b. The catalog spans the Qwen 3.5, 3.6, and 3.8 generations plus the Gemma 4 family — run atomic-agent models list for the current set rather than trusting a count here, since the catalog moves between releases.

Every chat model in the catalog is vision-capable, paired with an mmproj projector. There’s no separate “vision models” shortlist to pick from: whichever model you run can look at images, and the only cost is the extra memory the projector needs.

What models start actually does

When you start a managed daemon, Atomic Agent walks through a fixed sequence:

1. Verify prerequisites — checks the llama.cpp backend binary and the GGUF model are downloaded; fetches them if not.
2. Resolve a GPU device — enumerates GPUs and picks one (see GPU acceleration).
3. Spawn the daemon — launches llama-server detached, writes a PID file, and polls http://localhost:<port>/health until it returns ok (30s timeout).
4. Optionally start an embedding daemon — a second llama-server on a separate port for hybrid memory recall.

flowchart TD
    Start["atomic-agent models start"] --> Verify["Verify backend + GGUF<br/>(download if missing)"]
    Verify --> GPU["Resolve GPU device<br/>(Vulkan / nvidia-smi)"]
    GPU --> Spawn["Spawn llama-server detached<br/>write PID file"]
    Spawn --> Health["Poll /health<br/>until ok or 30s timeout"]
    Health -->|ok| ChatUp["Chat daemon up<br/>(primary)"]
    Health -->|timeout| Fail["Fail + print log tail"]
    ChatUp --> Embed{"Embeddings<br/>enabled?"}
    Embed -->|yes| EmbedDaemon["Spawn embedding daemon<br/>on separate port"]
    Embed -->|no| Done["Ready"]
    EmbedDaemon -->|ok| Done
    EmbedDaemon -->|fail| Degrade["Chat continues<br/>FTS5-only memory fallback"]
    Degrade --> Done

Chat is primary, embeddings are optional

The chat daemon is the primary process — if it fails to start, that’s fatal. The embedding daemon is a graceful add-on: if it fails, your chat agent keeps working and memory recall falls back to keyword (FTS5) search. You won’t be blocked.

Manage the daemon and models

atomic-agent models stop          # SIGTERM, wait 3s, then SIGKILL; remove PID file
atomic-agent models update        # update the llama.cpp backend binary
atomic-agent models remove <id>   # delete a model's weights from disk
atomic-agent models devices       # list detected GPU devices
atomic-agent models use-device auto   # or: cpu | Vulkan0 | Vulkan1 ...

Embedding models have their own subcommands:

atomic-agent models list-embeddings
atomic-agent models pull-embedding nomic-embed-text-v1.5
atomic-agent models use-embedding nomic-embed-text-v1.5
atomic-agent models use-embedding --disable

The default embedding model is nomic-embed-text-v1.5. The catalog carries five: one Nomic, three BGE (bge-small-en-v1.5, bge-base-en-v1.5, and the multilingual bge-m3), and one Jina long-document model. Reach for bge-m3 if your notes aren’t all in English.

Caution

The models subcommands only work in managed mode. In external mode, list, pull, use, start, stop, update, and remove expect you to be running llama-server yourself elsewhere.

External mode

If you already run llama-server (for example with a custom build, a shared GPU box, or a model Atomic Agent doesn’t bundle), use external mode and just point at it.

{
  "localModels": {
    "mode": "external",
    "url": "http://localhost:8080"
  }
}

Atomic Agent will probe that URL’s /health and /props endpoints at startup to detect the model profile and plan KV-cache slots. Everything else — tool grammar, hot-swap detection, prompting — works the same as managed mode.

Hot-swapping models behind llama-server

Atomic Agent watches the modelId field on each completion response. If you swap the model behind your llama-server mid-session (e.g. from Qwen to Gemma 4), it detects the mismatch, re-probes /props, rebuilds the tool-call grammar, and adapts — no restart needed. This relies on the modelId field, which older llama.cpp builds may not send.

GPU acceleration and device selection

Atomic Agent uses Vulkan for GPU acceleration in managed mode, which covers all major vendors (NVIDIA, AMD, Intel) through a single path. It enumerates devices by running llama-server --list-devices and parsing the Vulkan table, with an nvidia-smi fallback for pre-download VRAM checks.

atomic-agent models devices

How a device is chosen

• auto (default) — Atomic Agent picks the best GPU by VRAM, preferring a discrete GPU over an integrated one even if the integrated GPU reports more memory.
• A specific device — e.g. Vulkan0, Vulkan1.
• cpu — skip the GPU entirely.

atomic-agent models use-device auto
atomic-agent models use-device Vulkan0
atomic-agent models use-device cpu

The setting persists as localModels.managed.device in config.

VRAM budgeting

Atomic Agent computes a VRAM budget before loading:

• macOS uses 75% of system RAM, because Apple Silicon has unified memory. The conservative fraction avoids false “insufficient VRAM” warnings.
• Linux / Windows use the selected device’s reported VRAM.
• CPU mode has no GPU budget.

Vision needs headroom

Every catalog model loads extra CLIP projector weights and runs with special batch flags (--image-min-tokens 560 and friends) so image embeddings aren’t near-noise. Budget for the mmproj projector on top of the weights themselves.

Context auto-sizing

localModels.managed.contextSize defaults to 0, which means auto: Atomic Agent measures the free VRAM it has after the weights load and fits --ctx-size to what’s actually left. On a busy GPU you get a smaller window instead of a failed load. Set a positive integer to pin the context yourself when you’d rather have a predictable window than a guaranteed start.

The daemon’s lifetime

localModels.managed.stopOnExit defaults to true: when the last CLI session exits, the managed daemon shuts down with it rather than lingering in the background holding VRAM. Set it to false if you want the daemon to survive across invocations — worth it when you’re running many short commands and don’t want to pay the model load each time.

Reaching the same model from other surfaces

Whatever model you’re running — managed or external — is shared across every way you talk to Atomic Agent. They all connect to the same llama-server.

OpenAI-compatible HTTP

Run an OpenAI-compatible HTTP server and point any OpenAI SDK at it:

atomic-agent serve --host 127.0.0.1 --port 8787 --api-key <key>

Then call the chat completions endpoint:

curl http://127.0.0.1:8787/v1/chat/completions \
  -H "Authorization: Bearer <key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "atomic-agent",
    "messages": [{ "role": "user", "content": "List the files here." }],
    "stream": false
  }'

Notes:

• POST /v1/chat/completions supports both streaming (stream: true, SSE) and synchronous responses.
• GET /v1/models returns a single atomic-agent entry and needs no auth.
• GET /health is a no-auth liveness probe (also checks the underlying llama-server).
• Session IDs are derived deterministically from the prompt, or set explicitly with the X-Atomic-Session-Id header.
• The API key can also come from the ATOMIC_AGENT_API_KEY environment variable.

Tauri sidecar

For desktop apps, Atomic Agent ships a sidecar process that speaks newline-delimited JSON (NDJSON) over stdin/stdout. A Tauri (or any) host spawns it, pipes the streams, sends typed requests, and reads back streamed events.

atomic-agent-sidecar

• One active session per sidecar process. Start a new session and the previous one is disposed.
• Requests: start_session, send_message, cancel, approval_response, get_session, skill_*, ping, shutdown.
• Events stream back as you go: step_started, tool_call_started, assistant_delta, assistant_reply, approval_request, and more.
• The sidecar reads the same localModels.url / localModels.managed.port config as every other front end.

The sidecar uses the same managed/external model under the hood — it’s just another front end for the runtime.

Telegram

You can drive the same agent — and the same local model — from your phone via a single-user Telegram channel.

{
  "telegram": { "enabled": true }
}

Put your bot token in <stateDir>/.env as TELEGRAM_BOT_TOKEN (never in config.json — the token lives in the secrets file, mode 0600). Then pair as the owner and chat. Dangerous tool calls arrive as inline approve/deny buttons.

/help     show commands and setup
/status   active session, turn count, last error
/new      start a fresh session
/cancel   abort the running turn

Telegram is intentionally single-user: once paired, only the owner’s messages are accepted; everyone else is dropped.

Configuration reference

Key	Meaning
localModels.mode	managed (Atomic Agent runs llama.cpp) or external (you do)
localModels.url	HTTP URL of the chat server
localModels.managed.modelId	Active model in managed mode (e.g. qwen-3.5-4b)
localModels.managed.port	Chat daemon port (default 19091)
localModels.managed.device	auto, cpu, or a device id like Vulkan0
localModels.managed.contextSize	--ctx-size; 0 (default) fits it to free VRAM
localModels.managed.stopOnExit	Stop the daemon when the last CLI session exits (default true)
localModels.managed.autoUpdate	Auto-update the backend binary
localModels.completionMaxTokens	Max new tokens per completion (default 8192)
localModels.embeddings.enabled	Run the secondary embedding daemon
localModels.embeddings.modelId	Embedding model (e.g. nomic-embed-text-v1.5)
localModels.embeddings.port	Embedding daemon port (default 19092)

Useful environment variables:

Variable	Effect
ATOMIC_AGENT_LLAMA_API_KEY	Bearer token for the llama-server (optional)
ATOMIC_AGENT_LLAMA_MAX_TOKENS	Max new tokens per completion — sets localModels.completionMaxTokens (default 8192, clamped 64–131072)
ATOMIC_AGENT_LLAMA_TEMPERATURE / _TOP_P / _TOP_K / _SEED	Sampling overrides (parsed at startup)
ATOMIC_AGENT_STATE_DIR	Root state directory holding config.json, .env, and model files

Cloud providers (opt-in)

Local llama.cpp is the default and the point of the product, but it isn’t the only option. Atomic Agent ships six provider kinds, and you can mix them:

Kind	What it talks to
llama-server	Local llama.cpp — the default
openai-compatible	Any OpenAI-shaped endpoint (vLLM, LM Studio, a hosted API)
qwen-openai-compatible	The same, with Qwen’s tagged tool-call dialect
openrouter	OpenRouter’s model marketplace
aimlapi	AIMLAPI
gemini	Google Gemini

Ready-made presets

You rarely need to type a base URL. The TUI’s provider wizard ships eleven pre-verified presets — pick a vendor by name and it fills in the endpoint for you. Every one resolves to the ordinary openai-compatible kind, so a preset is a shortcut, not a seventh provider kind, and model lists still come from the server’s own /v1/models rather than a bundled list that goes stale.

Preset	Base URL	API-key env var
Cerebras	https://api.cerebras.ai	CEREBRAS_API_KEY
DeepSeek	https://api.deepseek.com	DEEPSEEK_API_KEY
Fireworks AI	https://api.fireworks.ai/inference	FIREWORKS_API_KEY
Groq	https://api.groq.com/openai	GROQ_API_KEY
LM Studio (local)	http://localhost:1234	LMSTUDIO_API_KEY
Mistral	https://api.mistral.ai	MISTRAL_API_KEY
Nous Research	https://inference-api.nousresearch.com	NOUS_API_KEY
Ollama (local)	http://localhost:11434	OLLAMA_API_KEY
Ollama Cloud	https://ollama.com	OLLAMA_CLOUD_API_KEY
Together AI	https://api.together.xyz	TOGETHER_API_KEY
xAI (Grok)	https://api.x.ai	XAI_API_KEY

Each preset has its own env var, deliberately. An earlier shared OPENAI_COMPAT_API_KEY meant connecting a second vendor overwrote the first one’s key. Now Groq and Together can hold keys at the same time without touching each other, and adding a second key for the same vendor gets a numbered entry id (groq-2) that still resolves to the right service.

Two behaviours worth knowing:

• The two local presets — LM Studio and Ollama — point at a server on your own machine. There is no key to have, so the wizard saves them with an empty key and requests carry no Authorization header at all.
• Some presets list models before you paste a key. Nous Research (350+ ids) and Ollama Cloud answer /v1/models unauthenticated, so you can browse the catalog first and add credentials later. The rest want a key before they’ll tell you anything.

Cloud providers send your prompts off your machine

This is the one setting that breaks the local-first guarantee, so it is strictly opt-in — nothing here is active until you add an llm.providers[] entry and point activeTextProvider at it. Once you do, every prompt routed to that provider, including whatever memory and file contents the agent injected, goes to a third party under their retention policy. Keep sensitive work on a local provider.

There is no atomic-agent llm command

Cloud providers are configured in exactly two places: the llm block of config.json, or the TUI’s LLM panel (which writes the same block). The CLI surface is run, skill, config, repl, tui, serve, trace, task, models, and import — models drives the local llama.cpp daemon only. If you’re looking for a command to add a hosted key, it doesn’t exist.

Providers are declared in an llm.providers[] block, with activeTextProvider choosing which one handles a turn:

{
  "llm": {
    "activeTextProvider": "local",
    "providers": [
      { "id": "local", "kind": "llama-server" },
      {
        "id": "or",
        "kind": "openrouter",
        "apiKey": "sk-or-...",
        "defaultChatModel": "qwen/qwen3-max"
      }
    ]
  }
}

What goes in a provider entry

That example is the short form. A provider entry carries considerably more than id / kind / apiKey:

Field	Meaning
id	Name you refer to in activeTextProvider, activeEmbeddingProvider, and llm.fallback.chain
kind	One of the six kinds above
baseUrl	API root for the OpenAI-shaped kinds. Stored without the /v1 suffix — call sites append it
url	llama-server base URL override (llama-server kind only)
apiKey	Credential. Empty string when the endpoint is local and unauthenticated
defaultChatModel	Model used when a turn doesn’t name one
defaultEmbeddingModel	Model used for embedding requests
headers	Extra HTTP headers sent verbatim on every request
supportsTools	Whether the endpoint handles parallel tool calls (defaults to true)
supportsVision	Whether the endpoint accepts images (defaults to true)
requestTimeoutMs	Per-request timeout
promptCache	auto, off, or explicit-markers
providerPreferences	Free-form object passed through to the provider (OpenRouter routing preferences and similar)
userModels[]	Per-model capability and pricing declarations — see below

openai-compatible and qwen-openai-compatible refuse to boot without two fields

Both kinds require baseUrl and defaultChatModel. Leave either out and the provider factory throws at startup — openai-compatible provider "<id>" requires baseUrl and defaultChatModel — which takes the whole runtime down, not just that provider. A minimal working entry is four keys:

{
  "id": "vllm",
  "kind": "openai-compatible",
  "baseUrl": "https://vllm.internal.example",
  "apiKey": "sk-...",
  "defaultChatModel": "qwen/qwen3-32b"
}

The other four kinds are more forgiving: openrouter, aimlapi, and gemini each fall back to a built-in default chat model, and llama-server reads its URL from localModels.

Declaring models yourself

Each provider entry can carry a userModels[] array describing individual models. Every entry needs an id and a kind (chat or embedding); the rest is optional:

{
  "id": "vllm",
  "kind": "openai-compatible",
  "baseUrl": "https://vllm.internal.example",
  "defaultChatModel": "qwen/qwen3-32b",
  "userModels": [
    {
      "id": "qwen/qwen3-32b",
      "kind": "chat",
      "contextWindow": 32768,
      "supportsVision": false,
      "supportsTools": "basic",
      "supportsPromptCache": false,
      "pricing": { "input": 0.15, "output": 0.6 }
    }
  ]
}

supportsTools is a four-way setting — none, basic, parallel, or strict — not a boolean, and embedding entries can also declare dim. Pricing takes input and output, plus optional cacheRead and cacheWrite.

Resolution runs in a fixed order: userModels beats the built-in catalog, which beats the defaults. That last step is the one to watch. A model nobody declared and the catalog doesn’t recognise silently resolves to a 128,000-token context window with parallel tool support and no vision — plausible-looking numbers that are simply wrong if your endpoint serves an 8K window or can’t do parallel calls. Nothing warns you; you find out when requests start getting truncated or tool calls come back malformed.

userModels[] is also the only way to make cost tracking price a model the catalog doesn’t know. That’s the fix for the $0 readout described just below — declare pricing here and the numbers become real.

How tool calls travel

llm.toolTransport sits next to providers[] and decides how the model is asked for a tool call. It takes three values, and the default is easy to misread:

Value	Behaviour
auto (default)	Defers to whatever the active provider declares. Not a synonym for grammar
grammar	Force GBNF-constrained decoding, whatever the provider says
native_tools	Force the provider’s own function-calling API

Because auto follows the provider, the effective transport changes when you switch providers — and it changes again mid-chain when a fallback hands the turn to someone else. A local llama-server turn constrained by grammar and a hosted turn using native function calling are both auto. If you’re debugging tool-call behaviour that appears only on one provider, pin the transport explicitly before you go hunting anywhere else.

Two things come with the cloud path.

Cost tracking accumulates spend per turn once you set llm.costTracking.enabled: true, so a remote provider’s bill is visible rather than a surprise. It prices a turn from the model’s published rates; for a model the built-in catalog doesn’t carry, declare the rates yourself in that provider’s userModels[].

A fallback chain lets a request roll to the next provider when the active one errors — useful for riding out a hosted outage without dropping back to a cold local load, though it also means a failure can quietly move your prompt to a different vendor. Put only providers you’re equally comfortable with in one chain.

Note that llm.fallback.appendLocal defaults to true: your local llama-server is silently appended as the last link, so a cloud outage can hand the turn to a local model whose answers differ sharply in quality. Set it to false if you would rather the request fail than switch. The chain also behaves as a circuit breaker — after failureThreshold consecutive failures (default 3) the provider is skipped for a cooldown that escalates 30s, 60s, 300s before the primary is probed again.

Troubleshooting

• Daemon won’t come up. atomic-agent models start polls /health for 30 seconds; on timeout it prints the tail of the llama-server log. Re-run atomic-agent models status and check the log for load errors.
• Foreign daemon error. If a daemon was started by another user (e.g. with sudo) and you try to stop it as yourself, Atomic Agent raises a ForeignDaemonError rather than orphaning the process. Stop it as the original user.
• Embeddings failed but chat works. That’s by design — memory recall silently falls back to keyword (FTS5) search. Re-pull the embedding model or check its daemon port.
• Garbage memory recall. An embedding model’s pooling strategy must match how it was trained; the catalog encodes this. Use the catalog model IDs rather than hand-pointing at an arbitrary GGUF.
• GitHub rate limits during download. Backend downloads hit the GitHub API (unauthenticated ~60 req/h). Set GITHUB_TOKEN for heavy use or CI. GH_TOKEN works everywhere GITHUB_TOKEN does — model downloads, llama.cpp backend installs, and the app update check all read GITHUB_TOKEN first and fall back to GH_TOKEN, so the token the gh CLI already put in your environment is enough.

Related

CLI reference

Full Atomic Agent models, run, and serve command surface.

Configuration

Every config.json block and ATOMIC_AGENT_* environment variable.

Memory

How the embedding daemon powers hybrid recall over your notes.

HTTP API

OpenAI-compatible endpoints, headers, and streaming details.