How to Run an AI Agent Offline (No API Key, No Token Bill)

You can run a capable AI agent offline, and the setup takes about fifteen minutes plus a download. The model weights sit on your disk, inference runs on your own CPU or GPU, and the reasoning loop needs no API key and no vendor account. Atomic Agent talks to a local llama.cpp server on 127.0.0.1 and drives real multi-step tasks with Wi-Fi off.

The practical entry point in 2026 is roughly 16 GB of memory, with 24 to 32 GB the comfortable floor for real work. This guide is the full path: install, pull a model, raise the context window so the agent loop actually fits, disconnect the network, and run a task. The context step is the one that trips up most first sessions, and it gets its own fix below.

• Install once with the network on, then everything after that runs locally
• qwen-3.5-4b (2.7 GB) is the modest-hardware default, qwen-3.8-27b (17.9 GB) is the capable one
• Raise contextSize to 65536 before your first run. The default 16K window is too small for an agent, and the failure message is confusing
• Data stays on your machine by default and leaves only when a tool you can watch sends it
• Grammar-constrained decoding is what makes a small quantized model reliable at tool calls

What “fully offline” actually means here

It means the whole reasoning loop runs on your hardware. The weights are a file on disk. Inference happens locally against a llama.cpp server bound to localhost. No inference request crosses the network, and you need no API key for the agent to function.

The agent still drives your files, shell, and git with the network off. It reads and writes code, runs commands, and commits work without contacting a provider. Memory persists in a plain SQLite file on your disk rather than a cloud account, so context carries across runs without anything being uploaded.

Two honest boundaries are worth stating up front. First, data stays local by default and only leaves when a tool you can see makes it leave: if you ask the agent to search the web, that specific step uses the network. Second, the agent ships anonymous telemetry that you can turn off, which is separate from your files and prompts but is still traffic. Offline means your work and your reasoning stay on the machine, not that the binary is inert.

If the distinction between a local model and a local agent is new to you, what a local AI agent is covers the definition before you start installing things.

Why run an agent offline at all

Three reasons, in the order people actually hit them.

Cost. A cloud agent bills per token, and agents are token-hungry in a way chat is not. Every tool result lands back in the context and gets re-read on the next step, so a single multi-step task can burn through what a whole afternoon of chatting costs. Local inference has a fixed cost: electricity and the hardware you already own.

Data. Some work cannot leave the building. Client code under NDA, medical records, internal financials, anything covered by a compliance regime that names approved processors. A local agent removes the question entirely for the reasoning step, because there is no processor to name.

Availability. No rate limits, no outages, no deprecated model version breaking a workflow you built six months ago. The model on your disk is the model on your disk. If you want the full trade-off analysis rather than the summary, local agent versus cloud agent works through both sides including the parts where cloud wins.

Hardware requirements

Memory is the constraint that decides everything else. The weights have to fit, and then the agent needs headroom on top, because the KV cache grows with every tool result that lands in the context. Budget for both.

A CPU-only machine works. Inference is slower, sometimes by an order of magnitude, but nothing about the agent loop requires a GPU. On Apple Silicon the numbers above apply to unified memory, with the caveat that macOS caps GPU-addressable memory at roughly 75 percent of the total, so a 24 GB Mac gives the model about 18 GB.

The gap between the 4B and the 27B is real and worth understanding before you pick. For the memory math, quant table, and two measured agent runs on the larger model, see how to run Qwen 3.8 27B locally. If you are not sure which side of the line your machine falls on, start with the 4B: it downloads in a minute and tells you quickly whether the workflow suits you.

Beyond memory you need free disk for the GGUF file, the single-file format llama.cpp uses to store quantized weights, plus macOS, Linux, or Windows, and a terminal. No API key, no account, no payment method.

How to run an AI agent offline, step by step

Managed mode is the recommended path. Atomic Agent downloads the llama.cpp binary and a GGUF model for you, picks a GPU if one is available, and supervises the local daemon. You do not wire anything up by hand. Other local agents split these steps differently, and how Atomic Agent, Hermes, and OpenClaw compare lays out where each one draws the line.

Step 1. Install Atomic Agent

curl -fsSL https://atomicagent.io/install | sh

This is one of only two steps that need the internet. The script fetches the binary and puts the atomic-agent command on your path. Confirm it landed:

atomic-agent --version

If the command is not found, open a new terminal window so the shell picks up the updated path.

Step 2. See what models are available

atomic-agent models list

The catalog prints each entry with its slug and download size. Two matter for this guide: qwen-3.5-4b at 2.7 GB, which is the default and runs almost anywhere, and qwen-3.8-27b at 17.9 GB, which needs 24 GB of memory and is the one to pick if you have it. The larger entry is a quantized build of the open-weight Qwen3.8-27B release, whose model card lists the licence and the context length the weights were trained for.

Choosing between them is a bandwidth and memory question, not a preference. Read the table above, then decide once. Switching later means another download.

Step 3. Pull the weights

atomic-agent models pull qwen-3.5-4b

This is the second and last step that needs the network. pull downloads the GGUF file to your disk and is idempotent, so an interrupted download resumes rather than starting over when you run it again.

For the larger model, substitute the slug:

atomic-agent models pull qwen-3.8-27b

Expect this to take a while on the 27B. It is 17.9 GB, and nothing else in the setup depends on it finishing, so start it before you read the rest of this page.

Step 4. Make the model active

atomic-agent models use qwen-3.5-4b

use sets which downloaded model the agent runs against. Pulling a model does not select it, which is deliberate: you can keep several on disk and switch between a fast small one and a capable large one depending on the task.

Step 5. Raise the context window before you start the daemon

Do this now rather than after the first failure. An agent needs far more context than a chat client, and the default window is not enough:

atomic-agent config set '{"localModels":{"managed":{"contextSize":65536},"completionMaxTokens":8192}}'

Here is why it matters. The agent’s system prompt carries every tool definition before any work begins: file operations, shell, git, browser, memory, and whatever MCP servers you have connected, each of which advertises its own tool list on connect. That is tens of thousands of tokens on an empty conversation, before the model has read a single file. A 16K window is roughly a quarter of what the loop needs, and the agent refuses to start rather than silently truncating its own instructions.

completionMaxTokens is the other half. Lowering it to 8192 caps how much of the window a single response can claim, which leaves room for the conversation to keep growing as tool results accumulate.

If you skip this step, the agent fails at launch with a message about the context window. The exact warning and its fix are in Troubleshooting.

Step 6. Start the local model server

atomic-agent models start

This launches the llama.cpp daemon on localhost. It loads the weights into memory, which on the 27B takes a noticeable moment and on the 4B is nearly instant. The daemon keeps running in the background, so you start it once and leave it.

Verify it came up healthy:

atomic-agent models status

A working state reports the mode as managed, names your active model, and shows health as ok. It also prints the context window the daemon actually allocated, which is your confirmation that Step 5 took effect. If it says 16384 rather than 65536, the config did not apply and you should re-run the config set command followed by a restart.

Step 7. Pull the network plug

Now prove it. Disconnect Wi-Fi, unplug the Ethernet cable, or turn on airplane mode.

The weights are already on disk and the local server is already running, so the agent has everything it needs. This is the step that separates a real local agent from a cloud wrapper: if it were quietly calling an API, it would break here.

Doing this on your first run is worth the thirty seconds. It converts a claim you read on a website into something you watched happen on your own machine.

Step 8. Run a task

Change into the directory you want the agent working in, then start it:

cd ~/projects/my-project
atomic-agent run

run opens an interactive session. It does not take the task as an argument: the agent starts up, prints a you> prompt, and you type the goal there. This catches people out, because most CLI tools expect the instruction inline.

The agent operates in whatever directory you launched it from, which is why the cd comes first. Three flags are worth knowing:

• --cwd sets the working directory explicitly instead of relying on where you launched from
• --max-steps caps how many steps a single run can take, which is useful when you are testing and do not want a task to wander
• --no-approval skips the confirmation prompt before each action

Leave --no-approval off for your first few runs. Watching the agent ask permission for each file write and shell command is the fastest way to build an accurate picture of what it actually does, and it is also the moment you would notice a tool reaching for the network.

Give it something real: rename files across a folder, refactor a function, summarize a local document, stage and commit your changes. It reasons locally and uses local tools, with the network still off.

Verifying that it really runs offline

The airplane-mode test is the honest one, but there are two more checks worth running if you want to be thorough.

Check what the agent is pointed at. The config lives at <stateDir>/config.json, where the default state directory is ~/.atomic-agent:

atomic-agent config get

For local inference you want the mode set to managed, and the URL pointing at localhost. Anything else means the agent is talking to a server you should be able to name.

Watch the connections. On macOS or Linux, run a task with a network monitor open and look at what the process opens. In managed mode you should see traffic to 127.0.0.1 on the llama-server port and nothing else, unless a tool you invoked went looking for the web.

If your local server runs somewhere other than the default, set it in config.json:

{
  "localModels": {
    "mode": "external",
    "url": "http://localhost:8080"
  }
}

As long as this points at localhost and no cloud key is configured, the inference stays on your machine.

Does a small local model really work well enough

Reliability and capability are two separate questions, and they have different answers.

Reliability is solved, and not by luck. The failure everyone expects from a quantized local model is malformed tool calls: the model emits JSON that is one comma away from valid, the step fails, and the task dies. Atomic Agent handles this with GBNF grammars. A tool call is not requested by the prompt, it is constrained at the decoder: tokens that would break the schema are filtered out during sampling, so the model cannot physically emit an invalid structure. How grammar-constrained tool calling works walks through the mechanism and why it holds up on a quantized model. In our measured runs on the 27B, six steps across two tasks produced zero malformed calls, including one step where the model batched two tool calls in parallel.

Capability is a real trade. A 4B model at 4-bit is not a frontier cloud model. It knows less, it plans shorter, and on an open-ended task it will take a clumsier route. What grammar constraints buy is that its clumsiness shows up as a suboptimal choice rather than a crash, which means you can watch it, correct it, and let it continue.

Speed is the honest weakness. Local inference on consumer hardware is slow enough to notice. On a 32 GB Mac, the 27B took roughly three minutes to list a directory and describe three files, and seven minutes to read an image and write a summary to disk. The 4B is considerably faster and considerably less capable. Neither feels like a cloud agent. You are trading latency for cost and control, and whether that trade works depends on the task: it is excellent for work you can start and walk away from, and frustrating for anything you want to iterate on in real time.

What an agent can and cannot do offline

Works with the network off:

• Reading, writing, and generating code across your filesystem
• Running shell commands and reading their output
• Navigating directories, renaming and reorganizing files
• Git operations: status, diff, stage, commit, branch
• Summarizing and transforming local documents
• Persistent memory across sessions, stored in local SQLite

Needs a connection, because the task itself does:

• Web search
• Browser automation against live sites
• Fetching a URL
• Any MCP server that talks to a remote API
• Pulling a new model or updating the agent binary

The line is clean: the agent’s reasoning never needs the network, and a tool needs the network only when the task inherently does. If you ask it to find something on the web in airplane mode, that step fails and the agent tells you so. Nothing else degrades.

Troubleshooting

The agent refuses to start with a context warning

This is the most common first-run failure, and it is not a broken install:

WARN context window too small for the agent prompt
{"contextWindow":16384,"required":72048}

Cause. The agent’s system prompt includes every tool definition before any work starts, so the floor is tens of thousands of tokens on an empty conversation. The default 16K window is roughly a quarter of what the loop requires. This is the single biggest difference between running a model for chat and running it for agent work, and it is why a model that works fine in a chat app fails here.

Fix. Raise the context size, lower the completion ceiling, and restart the daemon so the new window is allocated:

atomic-agent config set '{"localModels":{"managed":{"contextSize":65536},"completionMaxTokens":8192}}'
atomic-agent models stop && atomic-agent models start

The restart is not optional. The engine allocates the KV cache when it loads the model, so a config change without a restart leaves the old window in place. After restarting, atomic-agent models status should report a context window of 65536.

The daemon fails to load the model, or the machine starts swapping

Symptom. models start hangs, the process is killed, or the whole machine grinds while the model loads.

Cause. The weights plus the KV cache exceed available memory. Raising contextSize to 65536 is not free: it allocates cache up front, and on a machine that barely fit the weights it is what pushes you over.

Fix. Step down to a smaller model with atomic-agent models use qwen-3.5-4b, or lower contextSize to 32768. On the 4B, 32K is usually enough for file and shell work. Close other memory-hungry applications before deciding you need a smaller model, since a browser with many tabs can easily be the difference.

The agent dies partway through a long task

Symptom. The first several steps run fine, then the process exits twenty steps in.

Cause. The KV cache growing. Every tool call returns data that stays in the context, so memory pressure builds as the task goes rather than showing up at load time.

Fix. Cap the run with --max-steps while you are calibrating, and step down one model size or quant if you are near the memory ceiling. Breaking a long task into several shorter runs also works, since memory carries over in SQLite even when the context does not.

The agent starts but cannot reach the model

Symptom. The startup log shows a health check failure against a port you did not expect.

Cause. The config was switched to external mode, so the agent is looking for a llama-server you are supposed to be running yourself, and it ignores its own managed daemon.

Fix. Run atomic-agent config get and check the mode. If it says external, set it back to managed and confirm the port in the managed config matches the one in the startup line.

Everything works, but responses take minutes

Symptom. A single step takes one to two minutes on a machine that is not swapping.

Cause. This is normal for local inference on consumer hardware, not a misconfiguration. A dense model at 4-bit on a laptop generates tokens at a fraction of cloud speed, and an agent generates a lot of them.

Fix. There is no fix, only choices. A smaller model responds faster. A GPU with enough VRAM for full offload is the single largest speedup available. Beyond that, restructure how you use it: give the agent a batch of work and come back, rather than sitting through each step.

Tool calls come back malformed

Symptom. Steps fail with parsing errors on the model’s output.

Cause. The model is generating freely rather than under a grammar. This happens when you point the agent at a third-party runtime that does not apply GBNF constraints.

Fix. Confirm you are running through the managed daemon rather than an external server configured without grammar support. Constraining structure at the decoder is exactly what keeps a quantized model from breaking the schema, and without it a small model will occasionally miss.

Frequently asked questions

Bottom line

Running an agent offline is now a fifteen-minute setup rather than a project. Install, pull a model that fits your memory, raise the context window before the first launch, disconnect, and type your goal at the prompt.

Key takeaways:

• Only two steps need the internet: installing the binary and pulling the weights
• Match the model to your memory: qwen-3.5-4b from 8 GB, qwen-3.8-27b from 24 GB
• Raise contextSize to 65536 before your first run, because agent prompts are far larger than chat prompts
• Grammar-constrained decoding is what makes tool calls reliable on a quantized model, and it is not something you configure
• You are trading speed for cost and control, and the trade suits work you can start and walk away from

Ready to try it? Follow the quickstart guide and run your first task offline in a few minutes.