How to Run Qwen 3.8 27B Locally

Qwen 3.8 27B is the first model in the Qwen 3.8 family that works as an agent on a single graphics card. Per the official model card, it is a dense 27-billion-parameter model with native vision, a 262,144-token context window, and an Apache 2.0 license. At 4-bit quantization the file is roughly 17 GB, which fits a 24 GB GPU or a Mac with 32 GB of unified memory.

The gap between running a model and running an agent is tool calling. An agent cannot just produce text: it has to emit a structured call that executes without error, then do it again dozens of times across a single task. On a quantized model, that is where things break.

• Qwen 3.8 27B shipped on August 13–14, 2026 under Apache 2.0, so commercial use carries no revenue thresholds
• The 4-bit build is about 17 GB, and 24 GB of memory is the practical floor for agent work
• Hybrid attention (Gated DeltaNet plus Gated Attention) keeps the KV cache small, so long context costs less here than on a standard model of this size
• Thinking is on by default but can be disabled with enable_thinking: False
• Atomic Agent uses GBNF grammars so tool calls stay structurally valid even at 4-bit

What Qwen 3.8 27B is

Qwen 3.8 27B is an open-weight model from Alibaba’s Qwen team, the smallest member of the Qwen 3.8 family. Alibaba announced the family on August 3, 2026, and published the 27B weights on August 13–14.

The model card describes it this way:

“Qwen3.8-27B brings these advances to a compact, deployment-friendly dense model: a native vision-language model that understands images and videos, with flexible thinking control”

The word that matters there is dense. The family ships three models and they are easy to confuse:

• Qwen 3.8 Max is the 2.4T-parameter MoE flagship with 95B active parameters, served through the Alibaba Cloud API
• Qwen3.8-2.4T-A95B is the downloadable version of Max: text-only, always reasoning, and a datacenter job to serve
• Qwen 3.8 27B is the dense vision-language model this guide covers, and the only one of the three that runs on an everyday machine

That last point is the whole story. The frontier open-weight models of 2026 are effectively unavailable to individuals: Kimi K3 needs roughly 1,680 GB of VRAM, and the open weights of Qwen 3.8 Max are in the same class. A 27B dense model that fits on a card you already own is a different proposition, and it is the reason this release matters more than the flagship it shipped alongside.

Specifications

The model card documents the layout. Of the 64 layers, 16 use Gated DeltaNet, with 48 linear attention heads for V and 16 for QK. The rest use Gated Attention, with 24 heads for Q and 4 for KV. The practical consequence: a Gated DeltaNet layer holds a fixed-size state that does not grow as the conversation gets longer, while a conventional KV cache grows with every token. Long context therefore costs noticeably less memory here than on a classic transformer of the same size, and for an agent that matters directly, because every tool result lands back in the context.

Published benchmarks

The numbers below come from the official model card. For agent work the top two rows carry the most weight: SWE-bench Pro and Terminal Bench measure whether a model can carry a multi-step task with tools to completion, not what it knows.

How to read this: the scores are Qwen’s own, measured on Qwen’s own harness. Every lab runs a different setup, so comparing these figures directly against another vendor’s table is not meaningful. One caveat matters more than the rest: these runs used the full weights, not the 4-bit build you are about to download. How much quantization costs you specifically on agentic tasks is a question vendor tables do not answer.

So we ran it ourselves. The section below covers what a quantized build actually does inside an agent loop on consumer hardware.

Why an agent needs more than a chat model

This is the section missing from ordinary “how to run a model” guides, and it is the reason to keep reading.

A chat workflow forgives a lot. The answer arrives as text, a human reads it, and asks again if something looks wrong. An agent workflow forgives almost nothing. The model has to emit a tool call in a strict format, take the result, account for it, and emit the next call. One syntax error in the JSON and the step fails. Across a long task there are dozens of those steps.

Quantization hits exactly there. It lowers the precision of the weights, and structured output reliability goes with it. A model that returns valid JSON 99 times out of 100 at full precision can slip to 90 at 4-bit, and an agent will then stumble on every tenth operation.

Atomic Agent handles this with GBNF grammars. A tool call is not requested by the prompt, it is constrained by a grammar at the decoder level: tokens that would break the schema are filtered out during sampling, so the model cannot physically emit an invalid structure. The mechanics are covered in GBNF grammar-constrained tool calling.

This reframes the question. It is not “can a 27B model at 4-bit hold up as an agent,” it is “how well does it choose the right tool,” because formatting is no longer the model’s job. If you are new to the distinction between a local model and a local agent, this piece covers it.

What it does in an actual agent loop

I ran Qwen 3.8 27B through Atomic Agent on a MacBook Pro with an M5 and 32 GB of unified memory, using the UD-Q4_K_XL build that ships in the agent’s own catalog: a 17.9 GB file, fully offloaded, with a 64K context window and approvals disabled.

Two tasks, both completed, zero malformed tool calls across six steps.

Task 1: read a directory and describe it

The prompt was “list the files in this folder and tell me what each one is.” The working directory held an HTML file, a photo of a hand-drawn floor plan, and a photo of a receipt.

The interesting part is step 1. The model issued two tool calls in a single batch, reading the HTML file and sending both images to the vision pipeline at the same time, because neither result depended on the other. Nothing in the prompt asked for that. A model that emits tool calls one at a time and waits would have taken noticeably longer.

Vision returned in 21.5 seconds and got both images right: a hand-drawn floor plan of a 55 m² apartment with room labels and furniture, and a crumpled Walmart receipt sitting on a dark textured wallet.

Task 2: extract data from an image and write a file

The second prompt was “read apartment.html and floorplan.jpg, then write a short summary of this apartment to summary.md.”

This one matters more, because it is the thing a chat window cannot do. The model read dimensions off a photograph of a hand-drawn sketch, kitchen at 3.5 by 2.8 m, living room at 4.5 by 3.4 m, and wrote them into a structured 969-byte Markdown file on disk. Reading a picture is one capability; turning what is in the picture into a file on your filesystem is another.

What this says, and what it does not

Reliability held. Six steps across two tasks, including a two-call batch, and every single call came back ok. That is what grammar constraints buy you at 4-bit: the failure mode people expect from quantized tool calling did not appear.

Speed is the honest weakness. 202 seconds to list and describe three files is slow, and 421 seconds for the second task is slower. A single reply step, which is just composing text, took 95 to 106 seconds. A dense 27B model at 4-bit on Apple Silicon is not a snappy experience, and anyone expecting cloud-agent latency will be disappointed.

The trade is straightforward. You wait minutes instead of seconds, and in exchange the model, the files, and the images are processed on the machine itself. Both tasks above ran with the network down, which is the setup described in running an AI agent offline. Whether that trade is worth it depends entirely on the work, and the full comparison against a cloud agent is here.

One caveat on these numbers: this is a single machine, two tasks, one quant. Treat them as a directional picture of what the model does inside an agent loop, not as a benchmark.

Hardware requirements

Qwen 3.8 27B system requirements come down almost entirely to memory. The 4-bit build needs roughly 17 GB for the weights alone, which makes a 24 GB graphics card or a 32 GB Mac the practical floor. The table below accounts for the fact that an agent needs headroom on top: tool results flow into the context and consume memory beyond the weights themselves.

The build in the Atomic Agent catalog is UD-Q4_K_XL, an Unsloth Dynamic quant that reports as 17.9 GB on disk. It loaded fully on a 32 GB M5 with room for a 64K context.

How much the context window adds

Those figures cover weights only. On top of them the engine allocates a KV cache, and for an agent that is not a detail: every tool call returns data that stays in the context.

The hybrid architecture works in our favor here. In a conventional transformer all 64 layers would store attention data for every token. Here 16 layers use Gated DeltaNet with a fixed-size state, and the attention layers get by with four KV heads. The result is a KV cache several times smaller than a standard model this size.

These are estimates carried over from Qwen3.6 27B, which shares the attention layout. In practice a 32 GB M5 ran the 17.9 GB build with a 64K context without trouble.

There is a catch here that costs people their first session, and it is covered in Troubleshooting: an agent needs far more context than a chat client does, because the tool definitions alone are large before any work begins.

What actually runs it

• RTX 3090 / 4090 (24 GB): Q4_K_M entirely in VRAM, the working configuration for agent use
• RTX 5090 (32 GB): Q5_K_M or Q6_K entirely in VRAM
• 16 GB cards (4060 Ti, 5060 Ti): Q3_K_S only. The model is dense, so offloading layers to system RAM costs speed on every single token, and an agent generates a lot of them
• MacBook Pro (24 GB): Q4_K_M with a modest context. macOS caps GPU-addressable memory at about 75 percent of unified memory, so a 24 GB machine gives the model roughly 18 GB
• Mac Studio / M4–M5 Max (36–64 GB): Q5 through Q6 comfortably, Q8_0 from 48 GB up
• Phones: no

If you are still deciding whether local is the right call at all, the trade-offs against a cloud agent are here.

Running Qwen 3.8 27B in Atomic Agent

Atomic Agent is an MIT-licensed open-source agent that runs on top of a local llama.cpp server. It is not a chat app: it drives a browser, files, shell, and git, keeps memory in SQLite, and connects external tools over MCP. The source is on GitHub.

Weights are only half the decision: the runtime you drop them into determines what the model can actually do. If you are still choosing, this comparison of Atomic Agent, Hermes, and OpenClaw covers the differences.

There are two routes: managed mode, where the agent downloads and supervises the model itself, and external mode, where you run llama-server yourself and point the agent at it.

Option A: managed mode

Install:

curl -fsSL https://atomicagent.io/install | sh

Then the model:

atomic-agent models list
atomic-agent models pull qwen-3.8-27b
atomic-agent models use qwen-3.8-27b
atomic-agent models start

The catalog entry is qwen-3.8-27b and the download is 17.9 GB.

Before starting the daemon, raise the context size. The default is too small for agent work, for reasons covered in Troubleshooting:

atomic-agent config set '{"localModels":{"mode":"managed","completionMaxTokens":8192,"managed":{"modelId":"qwen-3.8-27b","contextSize":65536}}}'

Confirm the daemon is healthy:

atomic-agent models status

A working state reports mode: managed, active model: qwen-3.8-27b, and health: ok.

Then start the agent:

atomic-agent run

run opens an interactive session rather than taking the task as an argument: you type the goal at the you> prompt once the agent is up. It works in whatever directory you launched it from, so cd to your project first. Add --no-approval to skip the confirmation prompt on each action, and --max-steps to cap a run.

Option B: your own llama-server

Take this route when you want control over the flags: layer offload, context size, sampling.

Build llama.cpp with CUDA:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
cmake --build build -j --target llama-server

On Apple Silicon, Metal is enabled by default and the CUDA flag is unnecessary.

Start the server with Qwen’s recommended sampling for thinking mode:

./build/bin/llama-server \
  -hf unsloth/Qwen3.8-27B-GGUF:Q4_K_M \
  --alias qwen3.8-27b \
  --jinja \
  --temp 1.0 --top-p 0.95 --top-k 20 \
  -ngl 99 \
  -c 32768 \
  --host 127.0.0.1 --port 8080

The sampling values come straight from the model card. Qwen specifies temperature=1.0, top_p=0.95, top_k=20, min_p=0.0 for thinking mode, and a different set for instruct mode: temperature=0.7, top_p=0.80, top_k=20, presence_penalty=1.5.

The context is set to 32K rather than the full 262K on purpose. The engine allocates the KV cache up front, and the maximum window will eat memory you need for the actual work. For agent tasks, 32K is a sensible starting point.

Now connect the server to the agent. The config lives at <stateDir>/config.json:

{
  "localModels": {
    "mode": "external",
    "url": "http://localhost:8080"
  }
}

Point the agent at it in config.json:

{
  "localModels": {
    "mode": "external",
    "url": "http://localhost:8080"
  }
}

Tuning for agent work

Output length is governed by ATOMIC_AGENT_LLAMA_MAX_TOKENS, which defaults to 4096 and accepts 64 through 131072. For an agent that writes long files, the default runs out:

export ATOMIC_AGENT_LLAMA_MAX_TOKENS=16384

Then there is thinking. It is on by default, and on agentic work it tends to work against you: the model spends tokens reasoning before every tool call, and there are dozens of calls in a task. The model card states plainly that the mode is disabled per request with enable_thinking: False.

Atomic Agent resolves this build under a qwen-think profile, so reasoning is active inside the agent loop too. In the runs above the model kept it brief, emitting empty <think></think> blocks on the simpler task rather than reasoning at length, so the cost is not fixed: it scales with how much the model judges the task to need.

Troubleshooting

The agent refuses to start with a context warning

This is the first thing most people hit, and it is not a broken install:

WARN context window too small for the agent prompt
{"contextWindow":16384,"required":72048,"completionMaxTokens":65536}

An agent needs far more context than a chat client. The system prompt carries every tool definition before any work starts, so the floor is tens of thousands of tokens on an empty conversation. A default 16K window is roughly a quarter of what the loop requires.

Fix it by raising the context and lowering the completion ceiling, then restarting the daemon:

atomic-agent config set '{"localModels":{"managed":{"contextSize":65536},"completionMaxTokens":8192}}'
atomic-agent models stop && atomic-agent models start

The warning should be gone on the next launch, and the startup line will report contextWindow: 65536.

The agent starts but cannot reach the model

If the startup log shows llama-server health check failed against a port you did not expect, check whether the config was switched to external mode:

atomic-agent config get

"mode": "external" means the agent is looking for a server you have to run yourself, and it will ignore its own managed daemon. To go back to the built-in runtime, set mode to managed and confirm the port in localModels.managed.port matches the one in the startup line.

The GGUF you downloaded is not the real model

Quantized builds ship in the GGUF format, and placeholder repositories claimed the Qwen3.8-27B name on Hugging Face before the weights existed. Before pulling from an unfamiliar repo, check that the publisher is either the official Qwen organization or a quantizer you recognize, that the model card exists, and that the file sizes are plausible. A 27B model at 4-bit cannot weigh 400 MB.

The agent dies partway through a long task

The weights fit, then the process dies twenty steps in. That is the KV cache growing: every tool call adds its result to the context. Set the context explicitly with -c instead of leaving the maximum, and step down one quant if memory is short. Moving from Q5 to Q4 frees roughly 2.5 GB.

Tool calls come back malformed

If you run the model through a third-party runtime without grammars, a quantized model will occasionally break the JSON. That is precisely what GBNF prevents in Atomic Agent: the structure is constrained at the decoder rather than requested in the prompt. If you are seeing malformed calls, check that the agent is actually running through grammars and not free generation.

The model answers but never calls a tool

Check that the server started with --jinja. Without it the chat template inside the GGUF is not applied, and the model never sees the tool definitions in the format it expects.

Frequently asked questions

Bottom line

If you have a 24 GB card or a 32 GB Mac, Qwen 3.8 27B is the Qwen 3.8 model worth running locally for agent work: pull Q4_K_M, start at 32K context, turn thinking off, and raise the ceiling as the task demands.

Key takeaways:

• Qwen 3.8 27B is a dense 27B multimodal model with hybrid attention, a 262K context window, and an Apache 2.0 license
• The 4-bit build is about 17 GB, and 24 GB of memory is the practical floor for agent work
• Hybrid attention keeps the KV cache small, which matters because an agent fills its context with tool results
• Vendor benchmarks (61.7 SWE-bench Pro, 73.0 Terminal Bench) were measured on full weights, not on quants
• Tool call reliability on a quantized model comes from GBNF grammars, not from luck