Use with vLLM

Short answer. vllm serve <model> starts an OpenAI-compatible server on http://localhost:8000 (vLLM quickstart). For an agent, turn tool calling on when you start it:

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --enable-auto-tool-choice --tool-call-parser hermes

Check it:

curl http://localhost:8000/v1/models

Atomic Agent has no vLLM preset. You add vLLM as a generic OpenAI-compatible provider with base URL http://<host>:8000 and the model id that /v1/models returns.

vLLM server cheat sheet

Flag or variable	What it does
--host / --port	Bind address and port (default port 8000)
--api-key KEY or VLLM_API_KEY	Makes the server check a key in the request header
--enable-auto-tool-choice	Lets the model emit tool calls; required for tool use
--tool-call-parser NAME	Parser for your model family, for example hermes for Qwen2.5

The parser list per model family is in vLLM tool calling. vLLM targets Linux with GPUs (NVIDIA CUDA, AMD ROCm, Intel XPU and others) and also runs on CPU; see vLLM installation.

Common vLLM errors with an agent

• Connection refused on 8000. The server is still starting (large models take a while to load) or runs on another port. Watch the startup log.
• 401 Unauthorized. The server was started with --api-key and the request carried no key or a different one.
• The model answers in text but never calls tools. The server was started without --enable-auto-tool-choice and a matching --tool-call-parser.
• 400 errors about message roles. If a server rejects tool-role messages (a 400 about roles or tool_call_id), set "messageShape": "flat" on the provider entry (see below); Atomic Agent also switches to it on its own after such an error, for the rest of the session.

Add vLLM to Atomic Agent

Desktop app

1. Start vLLM and confirm /v1/models lists your model.

2. Open Settings › Models › Cloud providers, click Add provider and choose OpenAI-compatible API (custom base URL). On first run, the same row is in the Cloud models list.

3. Enter the base URL (http://localhost:8000, or the GPU server’s address), the API key if you set one, and pick the model id.

Terminal

1. Run /llm, open the Cloud pane and press n.

2. Pick the openai-compatible row, enter the base URL, the key, then the model.

3. Switch to it with /llm provider <id>.

The wizard treats a loopback address (localhost, 127.0.0.1) as a local server and lets you leave the key empty. For a vLLM on another machine it asks for a key: use the value you passed to --api-key. If that server runs without --api-key, it does not check the header, so any placeholder value goes through.

Configure by hand

{
  "llm": {
    "activeTextProvider": "my-vllm",
    "providers": [
      { "id": "local-llama", "kind": "llama-server" },
      {
        "id": "my-vllm",
        "kind": "openai-compatible",
        "baseUrl": "http://gpu-box:8000",
        "apiKeyEnvVar": "MY_VLLM_API_KEY",
        "defaultChatModel": "Qwen/Qwen2.5-7B-Instruct",
        "userModels": [
          { "id": "Qwen/Qwen2.5-7B-Instruct", "kind": "chat", "contextWindow": 32768 }
        ]
      }
    ]
  }
}

• Put the key in <stateDir>/.env as MY_VLLM_API_KEY=.... Without apiKeyEnvVar the entry falls back to OPENAI_COMPAT_API_KEY or OPENAI_API_KEY.
• Store baseUrl without /v1; a trailing /v1 is stripped before each request anyway.
• Atomic Agent reads only model ids from /v1/models, not the context window. userModels[].contextWindow tells the agent the real window of your deployment.
• Add "messageShape": "flat" only if the server rejects tool-role messages (a 400 about roles).

Remote means your prompts travel

A vLLM on another machine receives everything the agent puts in the prompt: file contents it has read, tool output and memory. Use HTTPS or a private network, and set --api-key on anything reachable from outside.

FAQ

Does Atomic Agent support vLLM?

Yes, through the generic OpenAI-compatible provider. There is no dedicated preset, so you enter the base URL and model id yourself.

What port does vLLM use?

8000 by default. Change it with --port.

Why does the agent not call tools on vLLM?

vLLM only returns tool calls when started with --enable-auto-tool-choice and a --tool-call-parser that matches the model. Atomic Agent sends its tools through the OpenAI tools field, so without those flags the model can only answer in text.

Can I use vLLM in the llama.cpp External pane?

No. That pane expects llama.cpp’s own endpoints. If you enter a vLLM URL there, Atomic Agent detects an OpenAI-compatible server and points you to the provider list.

Related

• vLLM vs Ollama
• Use with llama.cpp, Use with Ollama
• Configuration