Bytes, AI, Llama.cpp, Local coding agent on Windows with OpenCode

What?

llama.cpp is a dependency-free C/C++ inference engine for running open-weight LLMs locally in the GGUF format. Its llama-server exposes an OpenAI-compatible API on localhost, which means any tool that speaks the OpenAI protocol — including the OpenCode terminal coding agent — can drive a model running entirely on your own GPU.

This byte walks through wiring the two together on a Windows PC with an NVIDIA RTX 5090 (32GB VRAM), using Qwen3.6-35B-A3B as the model. It is the llama.cpp counterpart to running models with Ollama.

Why?

Private, offline, free per token — no API keys, nothing leaves the machine, and an agent loop that makes dozens of tool calls per task costs nothing extra.
llama.cpp over Ollama / LM Studio — Ollama is fine casually but leaves ~15% throughput on the table, and for agentic work you want direct control over context size and KV-cache quantisation. llama.cpp gives you that (and everything else wraps it anyway).
Qwen3.6-35B-A3B is a 35B-total / 3B-active mixture-of-experts model (Apache 2.0, April 2026) with a 262K native context that scores well on agentic coding benchmarks (73.4 SWE-bench Verified, 51.5 Terminal-Bench 2.0). Because only ~3B parameters fire per token, it runs at roughly 80–120 tok/s on a 5090 — which matters far more than raw quality in an agent loop that round-trips constantly.

How?

0. Prerequisites

Windows 11 x64, NVIDIA driver up to date (the prebuilt CUDA 12.4 binaries need a driver that supports CUDA 12.x — any 2025+ GeForce driver does).
~30GB free disk for the model.
Node.js (only if you install OpenCode via npm).

All commands below are for PowerShell.

1. Install llama.cpp

Option A — winget (simplest, auto-updates)

winget install llama.cpp

This puts the llama unified CLI on your PATH. Verify with:

llama --version

Option B — prebuilt CUDA release zip (explicit GPU build)

Go to the releases page and download two assets from the latest build (bNNNNN):

Asset	Purpose
`llama-bNNNNN-bin-win-cuda-12.4-x64.zip`	the binaries
`cudart-llama-bin-win-cuda-12.4-x64.zip`	CUDA runtime DLLs (only needed if you don’t have the CUDA Toolkit installed)

Extract both into the same folder, e.g. C:\llama.cpp\, then add it to your PATH:

[Environment]::SetEnvironmentVariable("Path", $env:Path + ";C:\llama.cpp", "User")

Open a new terminal and check the GPU is detected:

llama-server --version

Tip

The zip ships both the new unified llama.exe (llama serve, llama cli, llama bench) and the classic llama-server.exe / llama-cli.exe / llama-bench.exe. They are the same thing — use whichever you prefer. This byte uses llama-server in the long-form commands.

2. Get the model

Grab the Unsloth GGUF — their uploads include the developer role in the chat template, which Codex/OpenCode-style agentic tools rely on, and improved tool-call parsing.

The easiest way is to let llama.cpp download it straight from Hugging Face on first run with -hf <repo>:<quant>:

llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q5_K_S

The file is cached under %LOCALAPPDATA%\llama.cpp\ (override with the LLAMA_CACHE env var if you want it on another drive) and reused on subsequent runs. Alternatively download the .gguf manually from unsloth/Qwen3.6-35B-A3B-GGUF and pass it with -m.

Picking a quant for 32GB VRAM

Q6 does not fit

The weights alone must leave room for the KV cache and compute buffers. On a 32GB card that rules out the Q6 variants for any serious context length.

Quant	Size	128K ctx w/ q8 KV	Notes
`UD-Q4_K_M`	22.1 GB	✅ ~9GB headroom	Unsloth’s recommended default
`UD-Q5_K_S`	24.9 GB	✅ ~6GB headroom	Sweet spot — best quality that still fits 128K
`UD-Q5_K_XL`	26.6 GB	⚠️ tight	Drop to 64K context
`UD-Q6_K`	29.3 GB	❌	Weights only, no room for KV
`UD-Q6_K_XL`	31.8 GB	❌	Won’t load

Start with UD-Q5_K_S. If you want 256K context, drop to UD-Q4_K_M.

3. Run the server

llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q5_K_S `
  -ngl 99 -c 131072 -fa on `
  --cache-type-k q8_0 --cache-type-v q8_0 `
  --jinja --alias qwen3.6-35b `
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 `
  --port 8080

What each flag does:

Flag	Why
`-ngl 99`	Offload every layer to the GPU
`-c 131072`	128K context — gives OpenCode a real repo-scale window; Qwen recommend ≥128K to preserve thinking quality
`-fa on`	Flash attention (required for quantised KV cache)
`--cache-type-k/v q8_0`	8-bit KV cache — halves cache memory vs f16, negligible quality loss; what makes 128K affordable on 32GB
`--jinja`	Use the model’s own chat template (needed for tool calling)
`--alias qwen3.6-35b`	The model name OpenCode will request — must match the config in step 5
`--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0`	Qwen’s recommended sampling for precise coding in thinking mode (use 0.7 / 0.8 / 20 for non-thinking)

Save it as a script
Put the command in C:\llama.cpp\serve-qwen.ps1 so it’s one call. Note the PowerShell line-continuation is a backtick `; in cmd.exe it’s ^.

If running it fails with “running scripts is disabled on this system”, that’s PowerShell’s execution policy (the Windows equivalent of chmod +x, but per-user rather than per-file — it defaults to Restricted on client Windows). Fix it once, for your user only, no admin required:
Set-ExecutionPolicy -Scope CurrentUser RemoteSigned
RemoteSigned runs locally-written scripts freely and only demands a signature on scripts downloaded from the internet. If you’d rather not change the policy, run the one script with a bypass instead:
powershell -ExecutionPolicy Bypass -File C:\llama.cpp\serve-qwen.ps1

Once it logs server is listening, check it:

curl http://127.0.0.1:8080/health
curl http://127.0.0.1:8080/v1/models

There’s also a built-in chat web UI at http://127.0.0.1:8080 — useful for a quick sanity-check before involving OpenCode.

4. Install OpenCode

Pick one:

# Chocolatey
choco install opencode
 
# Scoop
scoop install opencode
 
# npm
npm install -g opencode-ai

Verify:

opencode --version

5. Point OpenCode at llama.cpp

Create %USERPROFILE%\.config\opencode\opencode.json (C:\Users\<you>\.config\opencode\opencode.json):

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "llamacpp": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "llama.cpp (local)",
      "options": {
        "baseURL": "http://127.0.0.1:8080/v1"
      },
      "models": {
        "qwen3.6-35b": {
          "name": "Qwen3.6-35B-A3B (local)",
          "limit": {
            "context": 131072,
            "output": 32768
          }
        }
      }
    }
  },
  "model": "llamacpp/qwen3.6-35b"
}

Three things must line up:

models.<key> (qwen3.6-35b) == the --alias you gave llama-server.
limit.context == the -c value.
model == <provider-key>/<model-key>.

Swapping models often?
The opencode-local-provider plugin probes llama.cpp / LM Studio / Ollama at runtime and exposes whatever is loaded, so you skip the config dance:
opencode plugin --global opencode-local-provider

6. Run it

With llama-server still running in one terminal, open another in a project:

cd C:\src\some-repo
opencode

The status bar should show llamacpp/qwen3.6-35b. Give it a small, scoped task first (e.g. “add a unit test for X”) and watch it read files, edit, and run commands — that confirms tool-calling works end-to-end before you trust it with anything larger.

7. Measure

llama-bench reports real prompt-processing and generation speed at your settings:

llama-bench -m "$env:LOCALAPPDATA\llama.cpp\<the .gguf>" -ngl 99 -fa 1 -p 512 -n 128

Expect roughly 80–120 tok/s generation on the 5090. If you want more headroom for 256K context, drop to UD-Q4_K_M.

Gotchas

Keep tasks scoped. Local models are noticeably weaker than frontier cloud models at long multi-step chains. Aim for ≥64K context and prove tool-calling on a small repo task before handing over a big refactor.
Truncated write calls. There’s a known OpenCode issue where local models truncate large write tool calls and drop filePath. The "output": 32768 limit above mitigates it; check whether it’s been fixed in the current release.
VRAM OOM on load. Reduce -c first (65536), then drop a quant. Don’t drop -ngl — partial offload of an MoE is dramatically slower.
Thinking tokens in the transcript. If <think> blocks leak into OpenCode’s output, add --reasoning-format deepseek to the server so they’re returned in reasoning_content instead.
Port clash. If something else owns 8080, change --port and baseURL together.

Alternatives

Speed: unsloth/Qwen3.6-35B-A3B-MTP-GGUF adds a multi-token-prediction head for ~1.15–1.25× extra throughput via speculative decoding.
Judgement over speed: a dense model such as Qwen3.6-27B (or its successor Qwen3.8-27B) at Q5 (~20–24GB) is slower (~45 tok/s) but some prefer its reasoning on harder refactors. Try the 35B-A3B first.
LM Studio works identically — just point baseURL at port 1234.

Documentation

llama.cpp: https://github.com/ggml-org/llama.cpp
llama.cpp install options: https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md
llama-server reference: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
Qwen3.6-35B-A3B GGUF (Unsloth): https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF
OpenCode: https://opencode.ai/docs/
OpenCode providers (llama.cpp section): https://opencode.ai/docs/providers/
opencode-local-provider plugin: https://www.npmjs.com/package/opencode-local-provider