Running a 30B model on integrated graphics

When I set up kagent it talked to LM Studio on a different machine, a desktop with an RTX 3080. That works, but it means the cluster's AI only exists when that desktop is on. I wanted it on the homelab host, which has no discrete graphics card at all.

It has a Radeon 780M, the integrated GPU in the CPU. On paper that's the wrong hardware for a 30B model. It turned out to be a reasonable one, for a reason that has nothing to do with the GPU.

A container, not a VM

The server runs Ollama in an LXC on the Proxmox host, outside the Kubernetes cluster, listening on port 11434.

kagent (in-cluster, wave 7)
   │  ModelConfig ──> LXC on the host, :11434
   ▼
LXC "llm"        6 cores · 32 GB · 80 GB disk
   │
   ├─ ollama.service          OpenAI-compatible API on :11434
   ├─ /dev/dri/renderD128     bound in from the host
   └─ Vulkan / Mesa RADV ──>  Radeon 780M, 48 GB addressable

An LXC rather than a VM because passing the iGPU into a container is a device bind. The same thing for a VM needs VFIO, and the host loses its display in the process. Two smaller reasons: inference is CPU-bound in places and a container sees the host CPU with nothing in between, and LXC memory isn't pre-reserved, so the 32 GB ceiling costs nothing while no model is loaded.

Getting the iGPU to do anything

Ollama ships three ways to reach an AMD GPU and only one of them works here.

ROCm doesn't. The bundled runtime doesn't cover gfx1103, which is what the 780M reports, and it crashes during device discovery.

Vulkan does, through Mesa's RADV driver, and the backend is already compiled into the stock build. Three things have to line up:

1. /dev/dri/renderD128 present inside the container. The render node is enough; card0 is for display and inference never touches it.
2. mesa-vulkan-drivers installed in the container. Without it Vulkan finds no device and Ollama quietly falls back to the CPU, which is the failure mode to watch for, because nothing errors.
3. OLLAMA_IGPU_ENABLE=1. Ollama skips integrated GPUs unless asked.

That second one cost me an afternoon. A CPU fallback looks exactly like a working setup, just slower, and "slower" is hard to notice when you have no baseline.

The memory is the interesting part

The 780M has no memory of its own. It addresses system RAM through GTT, and on this host that's 48 GB:

cat /sys/class/drm/card0/device/mem_info_gtt_total
49452544000

So an 18.6 GB model loads entirely onto the GPU, all 49 layers. A discrete 10 GB card would not hold it. The shared memory that looks like this machine's weakness is the thing that makes the model fit.

What the GPU bought was narrower than I expected. Prefill went from 54 to 559 tokens per second, roughly ten times. Generation barely moved, 30 against 31.

That split makes sense once you look at where the bottleneck is. Prefill is arithmetic over the whole prompt at once, which is what a GPU is for. Generation is streaming weights out of memory one token at a time, and the iGPU reads the same RAM over the same bus the CPU was already using. Nothing about Vulkan widens that bus.

Mixture-of-Experts, and why it's the whole reason this works

A dense model reads every one of its parameters to produce every single token. A 30B dense model moves 30B parameters' worth of bytes through the memory bus for each token it writes. That's why CPU inference on big dense models is hopeless. It isn't short of arithmetic. It's short of memory bandwidth.

A Mixture-of-Experts model splits most of its layers into many small sub-networks, the experts, and a router picks a handful per token. Qwen3-Coder has 128 experts and activates 8:

   token in
      │
      ▼
   ┌ router ┐          128 experts, 8 chosen
   │        │
   ▼        ▼
 [e17]    [e42]  ...   8 of them run
   │        │          120 sit idle and are never read
   └────────┴──> token out

The model holds 30.5B parameters but only about 3.3B take part in any given token. Memory-wise you pay for all 30B, because every expert has to be resident in case the router picks it. Speed-wise you pay for 3.3B.

That split happens to be the exact shape of this machine: RAM it has, memory bandwidth it does not. A dense model with comparable knowledge would be ten times slower here, and a dense model at this speed would be far dumber.

The model, and why this one

qwen3-coder:30b. 30.5B parameters, 3.3B active per token, 262k context. The model card is blunt about the part that matters most here:

supports only non-thinking mode and does not generate <think></think> blocks in its output

The first model I ran was qwen3.6:35b-a3b, which reasons. Same prompt, both on the GPU:

The generation rate is basically identical. qwen3.6 simply wrote 27 times more text to answer the same question, almost all of it reasoning nobody reads.

Ollama can suppress that per request with the think field, but kagent doesn't send it, and there's no server-side switch. The Modelfile route doesn't help either: the model ships a built-in renderer rather than a text template you can override. Changing the model was the only lever I had.

Before swapping this for whatever comes next

Something will eventually be billed as the strongest model in the 30B class. glm-4.7-flash already is, and I checked it against this one:

Same speed class, since both activate around 3B. The deciding line is the last one. GLM's standing in benchmarks comes partly from reasoning, and reasoning is what cost eleven minutes here. A model that can't regress into it beats one that merely defaults away from it.

So the test for a replacement isn't the leaderboard. It's three questions: does it activate about 3B per token, does it do tool calling, and can it be made to shut up and answer.

Where this ended up

Same kagent question, measured through an afternoon of changing things:

The thread pinning is worth a note of its own. Left alone, llama.cpp reads the host topology through the container and picks 8 threads, which then contend for cache on 6 cores. Setting num_thread to 6 took prefill from 42 to 54 tokens per second before the GPU was involved at all.

Look at the last two rows. The iGPU and a warm CPU cache land in the same place. The GPU's real win is that it gets there on a cold prompt, every time, instead of only when the cache happens to be warm.

The eleven minutes are gone, but the ceiling is still there. Generation sits at 30 tokens per second and no configuration moves it, because system RAM bandwidth is the wall. A long answer takes as long as it takes. kagent works within that. An agent chaining many tool calls, each one growing the prompt, would not.