Local Frontier Inference — a 176B Model on Two 2080 Tis¶

Every local model tried against this NetBox agent had failed on the simple tier. That made model-handoff routing moot: there was nothing to hand off from. This page covers the run that reversed that premise, what it cost, and the one tuning flag that turned out to matter more than a five-month llama.cpp upgrade.

Why this model has a shape that fits¶

Qwen3.8-Flash-Next — internally qwen4_exp, an open-weight preview of the Qwen 4 architecture — splits into two very different halves:

part	size	what it does
MoE network	125B total, ~6B active/token	512 experts, 10 routed + 1 shared per token
N-gram embedding table	51B	a lookup table, not a network

The lookup table is the trick. It stores what common token trigrams usually mean, so the model reads the answer instead of recomputing it layer by layer. A lookup costs no multiplication, and the rows needed are known before the first layer runs — so the table can live in system RAM (or on SSD) while only the thinking half needs to be near a GPU. That is why a 176B-parameter model has a memory profile a commodity box can satisfy.

The idea is not original to Qwen and they say so: DeepSeek published Engram (arXiv 2601.07372, Jan 2026) and Google's Gemma 3n shipped per-layer embeddings — llama.cpp's implementation still borrows Gemma's tensor naming. What Qwen did was ship it at frontier scale in open weights.

The hardware, and the constraint that actually binds¶

2× RTX 2080 Ti (21.1 GiB VRAM total, compute 7.5, PCIe 3.0, no NVLink), 2× Xeon Silver 4210 (20 physical cores, 2 NUMA nodes, DDR4), 376 GB RAM, NVMe.

The published community reports for this model are dominated by people fighting RAM limits. That constraint simply does not apply here: at 105 GiB the UD-Q4_K_XL quant sits entirely in memory with room for the 188 GB Q8_0 if wanted. The binding constraint is VRAM: 21 GiB against a 105 GiB model means the experts live in RAM permanently, and the cost of that is measurable.

llama-server -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL \
  --numa isolate -t 10 \
  -ngl 999 -ncmoe 48 \
  -ot 'per_layer_token_embd=CPU' \
  -c 32768 -fa auto --jinja \
  --host 127.0.0.1 --port 58123

-ngl 999 sends everything to the GPUs, then -ncmoe 48 claws the expert tensors back to RAM and -ot per_layer_token_embd=CPU keeps the ~25–30 GB n-gram table off the cards. Dense layers, attention and the output head stay resident in VRAM.

The tuning flag worth more than the upgrade¶

The llama.cpp checkout was 2362 commits stale, so it was rebuilt against current upstream to see whether the measured memory-bandwidth ceiling was an artefact of an old binary. It was not — decode measured flat to slightly worse on the new build. The rebuild's value was answering the question, not raising the number.

What did raise the number was a default nobody had touched. llama.cpp sets --numa to disabled, so worker threads straddle both sockets and stream expert weights across the interconnect. On a dual-socket box that is expensive:

config (MoE analogue, `-r 5`)	decode tok/s
CPU-only, default	9.26
CPU-only, `--numa isolate`	13.57 (+47%)
experts-on-CPU, default	13.71
experts-on-CPU, `--numa isolate -t 10`	21.38 (+56%)

Thread count matters just as much and in the opposite direction to intuition: under isolate, -t 5 → 13.40, -t 10 → 21.38, -t 20 → 17.88; without it, -t 40 collapses to 5.89. Ten threads is exactly the physical core count of one NUMA node. Hyperthreads hurt.

Two corollaries fell out. Apparent per-GPU speed differences were a NUMA artefact — an 18.00 vs 12.28 gap between two identical cards shrank to 17.91 vs 16.44 once threads were pinned. And -sm tensor (tensor parallelism) measured slower than the default -sm layer, because NCCL is absent and its only packaged build targets a different CUDA major version.

Effective memory bandwidth rose from ~19 GB/s to ~28 GB/s — still only ~28% of the 101.5 GB/s STREAM triad this machine achieves, which is where the remaining headroom lives.

The test: three questions, one per tier¶

Not a benchmark. ADR-0037 established that 90 paired items resolve only ~9pp; three questions resolve nothing. The question here is narrower: can it complete a real multi-turn MCP tool loop and get the right answer?

Run through the agent's existing LLM_BACKEND=llamacpp path with no application code changed.

tier	time	tool calls	budget	coverage
simple	241.9 s	2	3	1.0
medium	198.1 s	2	7	1.0
advanced	998.7 s	20	12 ✗	1.0

3/3 correct. The medium answer produced the full per-rack breakdown (R01=11, R02=10, R03=8) matching the reference exactly, and spotted that one host is unracked — which is why the site total is 30 while racked devices number 29.

The advanced question is the interesting one. It asks which hypervisor has exactly one of two power supplies cabled, and to which outlet — a trace from power port through cable to PDU outlet. The model answered sea-dc1-esx05, PSU0 → sea-dc1-pdu03 Outlet 1, and classified every decoy the reference names correctly (esx01/esx02 cabled 2-of-2; esx03/esx09 0-of-2). It then went further than asked: it quoted the cable's own label — "single supply - D10" — as evidence the single feed is deliberate rather than a data gap, and flagged the six unconnected hosts as a records-audit item.

What it cost, which is the actual finding¶

Correctness did not separate the tiers — all three scored 1.0. Cost separated them by 10×.

Simple and medium each needed 2 tool calls. Advanced needed 20, against a budget of 12, over 16.6 minutes. That asymmetry is the routing signal: single-anchor lookups are cheap locally, multi-hop traces are not. It is direct local evidence for ADR-0037's conclusion that efficiency, not correctness, is what discriminates once models are competent.

Measured throughput: decode 11.2 tok/s, prefill 64.4 tok/s at 1.5k tokens, ~48 tok/s on realistic ~9k agent prompts. Prefix caching matters enormously in a tool loop — the first turn of a question costs 3–4 minutes cold, subsequent turns ~23–33 s, because the system prompt and tool schemas are reused.

Extrapolated to the full v5 set (30/30/30) from these three questions: ~12 hours per run. That estimate is superseded. Measured later across 12 questions: 412 s/question at 32k (~10.3 h, but that configuration loses 1 question in 4 to context) and 775 s/question at -c 131072 --no-kv-offload — ~19.4 h per run, ~2.4 days at the 3× replication standard.

All of that is now superseded by an actual run. The full 90-question set has been executed locally end to end: 6 h 47 m, 271.6 s/question, corr 0.906, 0 errors. See The full harness, finally run below.

The context window was the binding constraint¶

The three-question proof above ran at -c 32768. Scaling to 12 questions (4 per tier, 6/6 across both data islands, full evaluator set) showed that window was not merely a floor to clear — it was the thing limiting the result:

	`-c 32768`	`-c 131072 --no-kv-offload`
correct	8 / 12	11 / 12
wrong	1	1
lost to context	3	0
overflows / truncations	2 / 1	0 / 0
peak context	ceiling 32,768	44,295
tool calls	60	88
wall time	82 min	155 min (1.9×)

All three context-lost questions recovered and scored correct: point-to-point circuits (15 calls, 1564 s), branch-site firewalls (13 calls, 1862 s), changelog deletions (6 calls, 819 s). A fourth flipped wrong → correct. The single regression — a changelog count that went correct → wrong using fewer calls (4 → 2), finding 3 records where the baseline found 4 — is the model under-searching at n=1, not a context effect.

The fix uses the resource this box has in surplus. --no-kv-offload puts the KV cache in system RAM — 376 GB against 21 GB of VRAM — buying 4× context for ~37% of decode speed (11.2 → 7.0 tok/s). It loaded in 20 seconds.

WITHDRAWN — that trade-off was a false choice, and it cost ~half the throughput for nothing. The 128k window never needed the offload. Only 12 of 48 layers are full attention (indices 3, 7, 11 … 47); the other 36 are Gated DeltaNet, whose recurrent state is sized by sequences rather than tokens and so does not grow with context. Those 12 layers use 2 KV heads at 256 key/value length — 24 KiB per token, so -c 131072 is 3.00 GiB of attention KV plus ~0.46 GiB of recurrent state. Measured by VRAM subtraction: resident 14.7 → 17.4 GiB, ~3.1 GiB for both caches, ~3.3 GiB headroom left, no OOM.

	KV in system RAM	KV in VRAM
prefill (matched ~8.7k prompt)	89.86 tok/s	126.42 tok/s	1.41×
decode	5.1–7.5 tok/s	~10.8 tok/s	~2×
Q2 wall clock	171.2 s	91.6 s	1.87×
Q3 wall clock	665.3 s	329.4 s	2.02×

Correctness held — the advanced question still named sea-dc1-esx05 PSU0 → sea-dc1-pdu03 Outlet 1 and still declined to invent a PDU→feed→panel hop the data does not contain. Zero truncations. The first question after restart shows a 1.24× regression and is excluded: it paid a full 8,743-token prefill on a cold cache, 81.8 s in a single call, against a baseline whose server was already warm.

The flag was correct when it was set — the failure it fixed was -c 32768 losing 3 of 12 questions, one of them silently. Raising the window solved that, and nobody went back to ask whether the offload was still earning its cost. A workaround outlives the problem it was built for unless something forces the re-check.

Every runtime figure on this page — 775 s/question, ~19.4 h, ~15.8 h, and the 82 → 155 min comparison, wherever they appear above or below — was measured with KV offloaded, and is now superseded. The 12-question benchmark and the full 90 have since been re-run: 6 h 47 m for the full set, 271.6 s/question, corr 0.906. Those older figures are kept because the comparisons built on them (the 1.23× -ub 2048 result, the 82 → 155 min context trade) remain valid against their own baselines.

Replicated, and the cold first turn proven¶

A second run of the same three questions — launched from the committed serve_qwen4exp.sh rather than a hand-typed command line, so it validates the artefact actually in use — reproduced the result. 3/3 correct again, one accumulating thread, peak context 9.5k / 10.9k / 19.1k, all traces correctly labelled qwen3.8-flash-next-UD-Q4_K_XL:

	KV in system RAM	first VRAM run	replication
Q2 (warm, 2 LLM calls both runs)	171.2 s	91.6 s	76.9 s — 2.23×
Q3	665.3 s	329.4 s	413.9 s — 1.61×
Q1 (cold first turn)	101.6 s	126.1 s	155.9 s — excluded

Q2 is the comparison that carries weight: warm cache, same call count in both runs, and the replication came in faster than the first.

The cold first turn is now a measurement, not an excuse. Per-call timings inside Q1:

call 1  108.6 s   8,743 prompt tokens
call 2   20.4 s   8,958 prompt tokens
call 3   23.7 s   9,534 prompt tokens

More tokens, one-fifth the time, as soon as the prefix cache fills — and 152.7 s of the 155.9 s wall was server-side, so this is not harness overhead. A benchmark whose first question follows a server restart is measuring cache warm-up. Both VRAM runs show it; it is reproducible, and it should be discarded rather than averaged in.

The larger caution is variance, not speed. The same Q3, same configuration, took 9 LLM calls in one run and 14 in the other — and tool execution was only 5.4 s of 413.9 s, so the difference is round-trips, not tools. Both runs issued four netbox_graphql_schema introspections, and that is a reason to treat any single advanced-tier timing as one sample rather than a figure.

Correction — I called those introspections "re-learning the schema mid-query". That was wrong. The netbox-graphql skill requires them: "for any type or field you are not 100% sure of, call netbox_graphql_schema FIRST — never guess", as step 2 of its Route → Discover → Write → Execute workflow. And the tool caches the built schema per process, so after the first call they are local dictionary lookups, not network round-trips.

Measured across the full 90-question run: 44 introspections against 65 queries. At the marginal cost of a tool call (~5.3 s, from the A/B where 42 extra calls cost 3.7 min) that is ~3.9 min of a 6 h 47 m run — 0.95%. On the clearest single case, one introspection replaced 19 MCP calls.

What survives of the criticism: discovery is per-question reasoning and never amortised. The schema is cached in the tool, but the model re-derives which type it needs on every question. The obvious fix — inlining common types into the skill — was costed and rejected: twelve types is roughly 4,500 tokens on every request, paid by all 90 questions to save ~1% on the 22 that use GraphQL, and it would undercut the skill's deliberate "grammar + runtime discovery, not a fixed schema" design.

The detour: a tool-result cap, built at the wrong layer¶

Before testing the window, a per-result size cap was added to the MCP wrapper on the theory that oversized payloads were the problem. It cost two ~87-minute runs and was reverted.

First it was inert. langchain_mcp_adapters declares response_format="content_and_artifact", so tool coroutines return a (content, artifact) tuple; the guard measured only str/list and silently skipped every MCP result — three oversized results, zero cap firings. The unit test fed it a bare string, validating an assumption about the adapter instead of the adapter's contract. A test written from the same misunderstanding as the code cannot catch that misunderstanding.

Then it was mis-sized. At 60,000 chars (~15,000 tokens) one permitted result consumed ~63% of the 23,768-token working budget — still allowing a truncation, while turning a question the baseline had answered in 3 calls / 141 s into a 65-minute retry loop. The model behaved correctly throughout: it never reissued an identical call until the fourth firing, decomposed the changelog by action type, and discovered that limit belongs inside filters. The guidance was the problem.

The reusable lesson is about ordering, not about caps: test the cheap hypothesis first. Raising the context window took 20 seconds to verify and made the entire cap exercise unnecessary. A per-result cap, if ever needed, must be sized as context budget ÷ expected call count — and even then it treats the symptom.

Tuning: one flag worth 3.8×, and one that actively hurts¶

With the context ceiling gone, the remaining question was whether latency could be improved at all. Four configurations, measured on a 9,093-token agent-shaped prompt (NetBox records followed by a request to tabulate them), one server at a time, baseline re-measured in the same session:

config	prefill tok/s	decode tok/s	wall	drafts accepted
baseline	34.0	5.0	318.7 s	—
`-b 2048 -ub 2048`	129.7	5.9	113.7 s	—
`--spec-type ngram-simple`	36.9	3.8	314.4 s	0 / 163
both	101.4	3.7	159.7 s	0 / 147

-ub 2048 is worth 3.8× on prefill and 2.8× on wall clock. llama.cpp only copies CPU-resident expert weights to the GPU once a microbatch is large enough to amortise the PCIe transfer. The default -ub 512 never reaches that threshold, so every microbatch pays full freight. With 112 GB of experts in system RAM, that single threshold dominates prefill — and prefill is where this agent lives: 187,272 prompt tokens against 21,825 generated in the 12-question run.

Measured on a second full 12-question run, not modelled: 155.1 → 126.3 min (1.23×), at identical correctness (11/12, zero context failures) and 77 tool calls against 88. Extrapolated to the full v5 set: 19.4 h → ~15.8 h.

The first projection said ~105 min and ~12.6 h. It was optimistic by about 20%, and the reason is instructive: it applied the flag's isolated 3.8× prefill gain to all 187,272 prompt tokens, when ~85% of those were already served from the prefix cache (see below). Only the genuinely-new remainder and each question's cold first turn get faster. A 3.8× gain on an isolated measurement became 1.23× end to end — and decode, which this flag does not touch, now accounts for roughly two thirds of what is left.

The n-gram dead end, recorded so nobody repeats it¶

--spec-type ngram-simple looks made for this workload. The agent constantly echoes device names, IDs and JSON keys straight back out of tool results, and n-gram drafting is exactly the technique for repetitive output. It needs no draft model and costs no VRAM.

It produced 163 drafts and accepted zero, costing 24% of decode because the model pays verification for drafts it always rejects. Combined with -ub 2048 it degrades both axes (prefill 129.7 → 101.4, decode 5.9 → 3.7).

The reason is mechanical: ngram-simple needs an exactly-repeating 12-token run to predict the next 48, and a table of distinct device names has almost none. A 0% acceptance rate is not a tuning problem — no --spec-ngram-*-size-n value fixes it. Draft-model speculation is separately impossible: common/speculative.cpp throws on vocab mismatch and no qwen4exp-vocab draft model exists (the MTP head PRs are still unmerged).

Two negatives worth as much as the win¶

Prompt caching was already doing its job. The obvious hypothesis — that a ~9k system prompt plus tool schemas was being re-read every turn — is false. Median LCP similarity 0.956 across 75 slot selections, 75 of 76 requests served warm, mean context 16,405 tokens of which only 2,432 were actually prefilled. ~85% of every prompt came from cache. The remaining prefill is genuinely new tool-result text, which no caching strategy can remove. That lever was spent before it was pulled.

MoE CPU decode runs at ~19% of memory bandwidth and there is no available fix. Measured 19–20 GB/s effective against a 101.5 GB/s STREAM triad. The matched remedy is NUMA weight mirroring — duplicate weights per node, which 376 GB comfortably holds — and GGML_NUMA_STRATEGY_MIRROR exists in ggml's enum and nowhere else in the source: reserved, unimplemented, unreachable. Switching runtime does not help either: KTransformers requires Ampere+ and its headline numbers need AMX this Cascade Lake host lacks; vLLM and SGLang cannot host 112 GB on 21 GiB of VRAM at all.

Update, 2026-10-17: the decode half of that sentence stands; the prefill half did not. See the expert-offload question, measured and ADR-0042.

Two things recorded because they were wrong¶

The context failure was the operator's, not the model's. The advanced question first died with request (18181 tokens) exceeds the available context size (16384 tokens). The 16k limit was chosen deliberately — quantized KV could not be confirmed safe on this hybrid Gated-DeltaNet architecture, so f16 KV and a smaller window were the conservative choice. At -c 32768 the same question ran to 28,106 tokens and answered correctly. ≥32k is a requirement for advanced items, not a preference. WITHDRAWN — understated: 32k still lost 3 of 12 questions to context. The measured requirement is ≥128k ~~via --no-kv-offload~~ — and it needs no offload: 131k of KV is only 3.00 GiB on this hybrid architecture and lives in VRAM (see above).

The predicted tool-calling fragility did not appear — and this test could not have detected it. Third-party reports describe repetition loops, malformed <tool_call> emission and spurious mid-task termination in long agent loops. Across 17 logged turns: none of it, and zero truncation. But the reported rate is ~0.7% per tool turn, so 24 tool calls yields ~0.17 expected failures. Seeing zero is exactly what you would observe whether or not the problem is real. The earlier claim that this fragility disqualified the model was too strong and is withdrawn — it is not refuted.

Where this leaves routing¶

The premise has moved. It is no longer "local models cannot do this" — this one did, including the hard item. It is now a cost question with a measurable shape:

Cheap locally: single-anchor lookups, 2 tool calls, ~3–4 minutes.
Expensive locally: multi-hop traces, 13–20 tool calls, 17–31 minutes.
Slow but no longer impractical: the full 90-question harness ran locally in 6 h 47 m (271.6 s/question) at corr 0.906 with zero errors — an overnight job rather than a two-day one. The earlier ~19.4 h and ~15.8 h figures were measured with KV offloaded and are superseded.

Note the direction of the earlier trade. Fixing the context ceiling made the agent slower and more expensive — 60 → 88 tool calls, 82 → 155 minutes — precisely because it stopped hitting a wall and started doing the work. A cost measured on a configuration that silently drops a quarter of its questions is not a cost worth quoting.

That maps cleanly onto the anchor-count rule already used for GraphQL-vs-MCP routing: count the anchor objects. A handoff policy keyed on the same axis — local for one named object, escalate for set-anchored multi-hop work — now has direct supporting evidence rather than an untested premise.

The full harness, finally run¶

Every runtime number on this page before this section was an extrapolation from 3 or 12 questions. The full set has now been executed: 90 questions, one model, serial, 6 h 47 m.


wall clock	6 h 47 m (271.6 s/question)
correctness	0.906 — simple 0.917 / medium 0.933 / advanced 0.867
completeness	0.944 · entity coverage 0.866 · tool calls 5.52
errors	0 · truncations 0 · overflows 0
peak context	75,571 of 131,072
decode	13.47 t/s median across 507 requests; 10.25 at peak context

Where that sits against the cloud models, on the same 90 items and using the re-scored figures (three reference-wording fixes raised the cloud numbers after their runs):

model	correctness
deepseek-v4-pro	0.944
kimi-k2.6	0.928
deepseek-v4-flash	0.911
local qwen3.8-flash-next (Q4_K_XL)	0.906
qwen3.5:397b-cloud	0.817

0.5pp below flash — which this benchmark cannot resolve. ADR-0037 established that the set discriminates gaps of ~9pp and above and cannot separate the ~3pp between closely-matched models. So the defensible claim is indistinguishable from deepseek-v4-flash here, not "matches" and not "beats". The 8.9pp gap over qwen3.5:397b-cloud is within the set's resolution.

My projection was wrong again, and the reason is worth more than the number. From the 12-question sample I predicted 3.2–4.5 h; the answer was 6.78 h — optimistic by 1.5–2.1×. That sample peaked at 22,532 tokens of context; the full set reached 75,571. The sample was stratified across tiers and still missed the expensive tail. A subset representative of difficulty is not thereby representative of cost.

Three of the seven failures are one bug wearing three hats. On three questions the agent looked for "Halvorsen Logistics" as a circuits provider, found nothing, and replied "Halvorsen Logistics was not found in NetBox" — after 2–5 genuine tool calls, about the seeded tenant that 57 of the 90 questions rely on. Every other question found it. That is a scoping failure that yields a confident denial rather than an error: the same shape as the silent truncation recorded above, and a fixable routing bug rather than three separate capability misses.

Attribution caveat. This run differs from the ~15.8 h figure in four ways — KV in VRAM, reasoning_effort="low", max_tokens 8192, and a declared model profile. The 2.33× improvement belongs to the configuration as a whole. The clean per-flag figure for removing --no-kv-offload is still the 1.87×/2.02× from the three-question A/B, where nothing else varied.

The session harness: one wrong scope, four wrong answers¶

Every figure above comes from a harness where each question starts cold — fresh thread, empty context. That is not how the agent is used, and it makes the benchmark structurally blind to anything that goes wrong because turns share a conversation. tests/eval/run_session.py runs 11 turns down one accumulating thread, using v5-verbatim question text so matched turns score against the existing references.

	site-first	tenant-first
mean correctness (10 scored turns)	0.750	0.950
tool calls	39	38
wall clock	36 min	43 min
— of which agent turns	26 min	32 min
peak context	46,376	58,519
decode	14.83 → 12.62 t/s	13.77 → 11.93 t/s
compaction fired	0	0

Three turns scored 0.5 that had each scored 1.0 in the full v5 run, and all three trace to one origin. Turns 1–3 were scoped to site HVL-SEA-DC1, and turn 3 fetched every device at site_id: 25. Turn 4 then asked a tenant-scoped question — how many PDUs does Halvorsen Logistics have — and answered "there are 6 PDUs at the site" straight from that inherited list, never issuing a tenant query at all. Six PDUs and 48 outlets, against a true 12 and 96. Turns 5, 6 and 8 inherited the wrong denominator.

Reordering the same 11 questions recovers it. With the site questions moved to the end, turn 1 resolves tenancy.tenant → id 14, calls netbox_get_objects(dcim.device, tenant_id=14), and answers 12 PDUs, 8 outlets each. That question scored 0.0 in v5 and 0.0 in the first session — in v5 with the identical "Halvorsen Logistics was not found in NetBox" denial recorded above — so it is answered correctly here for the first time in any condition. Three recoveries, seven ties, zero regressions, at the same tool cost.

The load-bearing evidence is a turn that did no work. In both runs one question is answered with zero tool calls, purely from context. Contaminated it scored 0.5; clean it scored 1.0, correctly naming "the other 9 PDUs (HQ ×3, Portland, Spokane, Tacoma, and sea-dc1-pdu04/05/06)". Identical behaviour, opposite outcome. Context reuse is not the defect — reusing a wrong population is. The failure mode is propagation: cold, this defect costs one question; across a session it cost four. It also has a direction — with the site questions at turns 8–10, after seven tenant turns, all three still scored 1.0 and turn 8 re-queried site_id: 25 rather than inheriting.

Question order also decides GraphQL routing. The site-first run made zero GraphQL calls; the tenant-first run used it on three turns. The skill was never being declined — a cold cross-model question is what triggers it, and in the first ordering none arrived cold. It then routed badly: turn 1 burned four malformed queries (id__in, filter:, poweroutlet_count) before falling back to MCP, and the fallback is what produced the correct answer.

The projection missed a fourth time, and for the first time in the other direction. I predicted 3.5–4.5 h; it took 36 minutes, wrong by ~6× where the previous three misses were all optimistic about cost. The cause mirrors the v5 miss: sized from the 90-question run's expensive tail (75,571 peak context) when one thread of 11 related questions reaches 46,376 and reuses almost everything.

Limits. One run per ordering. Three improved, seven tied, none worse is p ≈ 0.25 on a sign test, so the evidence is the directly observed tenant_id query rather than the tally. One turn scored 0.5 in both orders despite an answer giving 12 PDUs / 96 outlets / 17 in use — substantively what v5 said when it scored 1.0 — so that is likely judge noise, left as measured rather than argued up.

The expert-offload question, measured rather than argued¶

The limitation above said no fix was available. A month later the question was put properly: the colibri engine (42k stars in three months for streaming experts from NVMe), the GPU expert cache llama.cpp merged on 2026-10-07 (PR #29887, 1.6–2.2× decode on 4090/5090 for this model), and ~55 papers on expert caching, prediction and CPU-GPU co-execution. The full report, with sources, is in the app repository (docs/development/2026-10-17_moe-offloading-research.md). Its framing result came first: every one of those techniques acts on decode, the cache is bypassed above 32-token batches, and this agent's worst turns are prefill — a cold 8.7k prefix, a 20k-token tool result. By Amdahl's arithmetic on the Gate-3 turn, a 1.5× decode gain is worth ~18 s of 167.

The cache was measured anyway, and lost at every size that fits. llama-bench on the production flags, three repetitions, cache split across both cards as the server would split it:

configuration	pp2048	tg128	hit rate
no cache	189.0	15.92	—
2 GiB	187.5	4.69	48%
4 GiB	189.9	7.87	71%
6 GiB	189.3	10.19	80%
2.5 GiB, one card, pinned RAM, process on that card's socket	412.7	13.31	60%
2.5 GiB, one card, pinned RAM, process on the far socket	413.6	13.47	60%

The last pair is the mechanism: moving the process across the socket changed the cached decode by 1%, so PCIe locality is not the problem. The miss path is — ~250 uploads of ~2.7 MB per token, serialised against compute; at 2 GiB the cache moved ~290 GB over the bus for 384 tokens. Six GiB is more VRAM than the 131k-context server has, and it is still 36% slower than nothing.

Colibri solves a problem this box does not have. It streams experts from disk for models that exceed RAM; this model is RAM-resident. Its own README puts this model at 1.9–4.0 tok/s on CPU; on a RAM-resident MoE and an RTX 3070 it is slower than llama.cpp-under-Ollama (30.0 vs 38.1 tok/s) at worse perplexity for equal bits. Skipped, with the numbers recorded so it stays skipped.

What did pay was not offloading. Two things, both free:

A newer build. b11053 → b11713 (660 commits, fused shared experts, grouped top-k): tg128 11.48 → 15.92, pp2048 176.9 → 189.0 on identical flags.
--load-mode none. Copy the CPU-resident experts into pinned RAM instead of mmap-ing the GGUF, so the per-microbatch expert uploads are DMA. pp2048 189 → 390; decode unchanged. The old server had printed "tensor overrides to CPU are used with mmap enabled - consider using --load-mode none" at every start for a month. That is the error of this page: the warning was in the log the whole time the page said no fix existed.

And one that was free but surprising: --numa isolate pins to whichever node the process starts on. Started on node 1, decode is 15.7; on node 0, 11.7, prefill equal. The serve script now starts under taskset -c 10-19,30-39, and the page cache was dropped once so the 74 GB pinned buffer sits on the same node as the threads.

On real traffic, from the server's own timing lines — the old server's 24-day log against the new server's first day:

	legacy	new
prefill, ≥256 new tokens, aggregate	96.9 tok/s	218.5 tok/s
prefill, ≥4,000 new tokens	113.3	242.0
decode, ≥32 generated, aggregate	13.64	14.60

Eight v5 questions through the agent, same evaluators, all eight correct both times: 1,285 s → 1,180 s summed (1.09×), but 35.7 → 28.5 s per model call and 3.0 → 1.9 ms per prompt token. The gap between those two numbers is planning variance — 12 tool calls where the September run made 4, three times the tool-result text on another question, and the cold prefix landing on the first item. Questions where the model made the same number of calls both times came in 1.22×, 1.34× and 1.56× faster. Per call, the server is faster; per turn, the call count is the larger term, which is the same conclusion ADR-0041 reached from the other side.

Then the full harness, the same night. All 90 questions, same evaluators and judge as the 2026-09-23 run: 6 h 47 m → 5 h 18 m of wall clock, agent time 1.34× faster, 73 of 90 questions quicker, correctness 0.906 → 0.922 (five up, four down — the same level), tool calls 5.52 → 4.88, zero errors. By tier: simple 1.12×, medium 1.23×, advanced 1.53× — the tier whose turns prefill the largest tool results, which is the axis the flag doubled. Per model call, 40.8 → 33.9 s; per prompt token, 2.77 → 2.12 ms; September's 26-minute outlier took 12.7 minutes.

The run also answers an attribution question, because ADR-0041 landed between the two harness runs. It is not the driver: September used __in in 8 of 497 tool calls on 5 questions; October used it never, read more tool text (6.2M vs 4.9M characters) and was still faster; and the quantities that moved are per-call and per-token rates a validator cannot touch. Its bounded share is in the call count (462 → 411), alongside ordinary planning variance.

Limits. One full run against one full run, neither replicated. Decode's 11.5 → 15.9 in llama-bench does not survive the server's overheads at 131k context (13.6 → 14.6 on real traffic). The node-1 advantage is measured, not explained. kernel.numa_balancing=0, the last open item from ADR-0038, was set and measured: decode 14.45 → 14.72 tok/s median over five 200-token runs, inside the spread of two runs with it on. Pinned and node-local, there is nothing left for the balancer to move.