Local model performance on your hardware

generation:

5.8-6.8 tokens/s

quant:

int4 (safetensors)

User reports GLM-5.2 at 5.8-6.8 tok/s single-request decode on 6x RTX 5090 with dual Xeon Silver 4510 and 251 GB RAM. Setup is colibri with int4 container, full expert residency across VRAM+RAM, CUDA_EXPERT_GB=auto PIN_GB=all, COLI_CUDA_PIPE=2 TC_W4A16, DRAFT=0. TTFT about 13 s, expert hit 89-100%. The model is a 744B frontier-class model on disk at ~370 GB; cold decode costs ~11 GB disk reads per token. On a 2-socket Xeon Silver 4510 host with 6x RTX 5090, selective COLI_NUMA=1 raised effective CPU-expert bandwidth from 42.42 to 58.26/65.89 GB/s and greedy decode from 7.66 to 9.02/9.17 tok/s (64 tokens, TEMP=0 DRAFT=0, byte-identical output).

generation:

0.2 tokens/s

quant:

fmt=4 grouped int4, gs=64

User reports GLM-5.2 744B at 0.2414 tok/s (10-run median) on an RX 9070 XT 16GB with colibri's Vulkan backend at its default 320 resident experts. Setup is colibri's Vulkan backend with COLI_VK_EXPERTS=320, MTP drafting at 2.00 tokens per forward, fmt=4 grouped int4 weights with group size 64, experts streamed from a Gen3 x4 NVMe and 64GB DDR4 system RAM, Resizable BAR on, 40 generated tokens. The box is storage-bound. The HIP arms (0.3189 tok/s at 6 GB, 0.3331 at 12 GB) ran with MTP disabled by a guard bug, and the author has withdrawn that Vulkan-vs-HIP comparison. 640 resident experts gave 0.2475 tok/s, and with Resizable BAR off the Vulkan run fell to 0.11 tok/s.

generation:

7.3 tokens/s

prompt processing (prefill):

135.0 tokens/s

quant:

UD-IQ2_M

kv:

q8_0

User reports GLM-5.2 UD-IQ2_M at ~7.3 tok/s decode and 135 tok/s prefill on 4x RTX 3090 (1 Ti + 3 FE), 96GB total VRAM with only ~83GB used. Setup is llama.cpp master with the glm-dsa arch, a 223GB on-disk model at 744B total / 40B active, expert offload to CPU, explicit expert placement via -ot to avoid OOM, and a ~6GB KV cache at 128K that scales linearly. Decode is bound by CPU compute, not bandwidth. IQ1_M and IQ2_M are identical in A/B tests (7.30 vs 7.29 tok/s), and 6 vs 12 CPU threads gave +22% (5.83 vs 7.14 tok/s). Thinking off gives ~5-6x wall-clock speedup.

generation:

14.5-15.2 tokens/s

prompt processing (prefill):

512.0 tokens/s

quant:

NVFP4

kv:

fp8

User reports GLM-5.2 NVFP4 at about 15 t/s decode on short context and about 13 t/s at long context on a 4x DGX Spark setup, with about 512 t/s prefill. Setup is 128K context with TP4/PP1/DCP4/MTP1 and an fp8 KV cache.

generation:

6.0 tokens/s

quant:

Q1_S

kv:

Q8

User reports GLM-5.2 Q1_S outperformed Qwen 3.6 27B Q8 in code generation quality despite the lower quant. Both models run under the pi harness. GLM-5.2 Q1_S ran at ~6 t/s at low context down to 3 t/s at 100k context, while Qwen 3.6 27B Q8 ran at ~60 t/s. GLM FP via OpenRouter was also tested, but Q1_S held up well.

User reports running GLM 5.2 on a system with 5x RTX Pro 6000 and 1x RTX 5090. User reports 98-99% task completion. User notes high heat and cost.

generation:

24.0 tokens/s

prompt processing (prefill):

475.0 tokens/s

quant:

NVFP4

User reports MTP3 as the default and MTP4 as the peak. Setup includes a bug fix for a draft parallel config missing a DCP copy. Prefill reaches about 475 tps and decode at bs=3 reaches about 48 tps.