GLM-5.2 744B (40B active)
on NVIDIA RTX 3090 · llama.cpp · 131,072 ctx
Summary
User reports GLM-5.2 UD-IQ2_M at ~7.3 tok/s decode and 135 tok/s prefill on 4x RTX 3090 (1 Ti + 3 FE), 96GB total VRAM with only ~83GB used.
Setup is llama.cpp master with the glm-dsa arch, a 223GB on-disk model at 744B total / 40B active, expert offload to CPU, explicit expert placement via -ot to avoid OOM, and a ~6GB KV cache at 128K that scales linearly.
Decode is bound by CPU compute, not bandwidth. IQ1_M and IQ2_M are identical in A/B tests (7.30 vs 7.29 tok/s), and 6 vs 12 CPU threads gave +22% (5.83 vs 7.14 tok/s). Thinking off gives ~5-6x wall-clock speedup.