GLM-5.3 320B (18B active) Flash
on 2× NVIDIA RTX 5090 · ExLlamaV3 · 262,144 ctx
Oct 18, 2026
Summary
User reports 64.8 t/s generation and 1973 t/s prompt processing.
Source evidence (verbatim):
Source: https://github.com/adrienbrault/glm5.3-flash-2x-rtx5090
# GLM-5.3-Flash (321B MoE, 2.05bpw EXL3) on 2x RTX 5090 + 64 GB DDR5, ExLlamaV3 + TabbyAPI
Hardware: two NVIDIA RTX 5090 (32 GB each, PCIe 5.0 x8), AMD Ryzen 7 9800X3D, 60 GB usable of 2x32 GB DDR5-6000. Engine: ExLlamaV3 dev 5783a93 served through TabbyAPI. Model: GLM-5.3-Flash by zai-org, 321B parameters with about 18B active, turboderp 2.05 bpw EXL3 quant (76 GB), with experts offloaded to system RAM (96 and 102 CPU-resident experts per MoE layer out of 288) and 8-bit KV cache at 262144 tokens. Single-stream decode with MTP draft enabled (DRAFT=1, DRAFT_N=1): 64.8 tokens/s median of ten requests, 1,024 forced output tokens at temperature 0, median of three boots. Cold-prompt prefill measured at 1,973 tokens/s at 32,738 prompt tokens. Without the draft, decode with 2, 3 and 4 concurrent streams was 74.4, 92.0 and 98.0 tokens/s in total.