Local model performance on your hardware

generation:

64.8 tokens/s

prompt processing (prefill):

1973.0 tokens/s

quant:

2.05bpw EXL3 (EXL3)

kv:

8-bit

User reports 64.8 t/s generation and 1973 t/s prompt processing. Source evidence (verbatim): Source: https://github.com/adrienbrault/glm5.3-flash-2x-rtx5090 # GLM-5.3-Flash (321B MoE, 2.05bpw EXL3) on 2x RTX 5090 + 64 GB DDR5, ExLlamaV3 + TabbyAPI Hardware: two NVIDIA RTX 5090 (32 GB each, PCIe 5.0 x8), AMD Ryzen 7 9800X3D, 60 GB usable of 2x32 GB DDR5-6000. Engine: ExLlamaV3 dev 5783a93 served through TabbyAPI. Model: GLM-5.3-Flash by zai-org, 321B parameters with about 18B active, turboderp 2.05 bpw EXL3 quant (76 GB), with experts offloaded to system RAM (96 and 102 CPU-resident experts per MoE layer out of 288) and 8-bit KV cache at 262144 tokens. Single-stream decode with MTP draft enabled (DRAFT=1, DRAFT_N=1): 64.8 tokens/s median of ten requests, 1,024 forced output tokens at temperature 0, median of three boots. Cold-prompt prefill measured at 1,973 tokens/s at 32,738 prompt tokens. Without the draft, decode with 2, 3 and 4 concurrent streams was 74.4, 92.0 and 98.0 tokens/s in total.

generation:

241.7 tokens/s

quant:

2.50bpw (EXL3)

kv:

8-bit

User reports Qwen3.8-Flash-Next at 241.7 tok/s on code and 254.3 tok/s on prose on 2x RTX 5090. Setup is ExLlamaV3 v1.5.0 with TabbyAPI, 2.50 bpw EXL3 quant, 8-bit KV cache, 262k context, MTP speculative decoding on, 2 cards. Aggregate decode at 4 streams is 592 / 595 tok/s; prefill is 10,034 / 9,865 t/s at 86-172k-token prompts; peak decode power 302.8 W / 285.0 W per card.

generation:

291.1 tokens/s

quant:

2.50bpw EXL3 (EXL3)

kv:

8-bit

User reports Qwen3.8-Flash-Next at 291.1 t/s single-stream decode on code and 273.5 t/s on prose, on two RTX 5090 32 GB GPUs with layer split [30, 30]. Setup is ExLlamaV3 v1.5.0 under TabbyAPI with a 2.50 bpw EXL3 pack, 8-bit KV cache, 262,144-token context and a 983,040-token page pool, using MTP draft depth 3 up to 4 jobs and 2 at 5 to 8 jobs. Per-stream decode falls to 153.2 t/s on code and 156.8 t/s on prose at 4 streams, and 103.4 t/s on code and 103.6 t/s on prose at 8 streams. Aggregate decode across concurrent streams is 292 t/s code and 272 t/s prose at 1 stream, 615 t/s code and 627 t/s prose at 4 streams, and 822 t/s code and 831 t/s prose at 8 streams. Time to first token ranges from 0.13 to 0.14 s at 1 stream to 0.76 s at 8 streams. Cold prefill holds its rate to the top of the window: 199,844 tokens in 18.8 s and 240,047 tokens in 22.8 s. Quality runs on the same configuration give GSM8K 5-shot (n=500) 0.978, SWE-bench Verified 397/500 resolved (79.4%), and tool-eval-bench 84.0 +/- 2.4.