Local model performance on your hardware

generation:

20.0 tokens/s

quant:

3.05bpw_h5_ng5 (EXL3)

kv:

Q4

User reports Qwen3.8-Flash-Next 125B at 20.0 tok/s decode on an RTX 4090 at the full 262k context window, with experts streamed from NVMe SSD. Setup is ExLlamaV3 with a 3.05bpw_h5_ng5 quant and Q4 KV cache, 16 GB VRAM and only 8 GB system RAM allocated. At 100k context the same card gives 20.8 tok/s decode and ~1,536 tok/s prefill; short 8k context gives 28-33 tok/s prose and 19-22 tok/s code with ~1,080 tok/s prefill. An RTX 3080 Laptop with 16 GB VRAM and 32 GB RAM reaches ~18-21 tok/s decode and ~650-700 tok/s prefill at 100k context, peaking at 15.5 GB VRAM.

generation:

35.9 tokens/s

quant:

EXL3 4.00 bpw (EXL3)

kv:

FP16

User reports AREX-2, a 27B dense multimodal agent model built on Qwen3.8-27B, at 35.94 tok/s decode on an RTX 4090 24 GB. Setup is ExLlamaV3 with an EXL3 4.00 bpw quant and FP16 cache, 8192-token runtime context, batch size 2, ~17 GiB artifact. The figure is a small local smoke benchmark, not a standardized result; a TabbyAPI smoke test on the same card saw about 49.4 tok/s on a very short request. Long-context quality and memory scaling at 32K/64K/262K were not validated.

generation:

33.6 tokens/s

prompt processing (prefill):

1944.0 tokens/s

quant:

4.05bpw (EXL3)

User benchmarks a MoE CPU-offload patch for ExLlamaV3 that streams expert weights directly from a page-locked shared arena, removing the host staging memcpy. On an RTX 4090 with a Ryzen 9 7900X, the Qwen3.8 Flash-Next 125B-A6B model at 4.05bpw reaches 1944 tok/s prefill at 4096 context and 33.6 tok/s decode at context 0 on Windows, roughly doubling upstream prefill. Setup is ExLlamaV3 with EXL3 4.05bpw weights, 32768 context, chunk size 4096, and CPU-offloaded MoE experts. The patch is PCIe-bound at 26-30 ms per ~1 GiB layer. Prefill figures are medians of 5 runs with min-max ranges; WSL2 reaches 2044 tok/s at 4096 context. Decode varies more with machine state than with the patch.