Local model performance on your hardware

generation:

28.6 tokens/s

quant:

exl3-3.05bpw (EXL3)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports GLM-5.3-Flash at 28.55 tok/s decode on an RTX 3090. Setup is ExLlamaV3 with exl3-3.05bpw and 131k context, all-RAM fast mode with CPU offload requiring ~238 GB free RAM. Lab suite passed 6/6 with 28.6 tok/s decode and 989 tok/s prefill; at concurrency 4 the aggregate was 28.62 tok/s (28.45 per stream) because fast mode decodes one stream at a time.

generation:

17.3 tokens/s

prompt processing (prefill):

844.0 tokens/s

quant:

3.05bpw (EXL3)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports GLM-5.3-Flash at 17.3 tok/s decode on an RTX 3090. Setup is ExLlamaV3 with a 3.05bpw EXL3 quant in NVMe mode (55 GB), 128k context, single card. The context gate read 93,437 prompt tokens at 844 tok/s prefill. An NVMe read cap of 8 GB/s cost about 8% of C1 decode and 26% of prefill; uncapped same-session figures were C1 18.23 and C4 19.59 tok/s. Capped standard table: 8k C1 16.61, C2 14.91, C4 13.10; 32k C1 14.43, C2 15.41.

generation:

60.0 tokens/s

quant:

exl3-5.00bpw (EXL3)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at ~60 t/s on a single RTX 3090. Setup is ExLlamaV3 with an exl3-5.00bpw quant at ~150k context; prefill was not benchmarked. The machine has an Intel 12400 CPU and 32GB DDR4 RAM. User asks whether to buy 128GB DDR5 and a Ryzen 9700X to run the IQ3 quant under Strata, and how Qwen3.8 27B compares to Qwen Next on output accuracy for coding and Hermes assistant use.

generation:

162.9 tokens/s

quant:

EXL3-4.00bpw (EXL3)

kv:

cq3

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 162.9 tok/s decode on an RTX 3090 with native DFlash2 speculative decoding in ExLlamaV3. Setup is ExLlamaV3 (community fork, pin 355c6ee) with EXL3 4.00bpw target and DFlash2 draft, cq3 KV cache, at the full 262,144-token context. Acceptance length 5.66; autoregressive decode 42.8 tok/s and MTP head 116.3 tok/s for comparison. At 150k-token depth: 25.3 tok/s decode, 594 tok/s prefill, peak 23.13 GB.

generation:

80-120 tokens/s

quant:

6bpw (EXL3)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 80-120 t/s on 6x RTX 3090. Setup is ExLlamaV3 with a 6bpw EXL3 quant from turboderp. The user describes the throughput as a range and praises the engine for CUDA, using it with the Hermes agent controlled from a phone via Matrix.

generation:

80.0 tokens/s

prompt processing (prefill):

1500.0 tokens/s

quant:

3bpw (EXL3)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 80 t/s generation and 1500 t/s prefill on 3x RTX 3090 with 128GB DDR4. Setup is ExLlamaV3 with an exl3 3bpw quant at 262,144 context, with vision and speculative decoding enabled. A single RTX 5090 with 128GB DDR4 reached 29 t/s generation at the same 1500 t/s prefill and 262k context. User plans to test 4bpw later and says ExLlamaV3 is replacing vLLM and llama.cpp for this model.

generation:

110.0 tokens/s

prompt processing (prefill):

1500.0 tokens/s

quant:

3.05bpw (EXL3)

kv:

FP16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash Next at 110+ t/s generation and over 1,500 t/s prompt processing on 3x RTX 3090s. Setup is ExLlamaV3 with a 3.05 bpw EXL3 quant at full context and FP16 KV cache, with only the engram table offloaded to RAM. All three cards run at 250-265W and one is on a TB4 eGPU. User also ran Qwen3.8 27B (UD-Q8_K_L) on 2x 3090s in TP and a UD-Q4_K_XL of Qwen3.8 Flash Next that partially offloaded to RAM, reaching about 50 t/s max output, 38 t/s average, and prompt processing in the low hundreds.