Local model performance on your hardware

prompt processing (prefill):

3843.0 tokens/s

quant:

W8A8 (safetensors)

kv:

bf16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B W8A8 on 4x RTX 3090 capped at 220 W with 3,843 t/s prefill at 512 prompt tokens (single request); decode is given only per task, 146.6 t/s narrative and 265.5 t/s code, with no overall figure. Setup is SGLang v0.5.19 on 4x RTX 3090 with TP=4, W8A8 INT8 weights, bf16 KV cache, DFlash2 speculative decoding with 8 draft tokens, HiCache ratio 1.5 and a 262144 context window. The c=1 prefill sweep ran 3,843 t/s at 512 tokens up to a 5,058 t/s peak at 4,096, and about 4,000-4,300 t/s at 10K and 3,029 t/s at 90K. All four GPUs were capped to 220 W of the 350 W default (62%), so every figure is power-limited. A concurrency sweep reached 1,811 t/s aggregate code decode at c=16 (113.2 t/s per request).

prompt processing (prefill):

904.0 tokens/s

quant:

3.00bpw (EXL3)

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B EXL3 3.00bpw on a bare RTX 3090 24 GB through SGLang with the sglang-exl3 plugin, with cold prefill at 904 tok/s. Setup uses fp8 KV cache, MTP speculative decoding on, thinking off, one request, at the full 262,144-token window; the 2.4 GB token embedding sits in pinned host memory. Decode is given per workload: 98.8 t/s on prose, 143.3 on code and 89.6 on prose with a 32k prompt. A Qwen3.6-35B-A3B EXL3 recipe on the same card reached 251.9 prose and 352.1 code. A prior AWQ-INT4 27B recipe with no draft is cited at 45.1 t/s prose.

generation:

96.2 tokens/s

quant:

EXL3 3.0 bpw (EXL3)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 96.2 tok/s on prose and 141.1 tok/s on code on an RTX 3090. Setup is SGLang with EXL3 3.0 bpw weights and an MTP draft model, single request. The figures come from the trellis-serve project's registry recipes; the same table also lists Qwen3.6-35B-A3B at 251.7 tok/s prose and 357.9 tok/s code on the RTX 3090, and Qwen3.8-27B at 91.2 tok/s prose and 66.1 tok/s code on an Intel Arc Pro B70.

generation:

136.8 tokens/s

prompt processing (prefill):

1581.0 tokens/s

quant:

AWQ-INT4 (AWQ)

kv:

bf16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 136.8 tok/s single-stream decode (8k context, concurrency 1) on 4x RTX 3090. Setup is SGLang with AWQ-INT4 (W4A16 Marlin) weights and bf16 KV cache, tensor parallel 4, DSpark speculative decoding enabled with a 1.36B draft model, 275,018-token KV pool. Prefill sustains 1,581 tok/s at 8k c1 and 1.3-1.6k tok/s across all context lengths; aggregate decode reaches 169.8 tok/s at 8 concurrent requests. Max context 128k, max 9 concurrent requests (GDN state bound). Also benchmarks 2x 3090 (112.9 tok/s decode at 8k c1) and 1x 3090 without DSpark (43.2 tok/s decode, 8k context limit).

generation:

98.2 tokens/s

prompt processing (prefill):

914.0 tokens/s

quant:

3.00bpw (EXL3)

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 98.2 t/s on prose and 225.1 t/s on code on one RTX 3090. Setup is SGLang with the sglang-exl3 plugin, EXL3 3.00bpw weights, fp8 KV cache, a 131,072-token window, and DFlash2 speculative decoding with a 5.0bpw draft. Prompt processing on a new uncached 32k prompt runs at about 914 tok/s with 35.9 s to first token. Compared with the MTP recipe on the same card, code is +57% (225 vs 143) and prose is the same (98 vs 98), with a 131k window instead of 262k. Max concurrency is 1 because two 32k streams stall each other's decode.