Local model performance on your hardware

generation:

180.0 tokens/s

prompt processing (prefill):

9000-10000 tokens/s

quant:

AWQ 4-bit (AWQ)

User reports Qwen3.6-35B-A3B at about 180 tok/s single-stream decode on two RTX 4090s at roughly 50k context, dropping to about 160 tok/s at roughly 110k. Setup is sglang with a QuantTrio AWQ 4-bit quant, TP=2 over PCIe with no NVLink. Prefill is 9-10k tok/s, and 8 concurrent requests reach 985 tok/s total. The user runs it as a GitLab code-review agent and says repetition is the main problem with local models, with the model looping on itself ending about 7% of runs (56 of 847).

generation:

67.0 tokens/s

prompt processing (prefill):

2600-2820 tokens/s

quant:

escha 2-bit (safetensors)

User reports Qwen3.8-27B-Escha-W2 at 67.0 tok/s single-stream decode on an RTX 4090. Setup is SGLang with a 2-bit escha quant (2.469 bits/weight, 10.15 GB of weights) at 64k context, CUDA graphs on, prefix caching off. With the model's own MTP head as a speculative draft the same card reaches 129.3 tok/s (1.92x). Peak server throughput is 649 tok/s at 16 streams; prefill is ~2,600-2,820 tok/s and TTFT at a 2k prompt is 0.73 s. The card also ran 87.1 tok/s on an RTX 5090 and 40.7 tok/s on an RTX 3090 with ESCHA_ROUTE=blackwell.

generation:

135.4 tokens/s

quant:

EXL3 3bpw (EXL3)

User reports Qwen3.8-27B at 135.4 tok/s decode on a bare RTX 4090 24 GB. Setup is SGLang with EXL3 3bpw weights, 207,356-token KV pool, 22.9 GB peak, 239 ms TTFT over 3 samples. A separate depth sweep on the same card gives 184 tok/s code and 126 tok/s prose at ~50 prompt tokens, falling to 115 and 91 tok/s at 133k-190k tokens; cold prefill was 2,112 tok/s at 19k and 1,636 at 134k. The recipe is the unchanged RTX 3090 SGLang EXL3 config, and it now displaces a TabbyAPI SC4bpw recipe that measured 86 tok/s. No coherence ladder, soak test or 3 bpw vs 4 bpw quality comparison was done.