Local model performance on your hardware

generation:

130.0 tokens/s

quant:

INT8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B Uncensored-HauHauCS at about 130 t/s decode on an RTX 4090. Setup is NInfer with INT8 quantization, 262K context, and a model size of about 17 GB. The user notes this is a port in progress, with a CPU build possibly running overnight; the 130 t/s is an expected figure rather than a measured result.

generation:

149.0 tokens/s

quant:

INT8

kv:

E8 4-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at up to 149 tok/s code decode on one RTX 4090. Setup is NInfer with INT8 prefill and E8 4-bit KV cache, full native 262K context, MTP3 speculative decoding, sm_89-retuned attention prefill, vision, and llama.cpp-compatible /metrics and /slots. User says it is their go-to even with Strata, which is limited to IQ2_XS.

generation:

127-140 tokens/s

kv:

rk4v4-e8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User compares Qwen3.8-27B dense on NInfer against Qwen3.8-Flash-Next IQ2_XS on Strata 0.1.39, both on one RTX 4090 24 GB with 262K context and MTP. The 27B decodes at 127-140 tok/s with rk4v4-e8 KV and 23,134 MiB peak VRAM; the MoE decodes at 173-185 tok/s with q4_0 KV in VRAM and experts in host RAM, peaking at 23,936 MiB. Recall at 55K/117K/200K is 3/3 for both, with the 27B at 3,037/2,391/1,857 prompt tok/s and the MoE at 3,494/3,904/3,833. On three graded tickets the 27B scored 112/150 versus 86/150 for the MoE. The user attributes the difference to cache reuse: 88.5% for the 27B, including one turn serving 172,343 of 172,578 tokens from cache with first token in 524 ms, against 32% for the MoE, whose typical turn re-read 107K tokens in 28 s with conversation cache off on 32 GB RAM. Limits are n=1 per ticket, one grader model, and the MoE ran first.

generation:

107.0 tokens/s

prompt processing (prefill):

5008.0 tokens/s

quant:

Q4/Q5

kv:

rk4v4-e8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 107 tok/s decode with MTP on mixed prompts and 5,008 tok/s prefill at 2K tokens on one RTX 4090 under native Windows. Setup is NInfer with Q4/Q5 weights, rk4v4-e8 KV cache, 262K context per request, MTP with 3 draft tokens and n-gram chaining. Decode reaches 289 tok/s with MTP + n-gram on edit-style prompts and 211 tok/s with DFlash2 on code; prose with thinking on gives 99 tok/s with DFlash2 versus 85 tok/s with MTP + n-gram. Against llama.cpp on the same card, prefill is 5,008 vs 2,729 tok/s at 2K, a 64K prompt takes 17.2 s vs 29.8 s and a 128K prompt 43.0 s vs 73.6 s. Ternary Bonsai 2 27B reaches 188 tok/s decode on mixed prompts, 532 tok/s with MTP + n-gram on edits and 360 tok/s aggregate with three concurrent requests. The 4090 also drives a 4K monitor at 60 Hz, costing about 15% of decode.

generation:

188.0 tokens/s

quant:

Q4/Q5 (GGUF)

kv:

E8 4-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Ternary Bonsai 2 27B at 188 tok/s decode averaged over six mixed prompts on a single RTX 4090 under native Windows. Setup is NInfer, a from-scratch C++/CUDA engine, with Q4/Q5 weights, E8 4-bit KV cache, MTP plus n-gram speculative decoding, and 262K context per request. On edit-style prompts the model reaches 250 tok/s with MTP and 532 tok/s with MTP + n-gram; through the server on 7K-11K-token file edits with thinking on it goes 215 to 378 tok/s. Three concurrent requests give 360 tok/s aggregate. Against Prism's llama.cpp fork on the same card: 101 vs 77 tok/s decode and 3,061 vs 1,363 tok/s pp512. N-gram gains depend on repetition and are zero on free prose (167 tok/s both ways).

generation:

148.6 tokens/s

prompt processing (prefill):

1849.0 tokens/s

quant:

E8 4-bit

kv:

INT8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 148.6 tok/s on a single RTX 4090 with MTP3 speculative decoding at 81.0% draft acceptance. Setup is NInfer with the official groupwise artifact, INT8 KV cache, CUDA Graphs on, prefill-chunk 1024, single request greedy decoding. Without speculation decode is 50.5 tok/s; at 128K context depth it is 39.6 tok/s. Prefill is 1,849 tok/s at 64K and 1,561 tok/s at 128K. The E8 4-bit KV default profile costs about 5.7% decode versus INT8, giving roughly 126.6 tok/s on the code probe.

generation:

80.0 tokens/s

prompt processing (prefill):

1200.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User compares an RTX 4090 and an RTX 5090 for speed factors. User mentions Muse Glimmer as an alternative but has not benchmarked it.