Local model performance on your hardware

generation:

172.0 tokens/s

prompt processing (prefill):

5905.0 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 3.8 Flash Next at 172.0 tok/s decode without speculative decoding on an RTX6000 96GB, at 512 context with non-expert weights downsampled from 16-bit to 8-bit. Setup is NInfer6000, the user's fork of NInfer, with NVFP4 quants and prefill at 5,905 tok/s on 512 tokens and 13,908 tok/s on 8,192 tokens. With MTP3 and --lm-head-draft the same setup reaches 274.8 tok/s at 512 and 401.3 tok/s at 8K. Keeping non-experts at 16-bit gives 118.0 tok/s without speculative decoding.

generation:

600.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6 35B-A3B at 600 t/s single request on an RTX Pro 6000 Blackwell. Setup uses the Ninfer engine. User describes it as a drudgework model for read-and-find or code tasks, noting it may use 20x more tokens but is still faster than many local models, and compares it favorably to Cerebras.

generation:

104.8 tokens/s

prompt processing (prefill):

12400.0 tokens/s

quant:

groupwise-int

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B on an RTX PRO 6000 Blackwell at 104.83 tok/s weighted decode, running a long-horizon agentic task at 262K context. Setup is NInfer with a groupwise-int quant (mixed Q4/Q5/Q6). The run covered 966 model calls, 131.2M input tokens and 853.3K output tokens. Prompt processing was about 12.4K tok/s, measured client-side and including queue time. The user plans to try an NVFP4 profile and estimates API cost savings of $650+.