Local model performance on your hardware

generation:

26.2 tokens/s

quant:

NVFP4 (NVFP4)

kv:

turbo4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 26.2 tok/s single-stream on an RTX 5090 32 GB with 254 GiB system RAM. Setup is FreeToken 0.1.3 with NVFP4 routed experts streamed from system RAM (about 120 GB pinned for offloaded experts) and turbo4 KV cache; MTP is off because verification cost more than direct generation. Aggregate throughput reaches 47.5 tok/s at concurrency 2, 66.9 at 4, 64.6 at 6 and 68.8 at 8, topping out around 65-77 tok/s, limited by PCIe bandwidth and per-step expert fetch volume.

generation:

50.5 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next Uncensored NVFP4 at a median 50.5 t/s generation on one RTX 5090, measured at the maximum input of 259,601 tokens. Setup is a modified FreeToken engine following the llama-split-bench protocol, with 96 GB system RAM and 1,000 tokens generated per test. Across 3 rounds and 36 completed measurements, the median over 9 short practical prompts was 55.4 t/s.

generation:

50.0 tokens/s

prompt processing (prefill):

2300.0 tokens/s

quant:

NVFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 50 t/s generation and 2300 t/s prompt processing on a single RTX 5090 with 128 GB DDR5. Setup is FreeToken with the nvidia/Qwen3.8-Flash-Next-NVFP4 model converted to FreeToken format, expert caching enabled, max sequence length 229376, and max running requests 2. Generation speed ranges from 40 to 60 t/s depending on cache effectiveness. User notes llama.cpp lacks MoE expert caching, and FreeToken is early with no KV quantization or MTP yet.

generation:

73.6 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 73.6 t/s on an RTX 5090 with 32 GB VRAM and 192 GB system RAM. Setup uses FreeToken with a 123 GB model load; the user notes running out of context after a few prompts. The user compares the speed favorably to a 27B model and calls the experience impressive.