- generation:
- 71.0 tokens/s
- prompt processing (prefill):
- 861.5 tokens/s
- kv:
- INT8
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 71.00 tok/s decode on one RTX 3090 24 GB.
Setup is NInfer v0.6.1 with INT8 KV cache, ReplaySSM and MTP3 speculative decoding, CUDA Graphs, 8,192-token context, 19.6 GB VRAM used.
Decode by cohort: C2 90.66, C4 100.28, C8 165.33 aggregate tok/s. Prefill with 4,362 fresh input tokens per request: C1 861.51 tok/s (TTFT 4,893 ms), C8 844.10 tok/s. MTP3 acceptance 61.13%, C1 TTFT 149 ms.
- generation:
- 40.0 tokens/s
- quant:
- IQ3_S (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8-Flash-Next at ~40 tk/s on an RTX 4080 Super 16GB.
Setup is the strata engine with an IQ3_S quant, run through a custom mini swe agent v2 with powershell, edit_file, view_image, read_file and search_files tools, plus guards for tool failures and auto-compaction.
The one-prompt GTA-style three.js game took 3h 18m 6s and 22,845,556 tokens (22,533,061 input, 312,495 output). The user notes the IQ3 quant fucks up a lot.
- generation:
- 34.7 tokens/s
- prompt processing (prefill):
- 863.0 tokens/s
- quant:
- exl3-5.05bpw (EXL3)
- kv:
- Q8
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 863 t/s prefill and 34.7 t/s decode on a single Radeon AI PRO R9700 32GB at 230k context.
Setup is ExLlamaV3 (phoenixhaxor ROCm fork, commit cbbef08, patched for gfx12) with exl3 5.05 bpw, Q8 KV cache, 262144 context, chunk size 4096, batch 1. 112 of 512 experts per layer run on the GPU and 400 on CPU with 16 threads; the 102GB bf16 n-gram table is streamed from NVMe. MTP drafting is on with around 53-54% acceptance.
The prefill figure covers only the new 101k tokens, the rest came from the prefix cache. User asks whether this is normal for the card or if there is room to tune.
- generation:
- 36.4 tokens/s
- quant:
- Q4_K_M (GGUF)
- kv:
- q8_0
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8-27B at 36.4 tok/s on an RTX 3090 24GB, with throughput dropping to 20.9 tok/s at 120K context.
Setup is llama.cpp with Q4_K_M and q8_0 KV cache, all layers on GPU, 131,072-token context, flash attention, one generation slot.
The 36.4 tok/s figure is for a 2,073-token input with thinking disabled; cached continuations reuse almost the entire prefix and are not representative of fresh prompts. Peak GPU memory was 22,162 MiB. Three of four bounded coding tasks passed their checks, with one failing after spending its full 8,192-token output allowance on reasoning.
- generation:
- 40.8 tokens/s
- quant:
- Q4_0 (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingmathagentic
User reports Qwen3.8-Flash-Next at 40.8 tok/s on an M5 Pro 64GB Mac using the Slipstream engine.
Setup is Slipstream with Q4_0 GGUF weights and SSD expert streaming, running the 95.5 GiB model on a single 64GB machine.
The figure is the average across six reasoning and coding tasks, compared to 23.1 tok/s in a llama.cpp fork (1.76x speedup). Decode stays between 33 and 44 tok/s out to 130,000 tokens across 3,086 live requests. An optional Swift KV-sparse variant scores 70.3% overall on 145 paired evaluation problems versus 67.6% for the original.
- generation:
- 35.0 tokens/s
- quant:
- IQ3_XXS (GGUF)
- kv:
- q4_0
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports about 35 t/s or more with Qwen3.8 27B GSQ-RCO-Uncensored on an RTX 5060 Ti 16GB.
Setup is llama.cpp with IQ3_XXS quant, q4_0 KV cache, 131072 context, and MTP draft plus ngram speculative decoding.
The user is on Fedora 44 with an AMD Ryzen 9600x and 16 GB system RAM, and had struggled to get reasonable speed before finding this model.
- generation:
- 39.4 tokens/s
- quant:
- UD-Q4_K_XL (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 39.4 t/s median on normal prompts on a Strix Halo device.
Setup is Gufo 0.4.0 with UD-Q4_K_XL GGUF and a DFlash2 Q4_K_M draft model, single user, speculative decoding on.
The 70.22 t/s figure comes from a repetitive prompt and is a ceiling; normal prompts range from 22 to 52 t/s. In a head-to-head with halogen 0.13.8 on identical hardware, halogen was about 13% faster for one user and 18% faster with four, while Gufo was 16% faster at prompt processing and much faster on the repetitive prompt.
- generation:
- 50.0 tokens/s
- prompt processing (prefill):
- 1000.0 tokens/s
- quant:
- NVFP4 (NVFP4)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agentictool-use
User reports Qwen3.8 Flash-Next NVFP4 at ~50 t/s generation and ~1000 t/s prefill while running a computer-use agent harness built with Claude Code.
Drawing a simple image took 12m 1s, much of it spent debugging a WebGL issue.
The user asks about open-source computer-use harnesses for regulated fields like law and medicine, HIPAA compliance, and whether others use local or cloud models.
- prompt processing (prefill):
- 2500-3500 tokens/s
- quant:
- NVFP4 (NVFP4)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next 27B at over 200 tok/s generation on 4x V100 16GB GPUs.
Setup is vLLM with NVFP4 Nvidia checkpoints and DFlash speculative decoding, tensor parallel 4, with image support enabled and KV cache around 120k tokens.
Prefill is reported as 2.5-3.5k t/s. The user notes the server cost $5400 on eBay and used a custom repo to unpack NVFP4 into FP16 on the fly.
- generation:
- 12.8-13.2 tokens/s
- prompt processing (prefill):
- 42.7 tokens/s
- quant:
- ROCmFP4 (GGUF)
- flash attention:
- on
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B-ROCmFP4-FAST decoding at 12.8-13.2 t/s on an AMD RX 7900 XTX 24 GB, with later runs at 16.5-18.1 t/s, against an expectation of about 75 t/s.
Setup is llama-server-fpx (ROCmFPx fork, Vulkan backend) with the 13.5 GiB GGUF, -ngl 999, flash attention on, and MTP speculative decoding (--spec-type draft-mtp, --spec-draft-n-max 4).
Prompt processing is 42.7 t/s. MTP acceptance is 70-73%, later 79%. The machine is a Dell 5820 Tower with an Intel W-2125 and 128 GB RAM, 64 GB of it passed to a Docker LXC.
- generation:
- 171.0 tokens/s
- quant:
- Q4_K_M (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8-27B at 171 tok/s on an RTX 5090, solving 14 of 16 hard problems.
Setup is Ollama with Q4_K_M GGUF at 64K context, temperature 0.2 and a 32,768-token generation cap, using 17.5 GB of VRAM.
Nemotron 3.5 Lightning reached 216 tok/s and 14/16 at 25.4 GB, while Muse Glimmer reached 81 tok/s and 15/16 at 16.9 GB.
- generation:
- 68.9 tokens/s
- prompt processing (prefill):
- 380.0 tokens/s
- quant:
- IQ3_S (GGUF)
- kv:
- q8_0/q5_1
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 68.9 t/s on an RX 7900 XTX 24 GB at empty context with MTP n=3 speculative decoding, and 37.2 t/s without MTP.
Setup is llama.cpp (ROCm, hip-kvmix) with IQ3_S GGUF and q8_0/q5_1 KV cache at 262,144-token context.
At 240K fill with MTP the user reports 18.6-26.9 t/s (mean 23.3) and 380 t/s prefill, with 22,630 MiB peak VRAM.
- generation:
- 23.8 tokens/s
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 23.8 tok/s decode on a single Radeon RX 7900 XTX 24 GB.
Setup is a llama.cpp fork with a 131,072-token context and single stream (parallel 1), no speculation.
With draft-mtp plus ngram-map-k speculation the same setup reaches 47.6 tok/s. The card runs a 272 W power cap (factory 303 W), a -75 mV undervolt and max SCLK 2200 MHz.
- generation:
- 50.0 tokens/s
- quant:
- BF16 (safetensors)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at BF16 generating about 50 t/s single user on six RX 7900 XTX GPUs.
Setup is vLLM with tensor parallel 6, achieved by padding the model architecture numbers with zeroes to make them divisible by six, at a 256k context window.
Aggregate throughput is 200 t/s with 8 concurrent prompts. GPU KV cache size is 520,784 tokens, giving a maximum concurrency of 1.99x for 262,144 tokens per request.
- generation:
- 7.1 tokens/s
- prompt processing (prefill):
- 113.5 tokens/s
- quant:
- Q4_K_M (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User compares Qwen3.8 Flash-Next Q4_K_M at 160,000 context across a Strix Halo and a six-GPU machine. On Strix Halo with mainline llama.cpp 0.4.1 the run reaches 113.48 t/s prefill and 7.06 t/s generation.
Setup is llama.cpp with Q4_K_M weights, 103.68 GiB model, fully in the Strix Halo's 128GB unified memory.
At the same context the Gufo fork gives 1227.12 t/s prefill and 22.04 t/s generation, Halo Box gives 587.99 t/s and 21.24 t/s, and the pile of GPUs (2x RTX 5070 Ti, 2x RTX 5060 Ti 16GB, 2x RX 7900 XTX) gives 215.73 t/s and 16.29 t/s. User concludes Strix Halo shines for this model.
- generation:
- 59-74 tokens/s
- quant:
- oQ8e (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User compares Qwen3.8-Flash-Next against Laguna-S-2.1 on a Mac Studio M5 Ultra 256GB, both capped at 262K context with thinking on and unique content per run.
Qwen runs under oMLX with an oQ8e quant and MTP speculation, holding roughly 4,200 tok/s prefill and 59-74 tok/s decode across sizes. Laguna runs under LM Studio with an 8-bit quant, prefill degrading superlinearly from 10.3s at 8K to 455.4s at 200K, and decode falling from 68 to 34 tok/s with no speculation.
Quality was a draw at 4/4 each on four script-verified problems. At 200K, prefill is 94% of total time on both. The user notes an earlier run was invalidated by shared prefixes letting the KV cache carry over.
- generation:
- 1.0 tokens/s
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B running on an AMD NPU via FastFlowLM at 1 t/s decode.
No engine version, quant, context length or memory details are given.