1240 reports. These are individual setups, not a controlled benchmark. Qwen3.8 speeds by hardware →
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen 3.8 Flash running at 13-14 tok/s on a MacBook M5 Pro with 48GB. The model takes 21-24 GB of memory. The user asks others to share their own Mac specs and preferred local models.
NVIDIA RTX 3090 · 180,224 ctx
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports running Qwen3.8-27B on a dedicated headless RTX 3090 for coding work, with 1000-1500 t/s prefill and 45-60 t/s generation at 180224 context with Q8_0 KV cache. Setup uses one RTX 3090; the user says it replaced cloud subscriptions for most real coding tasks. The user also ran Ornith-1.5-9B on a home server with 8GB VRAM and found it successful on simple Python refactoring tasks, but has not benchmarked it and asks for others' real coding experience.
2× NVIDIA RTX 4070 · Unsloth Desktop · 120,000 ctx
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User asks for advice on orchestrating a local Qwen 3.8 27B GGUF model with cloud subscriptions (Claude, Codex, Antigravity) to avoid burning paid tokens. Setup is Unsloth Desktop on a remote rig with dual GPUs (RTX 4070 + RTX 5060 Ti), tunneled through Cloudflare, with a 120k context window. User reports output of around 20–40 tokens/s.
AMD RX 6900 XT 16GB · llama.cpp · 100,000 ctx
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports running Qwen3.8-27b (Huihui abliterated, IQ3_XXS) on an RX 6900 XT 16GB with llama.cpp and ROCm 10. Prefill drops from 400 t/s to 260 t/s at 40k context; decode varies between 30 t/s and 45 t/s. Setup uses 100k context, KV cache q8_0/q4_0, flash attention, ngram-mod, MTP with n=2, and mmproj in system RAM; VRAM fills to 15.7/16 GB.
M5 Pro 48GB · llama.cpp · 4,096 ctx
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at roughly 18-22 tok/s generation on an M5 Pro MacBook Pro with 48GB unified memory. Setup is llama.cpp b10930 with Unsloth UD-IQ1_M GGUF, 4096 context, flash attention on, 12 experts on CPU, and --load-mode none --lazy-mode on to avoid Metal OOM. At 16384 context for an OpenCode test, a 10,634-token prompt took 503 seconds of prefill then generated 192 tokens in 16 seconds; a follow-up reused 10,825 cached tokens but 5,113 more tokens took 165 seconds, and a 28,081-token request was rejected. About 94% of inference time went to prefill, and the user calls it not practical for agentic coding on this setup.
NVIDIA RTX 5070 Ti Laptop 12GB · llama.cpp · 98,304 ctx
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen 3.8 27B at ~9-11 t/s decode and 400-550 t/s prefill on an RTX 5070 Ti Laptop 12GB. Setup is llama.cpp with UD_Q4_K_XL quant, q8_0 KV cache, 98304 context, draft-mtp speculative decoding, and FFN weights offloaded to CPU. The user notes reasoning verbosity blows past 100K tokens in planning, and with Magic Context the session scaled past 3.7M total processed tokens.
2× NVIDIA RTX 5070 Ti · 256,000 ctx
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User asks whether Qwen3.8 Flash-Next can run on a mixed RTX 5070 Ti 16GB plus Radeon AI PRO R9700 32GB box with 96GB DDR5 system RAM. They report Qwen3.6 27B already runs on both cards with layer split at 40 tok/s using Q8 weights and Q8 KV cache at 256k context. They also mention losing two days to an AMD Pro driver unload bug that freezes the system, and ask for quant, tok/s, context size, launch flags, and mixed NVIDIA/AMD advice.
4× NVIDIA V100 32GB · llama.cpp · 16,384 ctx
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 42.2 t/s on 4x Tesla V100-SXM2-32GB (128 GiB VRAM total). Setup is a llama.cpp fork with tensor parallelism, CUDA_VISIBLE_DEVICES=0,1,2,3, GGML_CUDA_ALLREDUCE=nccl and GGML_CUDA_P2P=1, at 16k context. The 156 GiB model runs on 128 GiB of VRAM. At full 256k context with --ctx-size 262144 -ncmoe 12 decode is about 21.8 t/s. An expert-cache pool sweep on qwen35moe (256 experts, 8 used, 40 layers, -ncmoe 99) gives 16.1 t/s off, 20.4 t/s at N=8, 30.3 t/s at N=32, 39.2 t/s at N=64 and 46.7 t/s at N=128, with hit rates of 47%, 62%, 71%, 77%, 85% and 89%. Gains are sublinear in bytes moved: a run that copies almost nothing still tops out near 76 t/s against 102 t/s with no offload. All decode figures are single-stream.
M1 Max 64GB · Inco Splash
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 32.8 t/s average on an M1 Max 64GB MacBook Pro, using the Splash-M1 engine with a 4-bit MLX quant. Setup is the prebuilt splash-m1 1.0.2-m1 with incoai Splash packages, compared against oMLX 0.6.4. Prefill on an 8K prompt reached 101 t/s. Four parallel requests totalled 62.3 t/s. Splash was about 2x faster than the 4-bit MLX quant on code, math and parallel requests, but slower on prose (18.3 vs 16.1 t/s) and 32K prompts. Draft acceptance was about 94% on code but only 19% on prose. Time to first token at 32K was about 400 s. Accuracy matched the 4-bit quant on the user's test suites.
Unknown GPU
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 13-15 t/s on their main machine. They also mention a Mac Mini 16GB running Ornith 9B or Gemma 4 12B, and a Raspberry Pi 5 running a 3B model. The post asks how to build a router that distributes tasks across these machines, citing Claude Code as inspiration.
AMD Strix Halo 128GB · llama.cpp · 32,768 ctx
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen 3.8 27B at 14.02 tok/s unassisted decode on an AMD Strix Halo (Ryzen AI Max+ 395, 128 GB). Setup is a ROCmFPX-enabled llama.cpp fork with ROCmFP4_FAST quant (4.26 bpw, 13.55 GiB GGUF) and Asymmetric TurboQuant KV cache at 32K context. With MTP speculative decoding, decode reaches 34.82 tok/s (2.84x) at greedy sampling; deep spec n7/p0.35 reaches 36.04 tok/s on JSON and code. At temperature 0.8 draft acceptance collapses to about 25% and speculative decoding returns close to the unassisted 14.02 tok/s.
3× NVIDIA RTX 3060 12GB · llama.cpp
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 25 t/s on three RTX 3060 12GB cards. Setup is llama.cpp in layered mode with a Q6 quant. The figure is a current baseline; the user asks whether moving to four or eight RTX 3060s under vLLM would improve or hurt token generation.
AMD MI50 32GB · MI50-llama.cpp · 66,560 ctx
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 22.8 tok/s decode on a single AMD Instinct MI50 32GB, using the MI50-llama.cpp fork. Setup is llama.cpp with Q8_0 weights and Q8_0 KV cache, flash attention on, one slot, 66560 context. Decode holds 22.0 tok/s at 16k and 20.4 tok/s at 64k; MTP with 3 drafts reaches 28.8 tok/s on a 64k reasoning prompt, 51.8 tok/s on short code, 43.2 tok/s on prose, and 24.2 tok/s on a 106.7k prompt. Max context is 114,688 with -ub 512. All figures are single-stream.
NVIDIA RTX 5090 · NInfer · 262,144 ctx
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Swift-1.5 Qwen3.8-27B at 160.8 t/s decode on a single RTX 5090 at 262,144 context. Setup is the NInfer v3 engine with an all-NVFP4 (W4A4 gs16) artifact plus a z-lab DFlash2 drafter at K=7, k8v4 KV cache, 18.0 GiB of weights in VRAM, 450 W power cap, and concurrency 4. Prefill at 200k context measured 3,269 t/s. IFBench prompt-strict 69.0, prompt-loose 72.7, instr-strict 70.4, instr-loose 73.6; GSM8K-200 95.0% (190/200); long-context needle at 250,031 tokens exact across 3 depths.
User compares Qwen3.8 Flash-Next IQ4_XS against Qwen3.8 27B FP8 on a system with 64 GB DDR5 and 2x Radeon AI PRO R9700, serving both via vLLM. Both models were run through a reduced quick eval with the same questions: MMLU-Pro 83.8% vs 75.0%, GPQA Diamond 42.5% vs 27.5%, GSM8K 97.5% vs 90.0%, German MGSM 92.5% vs 90.0%, and IFEval 89.6% for both. The 250-case fast trial took 36m 27s on 27B FP8 and 2h 05m 47s on Flash-Next IQ4_XS. The user notes MMLU used only 80 random questions and the evals are not full, and that FP8 27B is around 4x faster than Flash-Next under concurrency.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at approximately 20 tok/s single-stream decode over 256 output tokens on an RTX 5090 with 128 GB system RAM. Setup is llama.cpp with GGUF UD-Q4_K_XL, with the model partly offloaded to system RAM. With 4 parallel requests of roughly 4K prompt plus 256 output, aggregate end-to-end throughput is 13.8 tok/s; with roughly 8K prompt plus 256 output it is 8.81 tok/s. The user notes these aggregate figures are a different measurement from the single-request rate and should not be read as single-stream speed.
2× AMD Radeon AI PRO R9700 32GB · vLLM · 16,384 ctx
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 184.9 t/s generation and 3,948 t/s prompt processing on 2x Radeon AI PRO R9700 32GB. Setup is vLLM with MXFP4 weights at 16,384 tokens of context; a 256K configuration is mentioned separately. The same model on the same machine reached 19.2 t/s generation and 590 t/s prompt processing with default llama.cpp Q8_0, and 29.7 t/s generation and 1,458 t/s prompt processing with llama.cpp Q8_0 using gfx1201 and tensor split.
2× NVIDIA RTX 5090 · ExLlamaV3 · 262,144 ctx
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 291.1 t/s single-stream decode on code and 273.5 t/s on prose, on two RTX 5090 32 GB GPUs with layer split [30, 30]. Setup is ExLlamaV3 v1.5.0 under TabbyAPI with a 2.50 bpw EXL3 pack, 8-bit KV cache, 262,144-token context and a 983,040-token page pool, using MTP draft depth 3 up to 4 jobs and 2 at 5 to 8 jobs. Per-stream decode falls to 153.2 t/s on code and 156.8 t/s on prose at 4 streams, and 103.4 t/s on code and 103.6 t/s on prose at 8 streams. Aggregate decode across concurrent streams is 292 t/s code and 272 t/s prose at 1 stream, 615 t/s code and 627 t/s prose at 4 streams, and 822 t/s code and 831 t/s prose at 8 streams. Time to first token ranges from 0.13 to 0.14 s at 1 stream to 0.76 s at 8 streams. Cold prefill holds its rate to the top of the window: 199,844 tokens in 18.8 s and 240,047 tokens in 22.8 s. Quality runs on the same configuration give GSM8K 5-shot (n=500) 0.978, SWE-bench Verified 397/500 resolved (79.4%), and tool-eval-bench 84.0 +/- 2.4.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 40 t/s generation and 300 t/s prompt processing on an RTX 3090 in a 4-GPU system totalling 72 GB VRAM. Setup is llama.cpp with an IQ4_XS GGUF. A second configuration with ExLlamaV3 at 3.05 bpw achieves 25 t/s generation and 1,200 t/s prompt processing. The user is considering an Epyc 7443 upgrade with 256 GB DDR4 for more PCIe lanes and CPU offloading, and asks whether the extra RAM and bandwidth would be worthwhile.
NVIDIA RTX 5090 · llama.cpp · 64,768 ctx
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 40 t/s decode on an RTX 5090 with 64GB system RAM. Setup is llama.cpp with the Atomic Q4_K_M GGUF at 64,768 context, using mmap, lazy mode, flash attention, all GPU layers, and 32 MoE layers offloaded to CPU. The run used 27GB of VRAM and 8GB of system RAM. User notes the prompt processing figure of 50 t/s is likely inaccurate due to the short prompt, and questions why the model is not buffering more into system RAM instead of reading from the NVMe drive.