Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User releases ATX-Swift-1.5-Qwen3.8-27B-Uncensored-MTP, a GGUF quantization suite for Qwen3.8 27B, and benchmarks it on a single RTX 3090 24GB.
Setup is llamAmpere with i1-Q5_K_M weights, turbo5/turbo4 KV cache, 131,072 context, and MTP speculative decoding with the draft head at Q8_0. The full model and KV cache fit in 23.4 GB of VRAM with -ngl 99.
At 80k-98k active tokens the custom quant sustains 50.05 to 53.72 t/s with burst peaks of 58.10 to 63.05 t/s, versus 44.94 to 45.79 t/s for the standard Q5 quant. Draft acceptance rises from 63.7-65.1% to 73.1-81.7%, with mean draft length 3.31 to 4.10 tokens per round.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-27B at 40-60 t/s on an M5 Pro Mac mini with 64GB RAM using the Tensorfold engine.
Setup is Tensorfold with the Vontra/Qwen3.8-27B-MLX-4bit model and z-lab/Qwen3.8-27B-DFlash2 draft model for speculative decoding.
The user compares this to an RTX 3090 Ti running Qwen3.8-27B-GSQ-RCO-GGUF with MTP IQ3_S at 50-70 tok/s, noting the 3090 Ti has roughly 4x faster prefill. They are considering replacing the 3090 Ti with the Mac mini as their main inference server.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 127 tok/s single-stream decode on one RTX 3090 at 250 W.
Setup is vLLM 0.30.0 with 38 custom patches, int8 tensor-core GEMMs, speculative decoding with MTP or DFlash2 drafting at 7 drafts per pass, and KVarN 4/2-bit KV cache at 150k-262k context.
Decode reaches 381 tok/s at 25k context when the answer quotes the prompt. At 64 concurrent requests the aggregate is about 1,035 tok/s decode and 948 end to end. Prefill is about 1,440 tok/s, or 1,850-1,940 tok/s depending on int8 activation settings. The card is shared with training jobs.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at about 34 tok/s real-world on a single DGX Spark (ASUS Ascent GX10, GB10, 128GB unified memory).
Setup is SGLang with an NVFP4 W4A4 checkpoint and DSpark block-speculative decoding, batch-1 single-stream.
User also measured 38.0 tok/s average on eval-style workloads and 46.7 tok/s peak on GSM8K-style prompts. On the same machine, llama.cpp UD-Q4_K_XL with MTP reached about 27 tok/s real-world and 24-30 tok/s on eval-style workloads, while vLLM 0.27 NVFP4 with MTP reached about 24.5 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B NVFP4 at 152 tok/s decode on one RTX 5090 under WSL2, with MTP speculative decoding and host embedding enabled.
Setup is SGLang v0.5.20 with NVFP4 weights, 147,456-token context, on a Ryzen 9 9950X3D with 96 GB RAM.
Plain 27B without speculation reached 93 tok/s at one stream (89.7-96.0) and 456 tok/s total at 5 streams; with MTP, 5 streams totalled 753 tok/s. Prefill of a 6K-token prompt was 13.4-15.0K tok/s plain and 9.2-12.4K tok/s with MTP. Energy was 2.91 J/token plain and 1.72 J/token with MTP.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agentictool-uselong-context
User reports Qwen3.8-27B at 113.7 to 114.8 t/s decode and ~1,785 t/s prefill on 2x RTX 3090.
Setup is vLLM v0.29.0 with INT8 W8A16 compressed-tensors weights, fp8_e4m3 KV cache, 262,144 context, tensor parallel 2, DFlash2 W4A16 drafter with 7 speculative tokens, and a 64 GiB CPU offload tier.
User notes NVLink gives +12% decode and +18% prefill over PCIe P2P off, aggregate decode is 302.5 t/s at 4 concurrent, and a W8A8 build prefilled at 2,377 t/s at 46K.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 26.47 t/s on a GMKtec NucBox EVO-X2 with AMD Ryzen AI Max+ 395 (Strix Halo) and 128 GB LPDDR5X-8000.
Setup is LM Studio's llama.cpp Vulkan build with Q4_K_M (lmstudio-community, 16.80 GB) and the model's built-in MTP speculative-decoding head on, thinking off.
Other quants measured with MTP: UD-Q4_K_XL (unsloth, 17.92 GB) 26.26 t/s, UD-Q8_K_XL (unsloth, 31.45 GB) 15.20 t/s, BF16 (unsloth, 54.66 GB) 10.50 t/s. Memory bandwidth measured at about 239 GB/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 7.24 tok/s on a Strix Halo 128GB (Ryzen AI Max+ 395, Radeon 8060S).
Setup is llama.cpp build 9867 with Vulkan backend, UD-Q8_K_XL GGUF, -ngl 999 -fa on, single request, no MTP.
With MTP self-speculative decoding the same configuration reaches 16.46 tok/s on code. Q6 baseline without MTP runs 8.43-8.44 tok/s and Q6 with MTP 18.67-18.72 tok/s. At four concurrent slots, Q8 aggregate is 21.75 tok/s without MTP and 16.94 with.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User compares Qwen3.8 27B Q8 with q8_0 KV cache at 262k context on two RTX 5090s across two systems. On a 9950X with 64GB DDR5 6000, infill was 2500-3000 t/s and output 100-150 t/s. After moving the cards to a Proxmox VM on a 7402P with 256GB DDR4 3200 ECC, infill rose to 3800 t/s but output fell to around 60 t/s.
Setup is Unsloth Studio with Q8 weights and q8_0 KV cache, two 5090s passed through to the VM.
User asks what could cause the output drop and mentions memory bandwidth tests showing 90-120 GB/s on the 4-CCD CPU versus a theoretical 204 GB/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B Q4 at 50-70 tok/s decode on a borrowed RTX 3090, with the RTX 5080 running ComfyUI at the same time.
The model was loaded entirely on the 3090. The user is deciding between a used RTX 3090, RTX 5090, Mac Studio M5 Ultra 96GB or 256GB, and DGX Spark for a two-user home setup with 1-4 concurrent requests.
Other figures in the post are gathered from benchmarks elsewhere, not measured by the user: RTX 3090 ~40 t/s (~66 with MTP), RTX 5090 ~60-75 t/s, M5 Ultra ~46-48 t/s, DGX Spark ~11-12 t/s (~24 with MTP). The user notes Flash-Next is 125B-A6B plus a ~48GB n-gram table that can live on SSD.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 51 t/s generation and 1500 t/s prompt processing on an RTX 5070 Ti Laptop 12GB with 64GB RAM.
Setup is the Strata engine with an IQ3_XXS GGUF quant and 8-bit KV cache, using 131k context. The model was partially offloaded, using 11GB VRAM and 56GB system RAM.
The generation speed was measured at 43k context depth. The user compares this to stock llama.cpp, which reached 23 t/s generation and 100 t/s prompt processing with the same quant.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next Uncensored NVFP4 at a median 50.5 t/s generation on one RTX 5090, measured at the maximum input of 259,601 tokens.
Setup is a modified FreeToken engine following the llama-split-bench protocol, with 96 GB system RAM and 1,000 tokens generated per test.
Across 3 rounds and 36 completed measurements, the median over 9 short practical prompts was 55.4 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 53.3 tok/s on a Mac Studio M4 Max 128GB.
Setup is oMLX 0.6.3rc2 with oQ4e 4-bit affine weights, 262,144 context, ANE prefill and native MTP speculative decoding with k=3 draft tokens, single stream.
Code generation measured 72.1 tok/s and prefill at 4K 273.7 tok/s. oMLX 0.6.1 with MTP gave 48.0 tok/s prose and 65.5 tok/s code; oMLX 0.6.3rc2 with ANE off gave 47.9 tok/s for both; vllm-metal 0.3.0 at 8-bit without MTP gave 13.2 tok/s code.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 48.02 t/s short decode and 1429 t/s prefill on an RTX 5090 32GB with 128GB DDR5 system RAM.
Setup is llama.cpp PR #27742 with UD-Q2_K_XL (78.9GB in 3 shards), 64k context, -ngl 999 and -ncmoe 24 keeping the first 24 layers' experts on the CPU, no speculative decoding, single stream.
Decode after about 16k tokens of input is 43.28 t/s, time to first token is 11.0 s at about 16k input, and VRAM use is about 30.9GB.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 1900 t/s prefill and 90 t/s generation on an M5 Max 128GB.
Setup is mlx-serve with Sushi-4 quant, an MLX-EXL3 hybrid, with MTP and Vision supported.
User also cites M1 Max 64GB at ~350 t/s prefill and 30 t/s gen, and M5 Pro at ~900 t/s prefill and 60 t/s gen.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Victoria, a fine-tune of Qwen3.8-Flash-Next with 44% of experts cut via REAP, at 280 tok/s single stream on one B300 with the draft head, versus 135 without it.
Setup is NVFP4 weights retrained at 4-bit, 48.0 GiB of weights including the draft head, with a separate 95.4 GiB n-gram table not counted in that number.
Terminal-Bench 2.1 scored 70.0% averaged over 3 runs with an 8h per-task timeout, versus 62.5% for the previous NVFP4 build; HumanEval 159/164. The GGUF Q4_K_M build is 49.17 GiB and scored 75.3% on Terminal-Bench in a single noisy run and 93.2% on HumanEval averaged over 5 runs. Uses 35% fewer output tokens than the previous build. A second fine-tune, Maple, is a Canada-first model; its figures are not reported here.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 50-70 t/s on an M5 Max 128GB after upgrading to macOS 27, down from 60-90 t/s on macOS 26.
Setup is oMLX with oQ5e quantization.
The user reports roughly 20% slower token generation after the OS upgrade and asks whether others see the same.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 74 tok/s on a Mac.
Setup uses the Inco Splash engine.
The post title gives no Mac chip or memory size, so the hardware is not identified.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agentic
User reports Qwen3.8-Flash-Next at 108 t/s generation and 2,887 t/s prompt processing on an M5 Ultra 256GB Mac Studio.
Setup is oMLX 0.7.0.dev2 with MLX backend, oQ4e 4-bit quant, multi-token prediction depth 3, 16K prompt.
M3 Ultra 512GB comparison: 70 t/s generation, 1,143 t/s prompt. At 256K context, M5 prompt processing was 2,544 t/s. Qwen3.8-27B on M5 Ultra: 48 t/s generation, 1,701 t/s prompt; RTX 5090 PC in LM Studio: 59 t/s generation, 3,031 t/s prompt.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User reports Qwen3.8 27B at 65.07 t/s decode at 1K context on a Tesla V100-SXM2-32GB.
Setup is llama.cpp pinned at tag b10793 with the repo's patches, Q4_K_M GGUF, 1 parallel slot, multi-token prediction speculative decoding enabled.
Decode falls to 42.22 t/s at 128K context; stock llama.cpp gives 35.95 t/s at 1K and 16.49 t/s at 128K, and the patched build without MTP gives 35.22 t/s at 1K and 23.83 t/s at 128K. Multi-turn agentic sessions with prompt caching take 3.08 s per turn versus 22.94 s without.