Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingcreative-writing
User reports Qwen3.8 27B at 50-60 t/s generation and 350 t/s prefill on 2x Tesla P100 16GB with a custom llama.cpp fork.
Setup is llama.cpp with Q6_K quant, 262144 context, batch 32768, ubatch 1024, and MTP speculative decoding.
At 260k context, generation is 30-35 t/s and prefill is 110 t/s. GPUs are capped at 175W/250W each and run at 79C with minor thermal throttling; user estimates 5-10% higher numbers with better cooling.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Swift-1.5-Qwen3.8-27b at 67 t/s on an RTX 3090.
Setup is the IQ4_XS quant, compared against Unsloth's Q4_K_S.
User says the model solved a custom Directory Opus script in 6 minutes that Gemini Flash failed to fix in 40 minutes, and was impressed that a low-thinking IQ4_XS 27B beat a major cloud model.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8 27B at about 30 t/s on a 16GB VRAM machine.
Setup is the Unsloth IQ4_XS quant with Q8 KV cache at 65k context, no MTP, vision offloaded to CPU.
User calls the speed acceptable but slow for agentic work and asks which 16GB quant gives the best speed, context and quality, questioning whether Q3 quants are safe for general knowledge.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agentictool-uselong-contextvisioncoding
User reports Qwen3.8-Flash-Next (125B MoE, ~6B activated) served on four RTX 3090s at full 262,144-token context, with an 806,792-token FP8 KV pool.
Setup is vLLM 0.30.0 (v2.2.0 release, 58 commits over the public tag) with a W4A16 AutoRound checkpoint, TP2 x PP2 plus expert parallel, MTP K=3, and FP8 KV cache with a scale sidecar.
Cold prefill runs 5,123-5,410 t/s from 10K to 200K tokens; decode with thinking on is 153.5-186.7 t/s (median of three prompts per depth, one boot). The release is a reliability update at parity with v2, not faster: decode lands at 0.973-1.040 of v2 in every cell. On a rented 4x RTX 3090 without peer-to-peer, a cold 54,713-token prompt took 13.8 s (about 4,000 t/s).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingmath
User reports Qwen3.8-Flash-Next (125B) at 12-15 decode tokens per second on a 2021 M1 Max with 32 GB of unified memory.
Setup is MoEspresso 3 with IQ2_K routed projections and KVarN K4/V4 KV cache, using Metal execution through the mlx-iqk library. Cache-Prior expert bias is enabled by default; prefill is unbiased. The user notes the speed holds only when no other memory-hungry applications are running.
The user also reports an overall score of 84.3% on a reproducible 48-question benchmark, ahead of Claude Opus 4.8 xhigh (80.3%) and other hosted solutions.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentictool-uselong-contextvision
User reports Qwen3.8-Flash-Next at 193.3 tok/s median on 4x RTX 3090 (96 GB VRAM, 31 GB host RAM).
Setup is vLLM with Intel AutoRound W4A16 experts (Marlin), BF16 attention, FP8 n-gram table, fp8_e5m2 KV cache, 262,144-token context, 6 seats, and albucino's INT4 MTP draft with 3 tokens and expert parallel.
Without the draft the same lane measured 55.8 tok/s; real prompts ran 109.5 tok/s prose, 108.4 chat, 145.8 code. Six parallel coding agents sustained 317 tok/s aggregate (570 peak burst, 53 tok/s per agent, TTFT 0.47 s); at gmu 0.95 six agents gave 252.6 tok/s aggregate and 48 tok/s per stream. A llama.cpp lane on the same box measured 96 to 102 tok/s on copy/edit tasks and 40 to 49 tok/s freeform, with ~311 tok/s prefill and two 262,144-token slots.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 127 tok/s single-stream on a single RTX 3090 at 250 W.
Setup is vLLM with a 150k token context and an OpenAI-compatible API, in single-user mode.
Aggregate decode is ~1,035 tok/s at 64 concurrent in batch mode, and 381 tok/s when the answer quotes its own prompt.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
visioncodingagenticlong-context
User reports Qwen3.8-Flash-Next running on a Jetson AGX Thor 128GB with vision support, decoding at 15.3-18.5 tok/s on free-form text.
Setup is llama.cpp (qwen4exp branch, commit d4a943f plus cherry-pick 24ea62d and canreuse-v2.patch) with UD-Q4_K_XL GGUF, 65536 context, ngram-mod speculative decoding, and the 51B n-gram table offloaded to CPU/NVMe via -ot per_layer_token_embd=CPU -lm mmap, leaving about 80GB resident.
Prefill is 90-170 tok/s depending on caching; ngram-mod speculation reached 82.9 tok/s on verbatim code reproduction (91% draft acceptance, mean accepted span 59 tokens) but only fires on long untouched spans. The 120W power mode costs about 10% versus MAXN, and images cost a one-off 1-2s to encode without affecting decode speed.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codinglong-contexttool-usemultilingual
User reports Qwen3.8-Flash-Next at 101.3 t/s decode on Japanese prose and 170.0 t/s on code, single stream, on two NVIDIA CMP 170HX cards.
Setup is vLLM with W4A16 weights, BF16 KV cache, MTP k=4 speculative decoding, 262,144-token context, and expert parallel across the two cards. The PLE n-gram table was converted to FP8 locally to fit in 92 GiB of host RAM.
Aggregate throughput reaches 392 t/s at 4 concurrent requests. Prefill measures 3,191 t/s at 6,954 tokens and 3,261 t/s at 27,853 tokens. A 200,087-token prompt returned the planted code with 70.5 s time to first token and 69.0 t/s decode at depth. MTP is worth about 1.6x on prose and 2.6x on code. The user notes xhigh reasoning effort spent the whole budget without answering in 5 of 6 runs.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-context
User reports Qwen3.8-Flash-Next 180B W4A16 at 116.6 t/s decode on four RTX 3090s capped at 220 W each.
Setup is vLLM built from a pinned PR head with five patches, MTP speculative decoding with k=4, 65,536 context, and 76,046 KV tokens.
Decode holds 115.1 t/s at 32k and 72.75 t/s at 128k; prefill reads 3,187 t/s at 32k and 2,835 t/s at 128k. The c8 aggregate is 180.7-230.3 t/s on the bf16 profile. Peak short-prompt decode reaches 198.5 t/s on code. Spread is plus or minus 15% run to run.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
long-contextagentic
User reports Qwen3.8-Flash-Next at 96.9 tok/s single-request decode on four NVIDIA CMP 170HX 40GB cards.
Setup is llama.cpp with UD-Q4_K_XL weights, q8_0 KV cache, MTP speculative decoding (draft length 4, GPU sampling), 262K context, and the SM clock pinned at 1410 MHz.
The same configuration reaches 87.7 tok/s at 70K context; three concurrent 3K requests give 34-37 tok/s each (about 100 tok/s aggregate). Earlier revisions of the fork measured 64-73 tok/s at 2K and 58-70 tok/s at 70K, against a baseline fork at 46-55 tok/s and 27-45 tok/s respectively.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B Uncensored at 81.3 tok/s generation on an RTX 5090 plus RTX 4070 Ti Super.
Setup is vLLM 0.30.0 with block-FP8 weights, fp8 KV cache, full 262,144 context, MTP K=3, and pipeline parallel across both cards.
Prefill is 3,160 tok/s at 32K, 68.5-68.8 s for 145K fresh prefill, and 169 s for 258K prefill; MTP acceptance is about 87% and the 5090 peaks at 31,472 MiB.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User reports a Qwen3.8-Flash-Next MTP draft head for GGUFs on llama.cpp, measured on an M3 Max 128GB with Metal.
Setup is llama.cpp with an IQ3_XXS trunk and an unsloth shared Q8_0 MTP head, one session.
Decode is 31.7-35.2 tok/s with MTP against 23.8-27.4 without, draft acceptance 84-99% at 2 drafts. Five concurrent agent sessions cost 8-20% throughput at 2 drafts, while 1 draft at 89% acceptance is neutral. The work is blocked on unmerged llama.cpp PR #28243.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagenticvisionlong-context
User reports Qwen3.8-Flash-Next at 52.6 tok/s decode and 762 tok/s prefill on an M4 Max 64GB.
Setup is mlx-serve with an iQ-MLX 3.3bpw pack, 64k context, 52 GB resident; the 32 GB n-gram table is mmapped and not resident.
The same run measured the mixed-4-8bit pack at 55.5 tok/s decode, 754 tok/s prefill and 278 ms TTFT; MTP is in the pack but default-off and not re-measured.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
tool-uselong-context
User reports Qwen3.8 27B (Swift-1.5-GSQ-RCO IQ3_S with MTP) at 43.7 tok/s decode and 700.8 t/s prefill on an RX 9070 XT 16GB.
Setup is llama.cpp (server-vulkan b11176) with IQ3_S weights and a q5_0-q4_1 KV cache at 65536 context, one card.
Decode is the mean of 3 runs of a fixed 512-token generation; prefill is one cold 30,461-token prompt. At 131072 context the same Vulkan launch gives 37.4 tok/s decode and 701.6 t/s prefill, while the pinned ROCm image (b10884) cannot load at 128K and decodes 33.5 tok/s at 65536. The user attributes the ROCm failure to a missing FlashAttention vector kernel for the q5_0/q4_1 K/V pair, which promotes both to f16 and exhausts the 16GB card.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentictool-usevisionlong-context
User reports Qwen3.8-27B at 2061.11 t/s prompt and 24.70 t/s decode on two Radeon RX 9070 XT 16 GB cards under ROCm 7.2 on Windows.
Setup is a llama.cpp fork with Q4_K_M GGUF, f8_e4m3 KV cache, FlashAttention, batch 8192 / ubatch 1024, layer split across both GPUs, one server slot, and cold prompt processing at 98,304 context (L3 lane).
With MTP n3 the same L1 lane gives 1817.81 t/s prompt and 36.73 t/s decode at 46.8% acceptance. Vulkan rows are lower on prompt processing (L1 1569.17, L2 1470.80, L3 1243.99 t/s) and lead only the L1 decode rows. Linux ROCm 10 results reach 2253.06 t/s prompt with MXFP4-requant and 50.13 t/s decode with Q4_K_M plus MTP n3.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticcoding
User reports Qwen3.8-27B at ~9.7 tok/s generation and ~225 tok/s prompt processing on a single RTX 3060 12GB.
Setup is llama.cpp with Q4_K_S weights and Q4_0 KV cache at 96K context, using MTP speculative decoding with 46 FFN layers offloaded to system RAM.
MTP acceptance was ~88% with 2.76 mean accepted tokens; peak generation reached ~10.6 tok/s and VRAM usage was ~11.04 GB.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-27B at 29.65-29.69 generated tokens/s on one RTX 3060 12 GB, with a 33.21 tokens/s high-acceptance peak.
Setup is llama.cpp with IQ3_XXS GGUF, Q4_0 KV cache, 65,536-token context, MTP depth 6, and Deferred-Commit Factor Replay.
A separate prefill sweep improved an 8,595-token prompt from 82.13 to 320.24 prompt tokens/s. Q4 at 32K measured 16.06 tok/s on a Rust task and 14.30 tok/s on a C++ task; Q3 at 96K and 128K measured 21.3-22.0 and 19.7-20.9 tok/s respectively.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 22.99 tokens/s three-scenario decode average on an Apple M5 Pro with 64 GB of unified memory.
Setup is oMLX 0.6.0 with 4-bit affine MLX weights and an external Qwen3.8-27B-MTP-4bit drafter (250.9 MB, block size 3), measured at 16384 tokens of context.
The average combines 26.28 short, 22.81 long, and 19.88 at 11k context. First uncached 11k prefill was 368.32 tokens/s with 30.18 s TTFT. MTP accepted 1,836 of 3,416 drafted tokens (53.7%).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-contextsummarizationtool-useagentic
User reports Qwen3.8-27B at 158 tok/s decode and 7,265 tok/s prefill on an RTX 5090 32GB (eGPU via OCuLink Gen4 x4).
Setup is NInfer with NVFP4 weights, FP8 KV cache, 240K context, MTP3 speculative decoding at 76% acceptance, and 2 concurrent lanes. Decode rises to 213 tok/s at 32K and 202 tok/s at 128K; prefill is 6,892 tok/s at 32K and 3,904 tok/s at 128K.
User compares against llama.cpp (Q5_K_M GGUF, q8_0 KV, 196K context, MTP on) at 114 tok/s decode and 1,545 tok/s prefill at 1K, and notes vLLM at about 70 tok/s decode at short context. Quality was statistically indistinguishable across engines on a 250+ item eval. NInfer does not support json_mode.