Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-27B at 73 t/s on code and 58.5 t/s on prose on dual RTX 3090s (24 GB each, 48 GB total).
Setup is a Q8_K_XL quant (31.5 GB) with mmproj-F16 (0.93 GB), 200K context and an f16 KV cache, tensor split 50/50. MTP draft depth 2 gives the best acceptance at 92% on code and 68% on prose.
The speeds are medians of 3 draws at temp 0. The hybrid architecture has 48 of 64 layers as Gated DeltaNet (linear attention), with only 16 full attention layers plus the MTP head, for 17 KV-caching layers. Cold start is 32s. The split is ~1.1 GB lopsided, causing OOM risk at 220K context. User asks for optimization suggestions for dual 3090 setups.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 70.19 tok/s end-to-end (71.00 tok/s decode) for a single request on an RTX 3090.
Setup is the RTX 3090 port of NInfer, a C++/CUDA inference runtime, with MTP3 speculative decoding at 61.13% acceptance, CUDA Graphs enabled and 1,024 output tokens per request, peaking at 19,641 MiB VRAM.
At eight concurrent requests decode reaches 165.33 tok/s. Qwen3-35B-A3B runs at around 260 tok/s single-stream on the same setup, and over 400 tok/s on highly repetitive workloads.
User benchmarks 200 prompts at max concurrency 100, reaching 15,058.74 tok/s total token throughput and 1,673.19 tok/s output token throughput, with a peak output of 3,000 tok/s.
Setup is vLLM with Model Runner V2, p2p enabled, and tensor parallel 2.
Mean TTFT is 1,741.86 ms and mean TPOT is 45.74 ms. An AMD Radeon PRO W6600 is mentioned but not used for this benchmark.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports a dual RTX 3060 12GB setup with tensor split.
MTP draft accept is 4/4 (100%) on probe.
VRAM usage is 23.2/24.0 GiB.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingmath
User reports speculative decoding with mlx-dspark on an 8-bit target at a 2.45x mean speedup, 8.3 to 20.3 tok/s.
Setup is mlx-dspark with a 4-bit target at 1.74x and 25.3 tok/s.
The 8-bit target with drafter beats plain 4-bit at 14.6 tok/s.
User compares GGUFs of Qwen3.8-27B on 4x RTX 5090.
Quants mentioned are Q8_0, IQ1_M, Q3_K_M, AD-IQ3_S and Q6_K, with KLD values reported.
No tokens per second are given.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at Q3_K_XL on an RTX 5070 Ti 16GB with llama.cpp, reaching about 73 t/s at 32k context with MTP on and fully GPU-resident.
At-depth generation is 64.5 t/s at 24.1k depth, with prefill at 1647 t/s at depth. MTP off gives about 50 t/s.
The practical context ceiling is 40k, or 49k with MTP off. The user also compares Q3_K_XL against Q2_K_XL and finds Q3 faster than its size predicts.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports that DSpark GGUF for Qwen3.8 27B shows no performance improvement and is extremely memory hungry.
Setup is llama.cpp with DSpark support merged in PR #25173, using a converted and quantized RadixArk/Qwen3.8-27B-DSpark GGUF, with context set to 64k to test the BF16 speculator.
Draft acceptance is 0.28922 with a mean length of 3.02.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User benchmarks Qwen3.8-27B on an M5 Max 128GB across four engines.
Setup is llama.cpp with native MTP at 31.8 t/s sustained and 22.7 t/s at 56k context, TTFT 125ms, prefix reuse 0.80, RAM 36.8GB; oMLX 0.6 at 29.8 t/s and 24.6 t/s at depth, TTFT 1968ms, prefix reuse 0.82, ~17GB; mlx-vlm with MTP at 43.3 t/s and 28.9 t/s at depth, TTFT 9982ms, no prefix reuse, 15.6GB; and MTPLX 2.6 at 27.1 t/s and 23.1 t/s at depth, TTFT 13400ms, no prefix reuse.
The fastest engine, mlx-vlm, is unusable for agents due to 10s TTFT and no prefix reuse. Without --spec-type draft-mtp, llama.cpp drops the MTP head; a draft cap of 4 gives 31.8 t/s versus 21.9 t/s with no speculation, cap 2 from Qwen3.6 presets gives only 22.2 t/s, and cap 6 loses gains. At 262k context with q8_0 KV cache llama.cpp gives 29.6 t/s and 18.5 t/s at depth; at 131k with f16 it gives 31.8 t/s and 22.7 t/s at depth. The model does 5 searches then calls more tools instead of writing the brief and dies at max_turns. reasoning_effort 'none' throws an exception on 3.8, so enable_thinking: false is needed; oMLX does not activate MTP on 3.8; the model field of the response should be checked rather than /v1/models; and the M5 Max throttles to 50% after 80-110s. No generational verdict versus 3.6 is given.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports decode speeds on an RTX 4090 with custom-tuned FP8 kernels: no MTP about 19.7 t/s, MTP=3 about 49.7 t/s, MTP=5 about 53.2 t/s, and with custom kernels about 68.4 t/s.
Setup uses MTP=5 speculative decoding.
The numbers come from real Cursor sessions, not fixed-prompt A/B.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 24 t/s on a Ryzen AI Max+ 395 and 51 t/s on a Radeon AI PRO R9700.
The 51 t/s figure is attributed to the R9700, not the Strix Halo.
User asks about t/s for various quants and settings.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User compares Q5 and Q6 quants of Qwen3.8 27B on a three.js 3D arena game. Q6 took 3h32m at 6.01 t/s with 84,183 tokens, while Q5 took 13m23s at 71.98 t/s with 57,803 tokens.
Qwen3.6 Q5 was also tested at 84.79 t/s, and Q5 with medium reasoning reached 95.87 t/s.
The user notes Q6 ran at temp 0.6 instead of 1.0.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 8 t/s without MTP, 14 t/s with MTP, and about 19 t/s with full optimizations.
Setup is beellama 0.4.3, a llama.cpp fork.
User reports KLD against BF16 for custom EXL3 quants of Qwen3.8-27B: 0.007406 for EXL3, 0.013126 for FP8, and 0.094978 for NVFP4.
The quants require a custom vLLM fork, Gilded Gnosis.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks context length against KV cache quants on an RTX 5090 with Qwen3.8 27B MTP Q8_0 GGUF.
Maximum stable context is 105,216 tokens with a Q8_0 KV cache, 137,984 with Q5_1, and 169,984 with Q4_0.
Throughput dropped from 100 t/s to 25 t/s, which the user attributes to a llama.cpp issue.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports an aggregate 880 t/s at 6 parallel requests, peaking at 967 t/s, and over 200 t/s single stream with MTP speculative decoding.
Prefill runs at about 5,950 t/s. Weights take 16.8 GiB, leaving about 13 GiB for KV cache.
Benchmarks are HumanEval+ 152/164 and AIME25+AIME26 55/60, identical to int4 but 1.56x-1.98x faster. The run requires Blackwell FP4 cores and a 6-line patch not yet upstream.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
long-context
User reports a recommended single-card preset for Qwen3.8 27B on an RTX 5060 Ti 16GB, with sustained decode at 45.9K prompt tokens.
Two retrieval and two sustained checks passed.
User also mentions a 2x RTX 5060 Ti setup with the Q6_K quant at 131K context, reaching 38.6 t/s decode and 597.9 t/s prefill.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
tool-usebusinessmath
User reports Qwen3.8 27B at 59 t/s single stream on an RTX 5090, with aggregate throughput above 100 t/s for two requests and 279 t/s with four parallel jobs.
Setup is vLLM 0.27.1 with the PrismaAqua 5.5-bit quant. The user also mentions the RTX 6000 Pro as target hardware.
Quality is close to BF16 on a custom suite. The user prefers medium reasoning over xhigh because of fewer token-cap failures.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingmathtool-uselong-context
User compares NVFP4 on vLLM against Q6_K on llama.cpp, both on 2x RTX 5060 Ti.
NVFP4 reaches 952.5 t/s prefill and 67.3 t/s decode with 0.696 MTP acceptance. Q6_K reaches 597.9 t/s prefill and 38.6 t/s decode.
On quality, NVFP4 scores 65/68 correct, 64/68 exact format and 61/68 both, while Q6_K scores 66/68, 68/68 and 66/68. Both score 10/10 on long-context retrieval. NVFP4 is the default endpoint, with Q6_K kept for stricter output.
User benchmarks Qwen3.8-27B on 4x RTX 3090 (24 GB each) with vLLM 0.20.1, reaching 587.4 tok/s at c32 with TP=2 on one NVLink pair versus 417.9 tok/s with TP=4.
Setup uses tensor parallelism across the cards, with a KV pool of 1.45M tokens on TP=4 and 503k tokens on TP=2, and a max context of 262k.
MTP speculative decoding is slower at all concurrency levels, and pipeline-parallel throughput stays flat with concurrency.