Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 37.9 t/s on an Apple M5 Max with 36 GB of unified memory.
Setup is Rapid-MLX 0.16.1 default with MLX 4-bit weights and MTP speculative decoding on the text-only lane, image input disabled.
The figure is a single-stream decode rate, not end-to-end throughput or prompt processing, and comes from a median of three requests with a 512-token cap.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 5.8 t/s generation and 60.5 t/s prompt processing on two Apple Mac mini M4 machines with 16 GB unified RAM each.
Setup is llama.cpp build 9731ad3 with Metal and RPC, Unsloth UD-Q4_K_M, model split 50/50 over a Thunderbolt 4 bridge, with -fa 1 -ngl 99 -b 1024 -ub 512 and three repetitions.
The 5.8 t/s is the tg128 native llama-bench microtest and 60.5 t/s is the separate pp512 phase; one mini cannot fit the 16.5 GB artifact at its 13 GB GPU limit.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 23.9 t/s generation and 119.9 t/s prompt processing on an M4 Max 64GB.
Setup is llama.cpp (llama-server 0.4.0-dev, Metal) with Q4_K_M, --ctx-size 4096, --parallel 1, -b 4096, -ub 1024, flash attention on, no prompt cache, and speculation disabled.
The headline Quick Chat run used 87 prompt tokens and 96 output tokens, with TG 23.88856 t/s and PP 119.88656 t/s as medians of three measured requests after one discarded warmup.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 31.8 t/s generation and 391.2 t/s prompt processing on an M3 Ultra 512GB.
Setup is MLX with Q4 and a fresh KV cache per call, batch 1 on one device, at 32768 input tokens and 128 output tokens.
The figures are separate five-trial medians, not pooled across lengths; the 32768-token row measured 31.768 t/s generation and 391.157 t/s prompt processing with a 23.246GB peak.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 oQ4e-mtp restored-prefix B3 / PR1 61bd8a06 at 8.4 t/s on an M5 Max 128GB.
Setup is oMLX with oQ4e 4-bit weights and TurboQuant 4-bit KV cache, three greedy requests batched at roughly 20k / 45k / 135k prompt tokens with SSD prefix cache enabled.
The 8.4 t/s is the per-row rate at the 20k row, measured at 8.35 t/s; the other rows ran 8.83 and 7.84 t/s per row, and the batch phase took 66 seconds.
2× Mixed AMD GPUs (one Strix Halo APU and one Radeon Pro V620) · llama-halo-hybrid (llama.cpp fork) · 131,072 ctx
generation:
58.2 tokens/s
quant:
Custom Q8T (GGUF)
flash attention:
on
mtp (multi-token prediction):
on
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 at 58.2 t/s on one Strix Halo APU/iGPU plus one Radeon Pro V620 with 32 GB VRAM over PCIe Gen4 x4.
Setup is the llama-halo-hybrid HIP/ROCm fork with a Custom Q8T merge, 131072-token context, one slot, -fa on, -ngl 999 and draft-mtp with at most two draft tokens.
The 58.2 t/s is the single-stream median of 40 real category prompts, not a token-weighted aggregate, with prefix caching disabled and dense trunk, KV and draft on the card and routed experts on the iGPU.
2× Mixed AMD GPUs (one Strix Halo APU and one Radeon Pro V620) · llama-halo-hybrid (llama.cpp fork) · 131,072 ctx
generation:
58.2 tokens/s
quant:
UD-Q4_K_XL (GGUF)
flash attention:
on
mtp (multi-token prediction):
on
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 58.2 t/s on one Strix Halo APU/iGPU plus one Radeon Pro V620 with 32 GB VRAM over PCIe Gen4 x4.
Setup is the sixvolts/llama-halo-hybrid HIP/ROCm llama.cpp fork with UD-Q4_K_XL GGUF, 131072-token context, one slot, -fa on, -ngl 999 and draft-mtp with at most two draft tokens.
The 58.2 t/s is the single-stream median of 40 real category prompts with prefix caching disabled, not a token-weighted aggregate; prose-category speed is 46.9 t/s.
2× Mixed AMD GPUs (one Strix Halo APU and one RX 7800 XT) · llama-halo-hybrid (llama.cpp fork) · 131,072 ctx
generation:
60.0 tokens/s
quant:
UD-Q4_K_XL (GGUF)
flash attention:
on
mtp (multi-token prediction):
on
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 60 t/s on one Strix Halo APU/iGPU plus one Radeon RX 7800 XT with 16 GB VRAM.
Setup is the sixvolts/llama-halo-hybrid HIP/ROCm llama.cpp fork with UD-Q4_K_XL, 131072 tokens of context, one slot, -fa on, -ngl 999 and draft-mtp with at most two draft tokens.
The 60 t/s is the single-stream median of 40 real category prompts, not a token-weighted aggregate; prose-category speed is 49.6 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B Splash at 20.2 t/s on an Apple Mac mini M6 with 32 GB unified memory.
Setup is LM Studio with the Splash backend, 32,768 tokens of loaded context, and speculative decoding with DFlash2 drafts; quantization is unknown.
The 20.2 t/s is the median of three English-prose runs, with individual runs of 20.6, 20.2 and 20 t/s; it is client-observed streaming decode throughput, not a pooled rate or prompt-processing rate.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Swift-Qwen3.8-27B Splash at 19.5 t/s on an Apple Mac mini M6 with 32 GB unified memory.
Setup is LM Studio with the Splash backend and speculative decoding using DFlash2 drafts, loaded at 32,768 tokens of context.
The 19.5 t/s is the median of three English-prose runs and is client-observed streaming decode throughput, not a pooled rate or prompt-processing rate; exact quantization is unknown.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 11.8 t/s on an Apple Mac mini M6 with 32 GB unified memory.
Setup is LM Studio with its llama.cpp GGUF backend, GGUF Q4_K_M quantization, and a loaded context of 32,768 tokens.
The 11.8 t/s is the median of three English-prose runs (Ruhr economy prompt), client-observed streaming decode throughput, not an overall mean or prefill rate.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 12.9 t/s generation and 25 t/s prompt processing on two NVIDIA RTX 3060 12GB cards.
Setup is llama.cpp with the IQ4_XS GGUF quant, run as separate llama-bench PP512 and TG128 tests rather than one request.
The 87.24 GiB model file exceeds the cards' combined 24 GB VRAM, and GPU offload, KV cache type and context length are unreported.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 30.8 t/s generation and 755 t/s prompt processing on two NVIDIA RTX 3060 12GB cards.
Setup is llama.cpp with Q4_K_S, run as separate llama-bench tests: PP512 with 512 newly processed prompt tokens and TG128 with 128 generated tokens.
Repetitions, KV cache type, context length, thread counts and GPU placement are unreported, and the 14.29 GiB figure is model-file size, not measured resident memory.
2× NVIDIA RTX PRO 5000 Blackwell (one 72GB card and one 48GB card) · llama.cpp
generation:
62.3 tokens/s
prompt processing (prefill):
3002.5 tokens/s
quant:
Q8_0 (GGUF)
kv:
F16
flash attention:
on
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 62.3 t/s generation and 3002.5 t/s prompt processing on one RTX PRO 5000 Blackwell 72 GB plus one RTX PRO 5000 Blackwell 48 GB.
Setup is llama.cpp CUDA b10941 with Q8_0, f16 K/V cache, flash attention on, tensor split 1/1, -ngl 999, batch 2048 and microbatch 512.
The 62.3 t/s is TG128 and the 3002.5 t/s is PP512, separate tests, each the mean of five repetitions; the author reports TG reproducibility around 1% and PP drift up to about 10% between sessions.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 38.6 t/s generation and 2275.9 t/s prompt processing on one NVIDIA RTX PRO 5000 Blackwell 72GB.
Setup is llama.cpp CUDA b10941 with Q8_0, f16 K and V cache, flash attention on, batch 2048 and microbatch 512, whole model on the single card.
The 38.6 t/s is TG128, generated tokens in a separate test from the PP512 figure; each cell is the mean of five repetitions, and the author reports prompt-processing drift up to about 10% across sessions.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 29 t/s generation and 274 t/s prompt processing on an M3 Ultra 96GB.
Setup is llama.cpp 0.4.0 with the Q4_K_M GGUF, run natively through llama-bench on one Apple GPU via Metal.
The native benchmark gives 28.97 +/- 0.03 tok/s generation and 273.98 +/- 0.03 tok/s prompt processing; context length, batch settings and offload flags are unreported.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Strata 0.1.39 against llama.cpp on Qwen3.8-Flash-Next (IQ3_S) on an RTX PRO 6000 Blackwell 96GB, reporting 205.263 tok/s generation at a 4K prompt versus 102.889 tok/s for llama.cpp.
Setup is Strata with its draft head on, one request at a time, 256 tokens written greedy, median of three runs; the whole model sat on the card.
Strata also led on a 3090 Ti (98.291 vs 22.490 tok/s) and a 5090 laptop (85.6 vs 24.4 tok/s), where the model did not fit and experts were kept in system RAM. Strata pins 46.84 GiB of system RAM for this file on every card.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8-Flash-Next (125B MoE, 6B active) at ~95-105 tok/s on a single RTX 5090, with three lanes giving ~130 tok/s aggregate.
Setup is the ignis engine with a trellis 2.5-bit expert quant (67 GB file), expert cache in VRAM (~17 GB) and experts in pinned host RAM (~38 GB) over PCIe 3.0.
Quality: MMLU-Pro proxy 70.1% vs 73.7% for BF16, top-1 agreement with BF16 82-98% by domain.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 80 t/s generation and 1500 t/s prompt processing on a 16GB VRAM GPU.
Setup is llama.cpp-adaptive-kv-streaming with UD-IQ4_XS quant and q4_0 KV cache at 262k context, using a 1.5GB kv-stream-arena and batch/ubatch of 128.
At full 262k context the user gets around 20 t/s generation and 400 t/s prompt processing. The user notes the speed drop is linear and they normally compact at 180-200k. The fork was modified to support q4_0 KV, and the user mentions q8_0/q4_0 works with lower context or batch sizes.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 66.39 tok/s on one RTX 3090, versus a 41.55 tok/s non-speculative baseline, a +59.8% within-host effect.
Setup is llama.cpp with Q4_K_XL GGUF weights, q8_0 KV cache, 8192-token context, flash attention on, single stream, batch size 1, using the built-in MTP path at --spec-draft-n-max 2.
The faster path was not bit-exact with serial greedy decoding: 92-100% of requests diverged by a 1,600-token cap, and semantic quality was not measured. Deeper drafting was slower (MTP n-max 3 at 63.29 tok/s, DFlash2 n-max 4 at 63.13, MTP n-max 5 at 54.89, DFlash2 n-max 7 at 50.95). Board telemetry estimated a 37% decode energy reduction for a fixed 400-token workload.