Qwen3.8 reports

generation:

57.6 tokens/s

prompt processing (prefill):

6574.0 tokens/s

quant:

GPTQ-INT4 (GPTQ)

kv:

int8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 57.6 t/s decode and 6574 t/s prefill on six RTX 3090s. Setup is vLLM with a GPTQ-INT4 checkpoint and int8 KV cache, 250k context, PLE table on SSD, pipeline parallelism across six cards. Decode falls to 29.4 t/s per user with seven concurrent sessions (206 t/s aggregate); llama.cpp-multigpu holds 30 t/s per user at five sessions. NVFP4 checkpoints are 10-12% slower than GPTQ on these cards. Load to healthy is 4-5 minutes.

generation:

701.2 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

Q8

flash attention:

on

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 701.2 t/s on one NVIDIA RTX 5090, the median of three repeats (695.3, 709.0, 701.2) for a copy/edit task from a 27,135-token prompt with 400 completion tokens. Setup is llama.cpp b11060 CUDA with UD-Q4_K_XL, q8_0 key and value cache, Flash Attention, greedy sampling, and draft-mtp plus ngram-mod speculation with draft maximum 16 and n-gram range 96-192. The 131,072-token context is the server limit, not the measured prompt length, and the figure is not general chat or novel-code speed. At 3,038 prompt tokens the same profile returned a 784.1 t/s median (range 768.0-814.3).

generation:

12.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

F16

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 12 t/s generation on an M4 Max 128GB, with the source's 128K benchmark point given as 11.98 tokens/s. Setup is llama.cpp with UD-Q4_K_XL and f16 K/V cache, all layers offloaded, flash attention on, batch 1024 and microbatch 256. The figure is generation after cache prefilling, not prompt processing or TTFT; the source's f16 KV series is 16.62 t/s at 64K, 14.93 at 81K and 11.98 at 128K.

generation:

11.4 tokens/s

prompt processing (prefill):

96.8 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 11.4 t/s generation and 96.8 t/s prompt on a Mac mini M4 Pro with 24GB unified memory. Setup is llama.cpp b10488 with Metal, Flash Attention on, all layers offloaded, and Q4_K_M quant. The figures come from separate llama-bench pp2048 and tg128 tests, mean of three repetitions, not one request; run-to-run variance is claimed under 3%.

generation:

6.7 tokens/s

prompt processing (prefill):

312.0 tokens/s

quant:

Q8_0

kv:

Q8

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 6.7 t/s generation and 312 t/s prompt processing on an AMD Strix Halo 128GB box with a Radeon 8060S. Setup is llama.cpp on the rdna35-perf ROCm/HIP branch with Q8_0 weights, q8_0 KV cache, Flash Attention on and microbatch 2048, measured with llama-bench. Both figures are separate benchmark cells at 128k tokens already in the KV cache, not one end-to-end request; the prompt test processes 2048 additional tokens at that depth.

generation:

12.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 12 t/s on an M1 Max 64GB. Setup is Ollama with Q4_K_M and a 65,536-token context, one parallel request. The 12.0 tok/s is the second 6,872-token prompt run, not an aggregate; separate retrieval tests report prefill of roughly 90 tok/s at about 1,900 and 7,400 tokens, about 83 at 22,200, and 71-74 at 55,600.

generation:

18.5 tokens/s

prompt processing (prefill):

381.6 tokens/s

quant:

UD-IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 18.5 t/s generation and 381.6 t/s prompt processing on an M5 Pro with 64GB unified RAM. Setup is llama.cpp with the Unsloth UD-IQ4_XS quant, run via llama-bench at -p 512 -n 128. The source gives 18.45 +/- 0.03 tok/s and 381.62 +/- 0.18 tok/s; the generation figure is not a post-512-prompt end-to-end run, and context limit, KV precision and offload settings are unstated.

generation:

15.6 tokens/s

prompt processing (prefill):

109.6 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 15.6 t/s generation and 109.6 t/s prefill on an AMD Strix Halo 128GB workstation. Setup is llama.cpp with UD-Q4_K_XL, F16 KV cache and flash attention, 40 GPU layers offloaded in a single inference slot with 2048 MiB prompt-cache capacity, configured to 262144 context. CPU-backed per-layer token embeddings make this a hybrid offload rather than a full-GPU test, and actual input/output lengths and cache hits are unreported.

generation:

37.9 tokens/s

quant:

MLX affine 4-bit, group 64, 4.501 effective bpw (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 at 37.9 t/s on an M3 Ultra 512GB, from a single run of one 68-token prompt and 120 generated tokens. Setup is MLX with affine 4-bit quantization, group size 64 and 4.501 effective bpw, peaking at 15.5 GB of memory. The author describes the figure as a rough guide rather than a repeated controlled benchmark; no prompt-processing throughput or TTFT is given.

generation:

13.2 tokens/s

quant:

BF16 unquantized (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 at 13.2 t/s on an M3 Ultra with 512GB unified memory. Setup is mlx-vlm 0.6.13 with the unquantized bf16 MLX VLM build, peak memory 55.8 GB. The figure comes from one 470-token image-plus-text prompt, one run per build, presented as an order-of-magnitude guide rather than repeated benchmarking.

generation:

71.1 tokens/s

prompt processing (prefill):

1488.7 tokens/s

quant:

UD-Q4_K_XL (GGUF)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 at 71.1 t/s generation and 1488.7 t/s prompt processing on an AMD Strix Halo 128GB system. Setup is Gufo with UD-Q4_K_XL, a 262,144-token configured context, 512-token prefill chunks and MTP ceiling 7 with a shared Q8 MTP head. The run used 2,048 new prompt tokens at depth zero and 512 generated tokens, with 447 drafted and 447 accepted tokens; the exact rates were 1,488.66 PP and 71.08 TG.

generation:

39.9 tokens/s

prompt processing (prefill):

1060.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next UD-Q4_K_XL at 39.9 t/s decode and 1,060 t/s prefill on an ASUS ROG Flow Z13 with AMD Strix Halo, Ryzen AI Max+ 395 / Radeon 8060S and 128 GB unified memory. Setup is Gufo v0.7.1 built with ROCm 10, one server session on native Linux, with a shared Q8_0 MTP sidecar at 68% draft acceptance. Each row is the median of three fresh prompts, source-labelled 32k prompt within a 64k window, at a 70 W power limit with measured power 67-70 W.

generation:

26.4 tokens/s

prompt processing (prefill):

276.2 tokens/s

quant:

oQ8e mixed affine 8-bit + FP16 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 26.4 t/s generation and 276.2 t/s prefill on an Apple Mac Studio M2 Ultra with 128 GB unified memory. Setup is oMLX 0.6.3 with oQ8e mixed affine 8-bit plus FP16 weights, run as an ANE and CPU split at a source-labelled 65,536-token prompt depth. The GPU-only control at the same depth is 231.2 prefill and 26.6 decode; the author reports gains apply to cold prefills, not prefix-cached turns, and peak memory at 64K rises from 44.0 GB to 60.0 GB with ANE.

generation:

3.6 tokens/s

prompt processing (prefill):

21.0 tokens/s

quant:

TurboQuant 3-bit g64 + 4-bit affine extras (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 3.6 t/s generation and 21 t/s prefill on an M4 Mac mini with 16 GB unified memory. Setup is turboquant-mlx 0.24.0 with TurboQuant 3-bit g64 weights and 4-bit affine embeddings, lm_head and vision tower, at a 2014-token prompt with 13.14 GiB measured peak. The source's four rows span 220 to 5017 prompt tokens, with decode 3.6 to 3.9 t/s and prefill 19.0 to 21.0 t/s; generated-token counts and KV precision are unstated.

generation:

56.5 tokens/s

prompt processing (prefill):

655.2 tokens/s

quant:

CIRU v5 compact indexed affine (experts 4.5175 bpw) (GGUF)

kv:

F16

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-CIRU-STRIX-Orca v5.0 at 56.5 t/s generation and 655.2 t/s prompt processing on one AMD Ryzen AI Max+ 395 / Radeon 8060S system with 128 GiB shared unified memory. Setup is CIRU v5 native HIP with compact indexed-affine experts at 4.5175 bpw, F16 target and draft attention cache, a matching Q8 Orca MTP drafter at depth 4, one slot and 262144-token configured context. The run reused all but four tokens of a 256000-token prefix, then appended 2048 tokens and generated 512 output tokens; MTP accepted 408 of 408 proposed tokens. Full append-plus-generation wall time was 12.409 seconds, with no repeated 256K result.

generation:

59.2 tokens/s

prompt processing (prefill):

780.1 tokens/s

quant:

CIRU v5 compact indexed affine (experts 4.5175 bpw) (GGUF)

kv:

F16

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-CIRU-STRIX-Orca v5.0 at 59.2 t/s generation and 780.1 t/s prompt processing on one AMD Ryzen AI Max+ 395 / Radeon 8060S system with 128 GiB shared unified memory. Setup is CIRU v5 native HIP with compact indexed-affine routed experts at 4.5175 bpw, F16 target and draft attention cache, a matching Q8 Orca MTP drafter at depth 4, and 262144-token configured context. The headline is one initial observation at 64,000-token prior prefix depth with 2,048 appended tokens and 512 output tokens, not an aggregate; the reversed-order repeat gave 903.82 t/s append and 58.67 t/s generation.

prompt processing (prefill):

1003.0 tokens/s

quant:

CIRU v5 compact indexed affine (experts 4.5175 bpw) (GGUF)

kv:

F16

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-CIRU-STRIX-Orca v5.0 at 1003 t/s prompt processing on one AMD Ryzen AI Max+ 395 / Radeon 8060S system with 128 GiB shared unified memory. Setup is CIRU v5 native HIP with the compact indexed-affine expert format at 4.5175 bpw, a 262144-token configured context, 16384 batch and microbatch, F16 target and draft attention cache, and a Q8 Orca MTP drafter at depth 4. The figure is a full-prefix prefill of 256000 tokens from zero cached prefix, not the later 2048-token cached append and not TTFT; one initial observation, no repeats.

prompt processing (prefill):

1251.4 tokens/s

quant:

CIRU v5 compact indexed affine (experts 4.5175 bpw) (GGUF)

kv:

F16

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-CIRU-STRIX-Orca v5.0 at 1251.4 t/s prompt processing on one AMD Ryzen AI Max+ 395 / Radeon 8060S system with 128 GiB shared unified memory. Setup is CIRU v5 native HIP with compact indexed-affine experts at 4.5175 bpw, F16 target and draft attention cache, a Q8 Orca MTP drafter at depth 4, one slot and 262144-token configured context. The figure is a full-prefix prefill of 64,000 prompt tokens from zero cached prefix, not TTFT and not the later 2048-token cached append; a separate reverse-order 64K run measured 1267.40 t/s.

prompt processing (prefill):

1821.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 1821 t/s prompt processing on an AMD Strix Halo 128GB host (GMKtec EVO-X2, Ryzen AI Max+ 395 / Radeon 8060S plus XDNA2 NPU). Setup is the gufo NPU-enabled fork with UD-IQ4_XS weights, 8192-token prefill chunks, GPU fabric clock held at its top level and NPU performance mode. The figure is a warmed synthetic-prefill microbenchmark at 16,384-token prefill length, not configured context capacity, and no generation throughput is published. The matched GPU-only baseline is 1698 t/s, a source-reported 7.2% improvement.

generation:

8.1 tokens/s

prompt processing (prefill):

466.0 tokens/s

quant:

Q8_0

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 8.1 t/s generation and 466 t/s prompt processing on an AMD Gorgon Halo 192GB machine. Setup is llama.cpp with Q8_0 and all layers on the GPU, on a Minisforum MS-S1 MAX-P495 with Ryzen AI Max+ PRO 495 and 192 GB unified memory. The 466 t/s and 8.1 t/s come from separate llama-bench pp512 and tg128 tests, not a paired 512-input/128-output run; the same source separately reports about 100 t/s prompt and 7.5 t/s generation at roughly 32k depth.