Qwen3.8 reports

generation:

91.8 tokens/s

quant:

4-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User compares Inco Splash 1.1 and oMLX 0.7.0rc1 running an abliterated Qwen3.8-27B on an M3 Max 48GB MacBook Pro. Splash with the DFlash2 draft model reached 91.8 tok/s on code prompts, 52.6 tok/s on creative writing, and 37.7 tok/s on stock Qwen, while oMLX with MTP was in the low-to-mid 30s tok/s. Setup used MLX 4-bit weights on Splash and oQ4e on oMLX, thinking off, temp 0.7, max 600 tokens, median of 3 runs, one engine at a time. User notes the quant is not identical between engines, prompts were short, and long context and concurrent requests were not tested.

generation:

70.2 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 70.22 tok/s on a Strix Halo device, reproducing Gufo's published 70.56 tok/s figure. Setup is Gufo 0.4.0 with UD-Q4_K_XL GGUF weights and a DFlash2 Q4_K_M draft model, greedy decoding, thinking off, 128 output tokens, prompt cache off. The 70 tok/s figure comes from a repetitive prompt ("Write the word red exactly 1000 times") where speculative decoding accepts nearly every draft token. On nine ordinary prompts the median is 39.4 tok/s, ranging from 22 to 52. With 8 users the aggregated figure is 122.6 tok/s but wall-clock token delivery is 82 tok/s on the repetitive prompt and 52 on normal prompts. Without the draft model Gufo's docs put it around 12 tok/s. In a head-to-head against halogen 0.13.8 on Qwen3.8 Flash-Next, halogen averaged 43.9 tok/s vs Gufo's 38.2 on normal prompts, and 76.6 vs 63.1 tok/s with 4 users; Gufo was faster at cold prefill (1495 vs 1288 tok/s) and on the repetitive prompt (87.4 vs 56.9 tok/s).

generation:

50.0 tokens/s

prompt processing (prefill):

850.0 tokens/s

quant:

IQ3_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports 850 t/s prefill and 50 t/s generation with MTP at temperature 0 on an AMD RX 9060 XT 16GB, running Qwen3.8-27B-GSQ-RCO IQ3_S in a custom inference engine. MTP is used for the generation figure; the user notes MTP with temperature above 0 is not developed yet. User asks for ideas to systematically test the engine, having already run KL divergence against BF12 on CPU, bit correctness checks, needle-in-a-haystack at 25/50/75/90% key positions with distractors (also against llama.cpp), and HumanEval (92/93 passed so far).

generation:

48.0 tokens/s

prompt processing (prefill):

198.0 tokens/s

quant:

NVFP4 (MLX)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 48 tok/s decode and 198 tok/s prompt processing at 4K context on an M4 Max with 36GB unified memory. Setup is Ollama 0.34.4 with NVFP4/MLX weights, q8_0 KV cache, flash attention enabled, and a 128K configured context. The GGUF Q4_K_M version of the same model reached only about 10.7 tok/s decode. Decode falls to 44 tok/s at 8K, 25 tok/s at 16K and 32K, and 14 tok/s at 115K, where time to first token reaches 21 minutes. The user notes that maximum context is not the same as comfortable interactive context and that memory pressure and swap grew during the stress runs.

generation:

29.0 tokens/s

quant:

oQ8e (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next 8-bit at 29 t/s decode on a 1M-token prompt on an M5 Ultra 256GB Mac Studio. Setup is oMLX 0.7.0 with oQ8e weights and YaRN x4 position scaling; the 1M-token first read took 5.3 minutes and the next turn 6.7 s, finding all three hidden codes. At 250k context the model prefills at 4,235 t/s and decodes at 63 t/s; the user notes YaRN stretches a 262k-trained model and that a needle test does not prove reasoning at 1M.

generation:

125.2 tokens/s

prompt processing (prefill):

4112.6 tokens/s

quant:

NVFP4 (NVFP4)

kv:

INT8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next NVFP4 at 125.17 tok/s decode and 4,112.65 tok/s prefill on a dual-GPU setup with an RTX 4090 D 48GB and an RTX 5070 Ti 16GB. Setup is the Strata engine 0.1.35 with an experimental dual-GPU NVFP4 fork, INT8 KV cache, 262,144-token context, and MTP K4 speculative decoding. The 4090 D handled dense layers, KV cache, prefill, output head and MTP, while the 5070 Ti stored 4,500 additional routed experts. The figures come from a single-run capacity test with a synthetic 256,018-token prompt built by repeating code-review material, which the user notes had unusually favorable locality and should be treated as a best-case result. An abliterated NVFP4 checkpoint reached 92.05 tok/s decode and 4,071.39 tok/s prefill with 69.7% draft acceptance. During real coding-agent workloads the NVIDIA model produced around 108-144 tok/s.

generation:

38-61 tokens/s

prompt processing (prefill):

1650.0 tokens/s

quant:

UD-Q3_K_XL (GGUF)

kv:

fp16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next running on an RTX 3090 24GB with 128GB system RAM, reaching about 1650 t/s prompt processing and 38 to 61 t/s generation depending on context under Strata. Setup is Strata with Unsloth UD-Q3_K_XL quant, fp16 KV cache, 256k context, speculative decoding with 4 draft tokens, and an expert cache of 6517 slots (~14GB). The same model on llama.cpp master gave up to 700 t/s PP and 23 t/s TG. Generation is a range: 38 t/s at 182k context and about 61 t/s at short context. The user notes the quant Strata recommends by default was faster but produced minor errors and lower quality, so Unsloth's was chosen for accuracy.

generation:

500.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User announces MegaCapybara, a purpose-built engine for RTX 5090 focused on Qwen3.8 27B. Claims roughly twice the decode speed of Ninfer, reaching up to 650 t/s in small bursts and 2600 t/s with 12 concurrent agent slots at 800k context. Setup includes custom kernels, Dflash2 with DSpark headers, draft trees, and smart VRAM-RAM-DISC cache management.

generation:

40-60 tokens/s

quant:

IQ3_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User asks whether to keep MTP draft enabled for agentic work with Qwen3.8-27B IQ3_S on 16GB of VRAM, served through llama.cpp for a local Hermes agent. With MTP enabled the user reports prefill is slower, the model seems less smart, and VRAM is tight enough to force a 96k context; decode runs around 40 to 60 t/s. With MTP disabled the context can go to 128k but decode drops to about 35 t/s.

generation:

29.2 tokens/s

quant:

UD-IQ3_XXS (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B at 29.18 tok/s decode on an RTX 5060 Ti 16 GB at 4K context. Setup is llama.cpp build ad1de39e0 with UD-IQ3_XXS quant and q4_0 K/V cache, full GPU offload, flash attention, one parallel slot. Baseline decode without speculation is 29.18 tok/s at 4K, 23.62 at 32K and 19.77 at 64K; MTP n=2 gives 56.91/42.53/34.95 and MTP n=3 gives 63.89/45.48/39.06 tok/s at the same contexts. N-gram speculation added only 1.7% (29.66 tok/s). A 96K prompt measured 617.73 prompt tok/s and 36.74 decode tok/s.

generation:

50-55 tokens/s

quant:

UD-IQ3_XXS (GGUF)

kv:

Q4_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at a steady 50-55 t/s with MTP on a single RTX 5060 Ti 16GB. Setup is llama.cpp with UD-IQ3_XXS GGUF weights and Q4_0 KV cache for K and V, 94208-token context, batch and ubatch 512, flash attention on, draft-mtp speculation with spec-draft-n-max 3, parallel 1. Without MTP the average is 35 t/s. The machine has an Intel i5 10th gen and 32GB DDR4.

generation:

63.7 tokens/s

prompt processing (prefill):

680-850 tokens/s

quant:

IQ4_XS (GGUF)

kv:

Q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 63.7 tok/s decode on an RTX 5060 Ti 16GB, using MTP-5 speculative decoding at 8K context. Setup is llama-tierkv (a llama.cpp fork) with IQ4_XS weights and a Q4_0 KV cache kept in three tiers (VRAM, RAM, SSD); the weights are on the GPU. Without speculation the same configuration decodes at 25.7 tok/s, and prefill is 680-850 tok/s. At 256K context without speculation, decode is 22.70 tok/s.

prompt processing (prefill):

176-190 tokens/s

quant:

4bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 31.1-33.3 tok/s with speculative decoding on an M3 Max. Setup is MLX with 4-bit weights and k=3 draft tokens; plain decode without speculation gives 18.2-18.4 tok/s in Python and 19.1-19.4 tok/s in C++, with 84% acceptance and 3.49 tokens per pass. Prefill is about 176-190 tok/s and peak memory is 16.3-16.9 GB. The M3 Max memory size is not stated.

generation:

29.7 tokens/s

quant:

W4A8 (safetensors)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 29.74 tok/s median (34.34 peak) single-request on two Huawei Atlas 300I Duo cards (four Ascend 310P3 ranks, 192 GB nameplate, ~172 GiB usable). Setup is a forked vLLM and vLLM Ascend with W4A8 packed experts, MTP2 speculative decoding, ACL graphs, and a 262,144-token context limit; ECC disabled. Four concurrent requests reach 60.88 tok/s aggregate; two concurrent 40K warm-prefix requests reach 30.76 tok/s aggregate; a 40K cold prompt takes about 120 seconds TTFT then 30-32 tok/s. A 106-case long-reasoning phase emitted 507,256 tokens in 2 h 52 m 46 s (48.93 tok/s aggregate, 12.50 tok/s median per request). GPQA Diamond scored 140/198 (70.71%) on the Ascend W4 service and the same on an RTX 6000 Pro with IQ4_XS GGUF in llama.cpp. GLM-5.3-Flash bring-up reaches 2.236 tok/s single-request and 4.476 tok/s aggregate at four requests.

prompt processing (prefill):

1796.0 tokens/s

quant:

IQ3_XXS (GGUF)

kv:

int8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 1,796 tok/s prompt and 128-134 tok/s decode on an RTX 5090 32 GB plus an RTX 4070 Ti SUPER 16 GB. Setup is Strata with IQ3_XXS GGUF, int8 KV cache, 32K cells resident per layer, MTP spec 4, vision on, and 262,144 context; the 76 GB model runs with 31 GB of system RAM and experts streamed from an mmap'd GGUF. Decode is a range of 128-134 tok/s median on real sampling and 95-108 greedy; stock 0.1.33 with a layer split at 36 gave 890 tok/s on an 80K prompt and about 110 tok/s decode. Three patches cut major faults from 24 million to 14 thousand and chat switching from 20-48 s to 0.6-1.2 s.

generation:

31.8 tokens/s

quant:

oQ8e

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Altworld's Hemmingway-1, a 27B fine-tune of Qwen3.8-27B, at ~31.8 tok/s average across 3 runs on an M5 Max, with ~29-31 GB VRAM peak. Setup is MTP on with thinking on; the model is an oQ8e quant. User compares it against the base Qwen3.8-27B-oQ8e-mtp on the same machine and settings, where an LLM judge scored one Qwen run 95.35 vs 94.65 for Hemmingway. User finds Hemmingway reads more like human-written prose while Qwen feels technical, and recommends the base Qwen for generalist or coding work.

generation:

36-41 tokens/s

prompt processing (prefill):

175-183 tokens/s

quant:

IQ4_XS (GGUF)

kv:

q4_1

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B CODER decoding at 36-41 t/s on an RX 7900 XTX with 110k-170k tokens already in context. Setup is llama.cpp b11235 on the Vulkan backend with an IQ4_XS imatrix GGUF, q4_1 KV cache, 262144-token context, all layers on the GPU, and MTP speculative decoding from the file's own Q8_0 draft head. Across 98 agentic coding requests the median decode was 41.2 t/s (33.8-46.0) at 65k-131k context and 36.0 t/s (30.7-43.0) at 131k-171k. MTP draft acceptance median 85% (75-94%), about 2.7 tokens per decode step at draft depth 2. Prefill 387 t/s for a cold 108k-token prompt and 175-183 t/s for about 4.5k new tokens at 147k-156k depth. VRAM use 24.2 of 24.6 GB at 245,760 tokens of context, with the vision projector on the CPU.

generation:

10.0 tokens/s

prompt processing (prefill):

150.0 tokens/s

quant:

IQ4_XS (GGUF)

kv:

q8_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 10 t/s decode and 150 t/s prefill on an RTX 3060 12GB with 128 GB RAM. Setup is llama.cpp with IQ4_XS GGUF and q8_0 KV cache at 200k context, with 48 MoE layers offloaded to CPU. The post is a general inference engineering guide; the figures come from the user's sample launch command for this model.

generation:

150-200 tokens/s

prompt processing (prefill):

5000-6000 tokens/s

quant:

IQ3_S (GGUF)

kv:

8-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at IQ3_S running on a power limited RTX 5090 with 96GB of DDR5-6400, giving 150-200 tok/s decode and 5-6k prefill. Setup is Strata with IQ3_S weights and 8-bit KV cache at 128k tokens of context. Both figures are given as ranges, so no single throughput is recorded.

generation:

12.9 tokens/s

prompt processing (prefill):

355.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 12.9 tok/s decode and 355 tok/s prefill at 155K context on an RTX 4080 Super 16 GB. Setup is the QwFNfer engine with UD-Q4_K_XL weights and q8_0 KV cache; the dense core and a VRAM expert tier stay on the card while the remaining experts are read from 30 GB of RAM and NVMe, with attention state partly in pinned RAM. Short chat decode is 13.2-15.7 tok/s for Q4 and 19.8-21.2 tok/s for Q3; at 155K context Q3 gives 348 tok/s prefill and 17.7 tok/s decode. A speculative block predicts the next layer's experts at 95-96% accuracy, with no token-level speculative decoding.