Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 47.3 tok/s decode on 2x RTX 3060 12GB.
Setup is llama.cpp (server-cuda12-b11146) with UD-Q4_K_XL GGUF, tensor split, auto-fit and the GGUF's built-in MTP head, at 131,072 tokens of context. TTFT 537 ms.
The same PR also validates 20 single-card RTX 3060 12GB recipes and adds a second two-card recipe, Qwen3.6-35B-A3B at 74.2 tok/s decode and 147 ms TTFT at 131,072 tokens. Long-context recall was tested at 104,941 tokens. Several candidates failed (missing quant tag pulling default Q4_K_M, empty model path, OOM) and 16 were skipped for 512-2,048-token contexts.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codinglong-context
User reports Qwen3.8-27B dense at 75.81 tok/s decode on an RTX 3090 eGPU attached to a MacBook Air M2 over Thunderbolt 4.
Setup is the ThunderLlamaX engine with IQ3_XXS GGUF weights, 100,352-token context, K=10 n-gram LOOKUP speculative decoding, and a custom DriverKit dext replacing the missing NVIDIA driver stack.
Prefill is 569.2 tok/s at 2k, 510.1 at 8k, and 342.0 at 100k. On novel prose the EAGLE draft head gives 43.7 tok/s. A Qwen3.6-35B-A3B MoE model runs at 97.6-104.0 tok/s quote-class and 40.1 tok/s prose with 982.0 tok/s prefill at 2k. GSM8K is 95.0% dense and 93.0% MoE. A 100k context restores from cache in about 6.5 seconds.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports Qwen 3.8 27B at about 40 t/s generation and 500 t/s prompt processing on dual GPUs (RTX 5060 Ti 16GB + RTX 3060 Ti 8GB).
Setup uses a Q3 quant with 140k context across 24GB of VRAM, for single-user agentic coding via Opencode.
User finds the usable work window between context loads too small for complex projects and is asking whether to add a second 5060 Ti and more RAM for a Strata setup, or switch to a 128GB Strix Halo.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticlong-context
User reports Qwen3.8 Flash-Next 125B at 105 t/s generation and 5000 t/s prompt processing on 4x RTX 3080 20GB modded cards.
Setup is Strata with IQ3_XXS quant at 150000 context length, running via docker-compose on Unraid.
User says they are absolutely impressed and tested with a Hermes agent at 150k context.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 Flash-Next Q4 at 17.5 t/s decode and 360 t/s prompt processing on a Mac Mini M5 64GB, with experts streamed from SSD.
Setup streams experts from two external SSDs with a carousel buffer for prompt processing, which the user says added 30% to prompt processing and 15% to both prompt processing and decode.
The user notes the GPU still waits on expert streaming during decode and sees room for an optimised Metal kernel.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingsummarizationtool-useagenticlong-context
User reports Qwen3.8 27B at roughly 148 tok/s on an RTX 5090 with MTP enabled.
Setup is llama.cpp with NVFP4 weights and MTP speculative decoding on a single 32GB card.
Without MTP the same model ran at about 72 tok/s, and a Q4_K_M build at about 70 tok/s. Other models tested: GPT-OSS 20B MXFP4 at about 311 tok/s, Qwen3.6 hybrid at about 250 tok/s, Gemma 31B QAT at about 68 tok/s, and Flash Next at about 199 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-contextagentic
User compares Qwen3.8-27B dense on NInfer against Qwen3.8-Flash-Next IQ2_XS on Strata 0.1.39, both on one RTX 4090 24 GB with 262K context and MTP.
The 27B decodes at 127-140 tok/s with rk4v4-e8 KV and 23,134 MiB peak VRAM; the MoE decodes at 173-185 tok/s with q4_0 KV in VRAM and experts in host RAM, peaking at 23,936 MiB. Recall at 55K/117K/200K is 3/3 for both, with the 27B at 3,037/2,391/1,857 prompt tok/s and the MoE at 3,494/3,904/3,833.
On three graded tickets the 27B scored 112/150 versus 86/150 for the MoE. The user attributes the difference to cache reuse: 88.5% for the 27B, including one turn serving 172,343 of 172,578 tokens from cache with first token in 524 ms, against 32% for the MoE, whose typical turn re-read 107K tokens in 28 s with conversation cache off on 32 GB RAM. Limits are n=1 per ticket, one grader model, and the MoE ran first.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 33.96 t/s generation on Strix Halo 128GB.
Setup is Gufo with Q4_K_XL GGUF and DFlash2 speculative decoding using a Q4_K_M draft model. The prompt processing figure of 183.3 t/s is noted as low because of the short prompt.
User notes it is slower on Q8_0 but still faster than llama.cpp, and is testing the new AUR package.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next Swift-1.5 at 105 t/s decode and around 2000 t/s prefill on an RTX 3090 Ti plus a V100 16GB.
Setup is Strata 0.1.39 with IQ3-XXS weights, int8 KV cache, 393216 max context, yarn rope scaling at 1.5, and speculative decoding with --spec 4. Layers are split 2/3 on the 3090 Ti and 1/3 on the V100.
User says the same model previously reached only around 50 t/s decode with llama.cpp layer splitting.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
long-context
User shares a tuning guide prompt for Strata, citing reference figures from a single RTX 4090 with 32 GB RAM running Qwen3.8-Flash-Next at IQ2_XS.
Reference setup uses q4_0 KV cache at 262K native context, and yarn rope scaling with rope-scale 2, --max-context 524288 and a streamed KV cache (--kv-resident 32768) for 512K. A 477K-token prompt read in about 150 s and decoded at about 85 tok/s at that depth on that box.
The guide is a template for other users to fill in their own hardware; the 4090 figures are starting points, not targets. 512K is marked experimental and requires a recall test at 400K+.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User reports running Qwen3.8 27B with tensor parallelism across 4x RTX 5060 Ti 16GB in a Lenovo P620 (Threadripper PRO 3975WX, 128GB DDR4).
Setup is vLLM with FP8 weights and MTP3 speculative decoding, headless Ubuntu, NCCL 2.30.7. Decode at C1 is 100-110 tokens/s and prefill is 2.5-3k tokens/s per stream.
User compares NVFP4+MTP3 (137-141 t/s C1, 422 t/s C4 sum, ~3.5k prefill), SGLang NVFP4+DFlash (118-137 t/s C1, 357-386 t/s C4 sum, ~3.4k prefill), and llama.cpp Q6+MTP3 (95 t/s C1, 83 t/s C4 sum, ~1.2k prefill). NVFP4 was 1.4x faster than FP8 but degraded in MTP3 tests (F1 0.44 vs 0.73).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-Flash-Next at 30.5 t/s generation and 250.4 t/s prompt processing on 2x RTX 5060 Ti with 32GB system RAM.
Setup is llama.cpp with an IQ1_M GGUF (27.58 GB), 98304 context, q8_0 KV cache, tensor split 1,1, and 8 MoE layers offloaded to CPU.
User asks whether their settings are correct and what config others run for this model on this hardware.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User reports Qwen3.8-Flash-Next at 94-99 t/s sustained decode on 4x RTX 5060 Ti 16GB, with short turns reaching ~110 t/s.
Setup is a custom Strata fork (1 main + 3 expert tiers) with UD-Q4_K_XL quant, 524k context via ported rope-scaling, and a draft vocab fix. Prefill measured at 2159 t/s at 16k and 2420 t/s at 220k; 312k warm replay in 12ms.
User abandoned the setup because the Q4_K_XL quant loops in agentic use, repeating thoughts and re-firing tool calls, unlike their int4 W4A16 lane. They compare it favorably to their vLLM lane (~40 t/s at 455k with batch 4) but say the quality bar is not met for daily driving.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagenticlong-contextvision
User reports Qwen3.8-27B at ~200 tok/s single stream on one RTX 5090.
Setup is Ignis with NVFP4 and DFlash2 speculative decoding (7 draft tokens), 512K context via YaRN x2, KV at 9 KB/token.
Aggregate throughput is >800 tok/s with 8 lanes; image questions cost 60-90 ms each; locate found the right log line 21/23 times at median 2.6 s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 Flash at 1200+ t/s prefill and 40 t/s decode on a Strix Halo laptop.
Setup is a Strix Halo with 128GB unified memory; engine, quant and context length are not stated.
User says the model handles their pull requests and rates its quality as Opus 4.8 level, also mentioning some 27B runs.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next Q8_0 at ~52 tokens per second prompt processing and ~10 tokens per second generation on a V100 32GB.
Setup is llama.cpp with a 262144-token context, --cpu-moe and -ngl 64, on a machine with 128GB system RAM; the GPU drew 95.85 W at 48% utilization with 14206 MiB of its 32768 MiB in use.
Both throughput figures are given as approximate.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8-27B at 67.8 tok/s after a 250,000-token prompt on an RTX 3090 Ti.
Setup is the llamAmpere v0.4 fork of llama.cpp with an IQ4_XS GGUF, turbo5/turbo4 KV cache, 262,144-token context, and the model's MTP drafter at depth 4.
The run generated 5,120 tokens and peaked at 22,346 MiB of VRAM. The release tree decodes 104.28 tok/s on the v0.3.1 ship corpus and 103.09 tok/s at 100K KV depth.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8-Flash-Next on a Bosgame M5 (Strix Halo 128GB) across three backends. Gufo v0.7.0 with unsloth UD-Q4_K_XL and MTP reaches 54.0 t/s decode on code, 53.6 t/s on JSON, 29.5 t/s on prose, and 32.9 t/s at 19k context, with prefill 1124/1183/1179 t/s at 4k/19k/38k. strix-llama (ROCm) with ISTA GSQ-RCO IQ3_S and MTP gets 37.9/36.0/24.4/26.6 t/s decode and 734/798/794 t/s prefill; mainline llama.cpp (Vulkan) without MTP gets 27.5/27.5/27.3/25.5 t/s decode and 329/328/285 t/s prefill. GPU memory at 128k context is ~85 GiB for Gufo, ~62 GiB for strix-llama, ~57 GiB for mainline. The user notes Gufo is fastest but only accepts unsloth Q4_K_XL, while IQ3_S on strix-llama leaves room for Gemma 4 26B-A4B alongside. MTP did not work on mainline Vulkan. llama-bench on strix-llama with IQ3_S and no MTP gave pp2048 944 t/s (845 at 32k) and tg128 26.4.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 48.42 tok/s MTP decode and 1873 tok/s prefill on a BOSGAME M5 with Ryzen AI Max+ 395 / Radeon 8060S and 128GB RAM.
Setup is Halogen 0.16.2 running inside WSL2 on Windows 11, with 64 GB carve, native MTP depth 2, one slot and a 262144-token session capacity.
Serial decode without MTP is 36.72 tok/s. Prefill at 8192 tokens is 1873 tok/s, at 16384 tokens 1681 tok/s, at 128K 1620.70 tok/s and at 260K 1368.54 tok/s. MTP acceptance is 60% at 8192 tokens and 93.94% at 128K and 260K.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 35B at about 20 tokens per second on an RTX 5050 8GB with 32 GB of DDR4 RAM.
Setup is FreeToken; the model runs with system RAM involved since 35B does not fit in 8 GB of VRAM.
User asks whether switching to Strata would help, and states hardware upgrades are not an option.