Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Hermes with Qwen 3.8 27B fp8 at 262k context on a quad R9700 setup, with prefill at 17636 t/s and TG at 36.6 t/s.
Setup is vLLM-Radiance, with one GPU on PCIe 4x8 and three on 4x16.
Best TG reached 106 t/s with 80% MTP 4 acceptance. vLLM-Radiance works well with the quad setup despite official dual support.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen 3.8 Next at ~10.1 t/s on an RTX 3090 capped at 250W, with a Xeon e5 2690v4 and 96GB DDR4 RAM.
Setup replaces the N-gram layer with Q8 precision on top of an IQ4_XS quant.
The user reports no meaningful speed penalty from the change.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User compares MLX and GGUF for Qwen3.8 27B on an M5 Pro, reporting generation at about 19 t/s.
Setup is llama.cpp with Metal, which the user says now matches MLX prefill at roughly 300-350 t/s.
The user questions whether MLX is still needed and mentions MTP and oMLX.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports 25-29 t/s decode at short and mid context on 2x RTX 3090 over PCIe 3.0 with dual Xeon E5-2696 v4 and 188 GB usable DDR4-2133 LRDIMM, up from ~17 t/s before.
Setup is llama.cpp with the Qwen3.8-Flash-Next UD-Q6_K_XL GGUF, f16 KV cache, full 261,888 context, all 48 expert layers pinned in host RAM and everything else on GPUs, using PR #27861 GPU-resident LRU expert cache with 135 cache slots per layer.
Decode holds ~17 t/s at 131k depth, prefill is unchanged at ~350 t/s on a 26k prompt, and the cache hit rate is 80-85% on code. Dropping ubatch from 2048 to 512 freed ~5 GB per GPU, and prefill is slower on long prompts. Thread count, poll, CPU masks, q8 KV, lazy PLE, n-gram drafts on prose, MTP at temp 0.7, and more than 2 cache uploads per step did not help. Cache size is ~100 MB per slot per GPU on Q6, and the user plans to test UD-Q4_K_XL next.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8 27B IQ3_S at about 11 t/s for code generation and 6-7 t/s on reasoning.
Setup is speculative decoding with a Q4_K_M draft model.
User asks for optimization advice.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User reports running the pi coding agent with llama.cpp on Strix Halo, decoding at 26+ t/s and prefilling at ~200 t/s in the fast band.
Setup uses Qwen3.8-27B with the UD-Q4_K_XL quant, a DFlash2 drafter, f16 KV cache and 256k context. A Flash-Next model with IQ4_XS quant and MTP is also mentioned.
The KV cache hit rate is ~94%.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B in bf16 on Kaggle's free TPU v5e-8 at about 130 t/s generation with MTP and about 78 t/s without, plus 10,300 t/s prefill.
The full 262k context is supported, and the user serves an OpenAI-compatible endpoint through a Cloudflare tunnel.
User benchmarks Qwen3.8-27B FP8 on one RTX PRO 6000 in Paddock, an open-source inference engine in Rust/C++ with its own CUDA kernels, reaching 1062 tok/s at 32 clients against 958 for vLLM and 844 for SGLang.
Paddock is faster than vLLM in 13/13 cells, by 1.02x to 1.19x, and faster than llama.cpp Q8_0 in 13/13, by 1.5x to 37x.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports prefill around 2000 t/s and decode around 40 t/s for a single request.
With 2-4 parallel requests, prefill drops to 500-1000 t/s and decode to 10-20 t/s.
User asks for a better setup for 2-4 slots with q4 quants or better and q8 KV cache or better, and mentions trying vLLM recipes before.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 11 t/s on Qwen 3.8 with 2x RX 7900 XTX and 128 GB DDR4.
User compares the result to 3090s and 9700s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports that 2 GPUs outperform 3 due to a slow lane.
Setup is Qwen3.8 Flash Next with the Q4_K_M quant, variant AD-4.27bpw.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User compares Ollama/ROCm and llama.cpp/Vulkan on an RX 7900 XTX.
Ollama reaches 215.8 t/s prompt processing and 34.4 t/s generation at 64K context. llama.cpp/Vulkan reaches 192.0 t/s prompt and 35.8 t/s generation at 64K context, and 230.5 t/s prompt and 36.0 t/s generation at 8K context.
The user notes Vulkan is only about 4% faster for generation, while Ollama is better at prompt processing at 64K. The user wonders whether higher reported speeds of 60-100 t/s come from MTP/speculative decoding or other optimizations.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B Q4_K_XL on a Radeon AI PRO R9700 with llama.cpp at 29 TG/s, 76% of a theoretical 38 TG/s.
The figure is a theoretical maximum TG/s calculation for dense models. An RTX 5090 with 1.8 TB/s bandwidth is also mentioned.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B-NVFP4-MTP-GGUF at 210k context on 3x RTX 5060 Ti 16GB in a Dell 5820 workstation with 128GB DDR4 RAM, with prompt processing at 1300-700 t/s and generation at 29-71 t/s, mostly 38-44.
Setup is llama.cpp with a Q8 KV cache and vision enabled.
The user added a third GPU but reports cooling and stability issues, and is considering a second workstation with subagents for Mixture of Agents. The user also tested Ornith 1.5 35B (MXFP) and Gemma 4 26B NVFP on 16GB VRAM. Use cases are web hosting, office documents, and prompt generation for image and video models.
User reports Qwen3.8-27B running on a 16 GB card at 262,144 context.
Setup is buun's llama.cpp fork with the GSQ IQ3_XXS quant and a VBR KV cache.
The user describes the setup as intended for agentic use.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports about 46 t/s on typical short interactive decode in use on dual P40s, up from roughly 15 t/s without speculative decoding.
Setup is MTP speculative decoding with draft-mtp and ngram-simple, draft maximum 6, Qwen reasoning medium and the F16 vision projector loaded.
A synthetic 128-token decode reached 48 t/s, long-context interactive decode runs at about 20 t/s, and prefill is 444 t/s at pp512. The model is a fine-tune of Qwen3.8 27B.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User asks which GPU to buy to run Qwen 3.8 27B Q4_K_M at over 20 t/s.
Options under consideration are an RTX 3080 20GB, an AMD Mi50 32GB, and a modified 2080 Ti 22GB.
No benchmark results yet.
User compares NVFP4 against Q5_K_M on Qwen3.8-27B.
NVFP4 with adjusted sampling (temp=0.9, min_p=0.05) reaches 80% strict on IFBench, matching local BF16 and near official BF16 79.5%, while Q5_K_M stays at 76%.
NVFP4 runs about 3x faster and uses less VRAM. The Q5_K_M run uses llama.cpp.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8-27B FP8 on 2x R9700 with vLLM Radiance TP2.
User also tests MXFP4 at 111.4 t/s and Qwen3.8 Flash Next at 35.4 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash Next at 45 tok/s on an M4 Max and 25 tok/s on an M2 Ultra.
The quant is oQ4e with MTP.
User says Qwen3.8 Flash Next gives similar speed to Qwen3.8 27B on Apple Silicon.