Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8 27B FP8 on a quad RTX 5070 Ti 16GB node, reporting 82.21 t/s generation and 6488.44 t/s prompt processing at depth 0.
Setup is vLLM 0.30 on Ubuntu 24.04 with CUDA 13.2, tensor-parallel-size 4, 65536 max model length, 0.85 GPU memory utilization, and no speculative decoding. GPUs are connected through a PLX 88096 PCIe switch with P2P enabled within the node.
A second quad RTX 5060 Ti 16GB node reached 43.35 t/s generation and 3218.53 t/s prompt processing on the same model. Throughput scales roughly linearly with GPU tier, and the user notes results are stock with no overclock or undervolt.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 44.8 t/s generation and 414 t/s prompt processing on a 12GB RTX 5070 at 128K context.
Setup is a custom Strata inference engine with the IQ3_XXS GGUF quant, 64GB DDR5-5600 and a Ryzen 5 7600 on Windows. The engine is CUDA-only and requires 47GB minimum in RAM+VRAM for this quant.
A Q2_0 quant reached 65.1 t/s generation and 543 t/s prompt processing, and IQ2_XS reached 52.0 t/s generation and 472 t/s prompt processing. The user previously measured 15 t/s output and 100-120 t/s prompt processing with llama.cpp on the same hardware.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B Q4 at up to 70.56 t/s generation and 656.33 t/s prompt processing on a Strix Halo 128GB.
Setup is the Gufo inference engine, a new vertically integrated engine built specifically for Strix Halo hardware.
At concurrency 8 the model reaches 123 t/s aggregate. The engine also supports DeepSeek V4 Flash, Qwen3.8 Flash-Next, Qwen3 ASR and TTS, Qwen Image 2.1, and MiniMax H3.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agenticcodinglong-context
User reports Qwen3.8 27B uncensored at 175 t/s on an RTX 5090 32GB.
Setup is NInfer with NVFP4 / groupwise-int quantization and MTP enabled, running at 262,144 context with a Q4 KV cache.
The user says the Q4 KV cache matched Q8 quality and that the longer context enabled long reasoning tasks in a Hermes harness.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash Next Uncensored at 74.7 t/s decode and 3,600 t/s prefill at 64k context on 2x Radeon AI PRO R9700 with 128GB DDR5.
Setup is the R9V engine with an IQ4_XS GGUF plus MTP and a KVA projector; concurrent throughput is 70.3 t/s at 8 streams, TTFT P50 338 ms, update P99 51.5 ms.
KVA projectors give 1.45-1.85x prefill speedup (1700 to 2500-3150 t/s) for +2.6% to +8% perplexity depending on start layer; a multi-layer variant reaches 1.55x at +2% perplexity. RAM use floats around 50GB for expert storage.
User describes an open-source project (jevm) that replaces the lm_head of Qwen3.8-27B with ordinal cumulative-link heads on several layers, trained with LoRA, to produce five calibrated answers in one prefill for Chinese A-share news screening.
No model results are reported: only a smoke test on Qwen3-0.6B has been run, and the 27B is planned for training on 8× RTX PRO 6000. The repo contains label validation and baselines, including a text-blind GBM at 0.55–0.57 AUC on price reactions and 0.68–0.70 on volume/propagation.
A plain LoRA plus single-letter-answer baseline is served with vLLM logprobs for comparison.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 27B at 39.1 t/s average on an M1 Max 64GB, up from 18.9 t/s with stock Splash kernels.
Setup is a ported Inco Splash engine with custom Metal kernels for the M1, 4-bit weights, DFlash speculative decoding, and a 256K context. The 39.1 t/s is the average of npanj's five-prompt Splash benchmark at 250 tokens with default reasoning.
A real coding-agent session (opencode, 30K-89K context) ran at 19.5-36.5 t/s decode. Prompt processing was 142 t/s at 2K, 133 t/s at 8K, and 110 t/s at 32K. Four parallel requests reached 65 t/s total. The 35B MoE variant averaged 94.7 t/s with ~450 t/s prefill.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at around 225 t/s on a single CMP 170HX with 40GB VRAM at 132k context.
Setup is vLLM with W4A16 4-bit weights and an fp8 KV cache. The GPU is overclocked to NDIV 60, giving 1.89 TB/s memory bandwidth at around 1500 MHz, and has been stable for over a week at 65-67C.
The user also runs Minimax H3 for video generation on the same card and notes NDIV 62 crashes while NDIV 60 is stable.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 16.5 t/s generation and 350 t/s prompt processing on an RTX 4080 16GB with 64GB system RAM, at 131k context.
Setup is llama.cpp built from a pr-mtp-fix branch with the Unsloth MTP head as the draft model, AtomicChat AD-4.27bpw Q4_K_M target weights, and a Q8 KV cache. The --spec-draft-cpu-moe flag keeps draft experts in RAM so the target's hot experts fit in the 16GB of VRAM.
An IQ4_XS quant was also tested and reached about 10 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingtool-useagenticlong-context
User reports Qwen3.8 Flash Next Q4_K_XL at 66.14 t/s prompt processing and 22.09 t/s generation on an RTX 3060 12GB, Intel i7-14700 and 96GB DDR5.
Setup is ik_llama / llama-server on Windows 11 at 262,144 tokens of context with MTP3 and NCMOE48. A fresh OpenCode agent session produced a 10,890-token initial prompt, and MTP draft acceptance for that request was 89.5%.
After the initial ~11K-token prefill, a second short prompt completed in 1.6s with short-generation throughput around 22–26 t/s. At approximately 71K active tokens, generation was around 10 t/s, with an OpenCode session average around 12 t/s.
User benchmarks Qwen3.8 27B on an RX 7600 XT 16GB, completing 15/15 tasks with 1.000 correctness in 348.5 seconds.
Setup is llama.cpp HIP ROCm with the GSQ-RCO-IQ3_XXS GGUF and a Q4_0 KV cache at 114,688 context, fully in VRAM with no CPU offload.
The same benchmark also ran Ornith-1.5-9B (15/15, 0.983, 184.9s) and K2-Horizon-7B (11/15, 0.909, 498.1s); the user calls Qwen3.8 27B the undisputed winner for agentic coding.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User is running Qwen Flash Next at IQ4_XS with 262k context on a 96GB Strix Halo. Reports 50 PP and 14 decode near context limit. Seeking speed improvements. Considering V620 32GB via Oculink. Mentions 27B variant as possible alternative.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next (QFN) running on an AMD Strix Halo 128GB box, with 30-40 t/s generation and 900-1000 t/s prefill in real-world use over several days.
Setup uses the Halogen engine, which the user praises as the best current setup for this hardware and model.
The user compares QFN to a 27B model and finds them similar in capability, feeling that 27B already saturates their personal use cases.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports Qwen3.8 Flash-Next on a base M5 Ultra 96GB (64-core GPU) at a median 828.2 t/s prefill and 42.7 t/s decode per stream, with a single-stream max of 3,328 t/s prefill and 152 t/s decode.
Setup is a custom mlx-serve build with continuous batching at 4-way concurrency, a 4-bit/8-bit mixed MLX quant with MTP, 4 x 128k context (512k total) and 8-bit KV cache using 90GB of unified memory.
Aggregate throughput was ~3,200 t/s prefill and ~170.8 t/s decode at 4-way concurrency. The run covered 112M tokens (109M prompt, 3M generated) with an 89% cache hit rate across ~1,700 sub-agent calls.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-27B at 148.6 tok/s on a single RTX 4090 with MTP3 speculative decoding at 81.0% draft acceptance.
Setup is NInfer with the official groupwise artifact, INT8 KV cache, CUDA Graphs on, prefill-chunk 1024, single request greedy decoding.
Without speculation decode is 50.5 tok/s; at 128K context depth it is 39.6 tok/s. Prefill is 1,849 tok/s at 64K and 1,561 tok/s at 128K. The E8 4-bit KV default profile costs about 5.7% decode versus INT8, giving roughly 126.6 tok/s on the code probe.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
tool-usevisionagentic
User reports Qwen3.8 27B in PrismML's Q2_64 quantization running at ~140 tok/s in chat on an RTX 3090, using under 10 GB VRAM including KV cache and vision projector.
Setup is a fork of PrismML's llama.cpp with a custom /v1/systemone endpoint for typed decisions, tested with 22,000 requests at massive parallelism; p50 latency ~171 ms including HTTP network latency.
User also cites ~143 tok/s for chat completion on an RTX 5090, and benchmark results of 76% Soft Accuracy vs ~88% for Jev-1.13 on a 22,000-request typed-decision benchmark.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8 27B 4-bit on an M2 Max 96GB, comparing mlx-serve with MTP speculative decoding against LM Studio. With the MTP head correctly loaded, mlx-serve reaches 20.1 t/s decode on long prompts and 13.0 t/s on short, versus 11.2 and 8.0 t/s without speculation.
Setup is mlx-serve 26.9.5 with MLX 0.32.2 on macOS 26.6.2, using the mlx-community Qwen3.8-27B-MTP-4bit sidecar. Prefill measured 49/145 tok/s with MTP loaded and 47/112 tok/s without. LM Studio on the same weights reached 11.8 t/s decode short and 11.6 t/s long, with 59/101 tok/s prefill.
The user reports that mlx-serve silently falls back to Prompt Lookup Decoding when the MTP sidecar tensors lack the required mtp. prefix, and that MoE MTP heads fail to load entirely, so Qwen3.6-35B-A3B got no speedup. Numbers are client-side medians of 5 runs with unique prefixes to avoid prefix caching.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-Flash-Next at 47.1 t/s on an AMD Strix Halo 128GB, up from 16.8 t/s baseline, for file rewrite at about 8k context.
Setup is llama.cpp on ROCm 7.1 with the UD-IQ4_XS GGUF, using GPU TOP_K improvements, native MTP speculative decoding and ngram-mod augmentation.
New code at 8k context reached 31.7 t/s; at about 24k context, file rewrite reached 28.6 t/s and new code 25.4 t/s, against baselines of about 15 t/s.
User reports a native ninfer implementation of a Jev-like decision API running Qwen3.8 27B (Swift fine-tune) on an RTX 5090, scoring 84.4% (195/231) on the JevBench v1.2 public set with three in-flight decisions and a 19 s wall clock.
Setup drives the native POST /v1/decisions route directly rather than the chat-completions path; latency p50 is 127 ms and p95 is 699 ms, with hard-tier p95 prefill-dominated by ~3.7k-token states and tail latency inflated by a single serialized decision worker.
Calibration is ECE 0.045 and Brier 0.081; the user calls it a dirty POC with bugs and incorrect allocation still to fix, and notes latency is not great but accuracy is almost on par.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User benchmarks three Qwen3.8-Flash-Next REAP builds for Apple Silicon on an M4 Max 40-core GPU with 128 GB unified memory at 4K context.
Setup is oMLX with mixed-precision packages, 48 transformer layers, top-10 routing, a retained 512-expert native MTP predictor, and 262k configured context. The REAP-384 oQ5e build scored 97/100 HumanEval no-thinking and 61/100 LiveCodeBench no-thinking at 679.1 PP tok/s, 53.9 TG tok/s, and 72.8 GB peak memory.
The REAP-384 oQ4e build reached 689.5 PP tok/s and 59.3 TG tok/s at 61.5 GB peak, while REAP-288 oQ4e reached 692.3 PP tok/s and 48.7 TG tok/s at 46.0 GB peak. Accuracy figures are 100-question stratified samples and throughput is workload-specific.