Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports a performance issue with Qwen3.8 27B on an M3 Max 48GB, comparing mlx-serve against LM Studio. The mlx-serve run used the ddalcu/Qwen3.8-27B-MLX-Serve-4bit model at 128k context and achieved 59.6 tok/s decode (54.9-60.2) and 196.4 tok/s prefill, while LM Studio with the Swift-1.5-Qwen3.8-27B-Splash model reached 96.4 tok/s decode (89.1-96.8) and 210.5 tok/s prefill.
Both runs used speculative decoding (MTP/draft active). The mlx-serve run showed a 42.1% slowdown under sustained load (59.6 to 34.5 tok/s over 8m 4s), and context scaling was worse than LM Studio at longer contexts (37.2 vs 76.5 tok/s at ~4.2k, 35.6 vs 60.5 at ~8.2k, 34.6 vs 57.6 at ~16.3k).
The user reports mlx-serve is 38% slower on decode overall and up to 51% slower at ~4.2k context, while being 18% faster on TTFT.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B UD-Q4_K_M at 57.25 t/s decode (depth 0) on a single Tesla V100-PCIE-32GB.
Setup is llama.cpp with llama-split-bench --profile, MTP enabled, measured up to 262k context. Prefill is about 720 t/s on new prompts.
Decode falls to 20.49 t/s at depth ~260k. An initial run suffered thermal throttling (15.68 t/s at 64k depth) due to BMC fan control; values shown are with fan mode set to Full.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-context
User reports a third-party sourced prefill measurement of Qwen3.8-Flash-Next at 1057.53 t/s on a single Strix Halo 128GB system.
Setup is Gufo with Unsloth UD-Q4_K_XL GGUF weights, a 256000-token prompt with zero cached prefix tokens, 512-token prefill chunks, and MTP enabled with a shared Q8 MTP head and draft ceiling 7.
The row is prefill-only and generates one setup token, so no generation throughput is reported. One observation per cell on a synthetic counting workload; no repeats, uncertainty interval, TTFT or row-level memory peak are available.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 34.19 t/s decode and 1541.16 t/s prefill on two AMD Strix Halo 128GB hosts, one Radeon 8060S gfx1151 GPU per host, tensor parallel world size 2 over RDMA/InfiniBand.
Setup is the Gufo RDMA fork with Unsloth UD-Q4_K_XL GGUF (four shards), configured context capacity 262144 tokens, greedy temperature 0, thinking off, speculative decoding and MTP off; the measured input was 2040 tokens, not 262144.
Single warmed prose sample, one session, HTTP OpenAI chat-completions SSE; source-maintainer measurement, not independently replicated, with no standard deviation. KV-cache precision, Flash Attention setting, measured resident RAM and verified binary identity are unknown; the prose labels revision ea76571 while the raw artifact records 795407e30460aeb730aaff2316739baa3a1348e2 with dirty=true.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticcodingtool-use
User reports Qwen3.8-Flash-Next at 125 t/s generation and 5,879 t/s prefill on an RTX 5090 with the Infernix engine.
Setup uses NVIDIA's NVFP4 quant with expert streaming from system RAM and SSD, n-gram drafting, and PCIe Gen5 x8. The engine is an offshoot of NInfer.
Compared to Strata running Unsloth's UD-Q4_K_XL quant, Infernix shows 82% lower average time to first token (5.9 s vs 32.9 s) and 65% lower workload wall time (7.8 min vs 22.3 min). At concurrency=2, output reaches 143 t/s. The user notes the model is broadly on-par quality-wise with the Unsloth quant and that PCIe Gen5 x16 may yield higher performance.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8 Flash-Next at 121 t/s generation and 630 t/s prefill on two CMP 170HX 8GB cards.
Setup is Strata with IQ3_XXS GGUF weights, int8 KV cache, MTP speculative decoding with 4 tokens, and a 262144 max context.
The table covers prompt lengths from ~0.5K to 140K and concurrency 1 to 4; the reported figures are the c1 ~0.5K row. Total output tok/s at c1 ~0.5K is 89.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentictool-use
User reports Qwen3.8 Flash-Next at 100 tok/s on a CMP 170HX 64GB and one RTX 3090, with system RAM offload.
Setup is Strata with Q4_K_XL weights; the run used two GPUs plus system RAM, and the user notes about a quarter of the results came from Strata .38, an eighth from .39, and the rest from .40.
Before Strata, the same model ran on three GPUs (CMP 170HX and two RTX 3090s) under Unsloth Studio at 39 tok/s with 84 s average TTFT; the Strata run cut average TTFT to 15.7 s and total LLM time from 5,795m 9s to 1,228m 53s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports daily driving Qwen 3.8 Flash-Next MoE (uncensored NVFP4) on an RTX 5090 with 136 GB DDR5 RAM, seeing 135-153 tok/s decode over 8 hours of agentic coding.
Setup is Strata (NVFP4 fork) with NVFP4 MoE weights, BF16 embeddings, FP8 PLE and INT8 dynamic paged KV cache at up to 262k context; 22.3 GB of expert slots held in VRAM with the remainder paged from a 64 GB host RAM arena.
Decode stayed flat from 50k to 212k context (r = +0.005), with 134.5 tok/s at 52k, 146.7 tok/s at 205k and 141.2 tok/s at 212k; MTP speculative decoding had a 79.60% acceptance rate and the MoE VRAM cache hit rate averaged 89.9%. Over 2,411 requests the KV cache hit rate was 99.31%.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Strata 0.1.38 on an RTX 4090 48GB with 128GB DDR5-6000, running Qwen3.8-Flash-Next at IQ3_S with 262K context, int8 KV cache and speculative decoding. Baseline decode is 128-151 tok/s and prefill 4,249-5,154 tok/s at 78K-92K context.
Keeping the 28.8 GB n-gram table in RAM gave only +0.65% prefill and +1.2% decode for +28.4 GiB RAM, while conversation parking returned to a 91,836-token conversation in 537 ms instead of 18.8 s for 3.1 GB of RAM. Raising --prefill auto to 32768 gave +9.7% prefill and +13.6% decode at a 78.7K prompt, and --calibrate picked 12 CPU workers for +3.2%.
User notes session-to-session spread of 11.5% on unchanged settings, that mmap made the first long prompt 3.7x slower, and that minimizing the engine console on Windows 11 cost 20% decode.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a repository of launcher scripts for running local LLMs on the 128 GB Jetson AGX Thor Developer Kit, with vLLM or NVIDIA's Thor-optimized llama.cpp container.
The default profile is Qwen3.8-27B NVFP4 with DFlash2 speculative decoding, a 30.3 GB download, listed at a published 27.69-34.44 tok/s on Thor.
Other profiles include Qwen3.6-35B-A3B NVFP4 + DFlash at 116.5 average and 139.1 peak tok/s, Nemotron Nano 9B v2 NVFP4 at 30 tok/s, Nemotron 3.5 Lightning NVFP4 + DSpark at 123.01-138.02 tok/s, Gemma 4 26B-A4B NVFP4 + MTP at 50 tok/s c1 and 180 tok/s aggregate c8, and Muse Glimmer 30B K-Quant + DFlash at up to 36 tok/s. The GLM-4.7-Flash and GLM-4.5-Air profiles are experimental with no published Thor speeds. The speeds come from different workloads and are not a controlled comparison.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticlong-context
User asks for hardware upgrade advice for running local LLMs, currently using an RX 7900 XTX with a Ryzen 5600X and 64GB DDR4. They report running Qwen3.8 27B at Q4 with 128k to 220k context and Q8 KV cache, getting up to 50 t/s generation and around 750 t/s prompt ingestion. They are considering 2-4 MI50 16GB cards or a second 7900 XTX, and ask about motherboard recommendations.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next decode at 47.6 t/s on Strix Halo 128GB, with prefill of 994 t/s at 2K context.
Setup is a custom Windows attempt with speculative decoding; prefill reaches 1,026 t/s at 8K and 1,015 t/s at 32K, and decode without speculation is 34.0 t/s short, 30.7 t/s at 32K and 28.6 t/s at 128K.
User compares against Linux and Windows results from Halogen, gufo, strix-alloy and strixllama, and reports 30/30 on a 68K truncation test. No repo shared yet.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 42.0 t/s generation and 887 t/s prefill on a GMKtec EVO-X2 with AMD Ryzen AI Max+ 395 (Strix Halo, 128GB unified, 64GB GPU carve-out).
Setup is Strata 0.1.40.2 (HIP/ROCm) with Unsloth UD-IQ4_XS GGUF, int8 KV cache, 32,768 resident KV tokens, --spec 4 with MTP enabled, and a fresh 17,844-token prompt generating 300 tokens.
Single-machine community self-report, not independently reproduced; MTP/speculative decoding is on, so this is not a non-speculative baseline. Chip power was about 121 W (peaks 138 W) and GPU clock median 2.75 GHz.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B-Q4 at over 31.8 t/s baseline on one R9700 in a Razer Core X V2 over TB5 connected to a MacBook Pro M5 Max.
Setup is the LSE engine with Q4 quant. Around 55 t/s with MTP=3, 54 to 80+ t/s with DFlash2 depending on context and text, and over 1400 PP/s on 1K+ token prompts.
DFlash2 reaches over 80 t/s on HumanEval+. The user notes HumanEval+ reports predate the prefill optimization.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingchat
User reports Qwen3.8 27B Bare Speed at 65.2 tok/s on an M5 Max MacBook Pro.
Setup is MTPLX with a flat 4-bit MLX quant and MTP speculative decoding at depth 3, single stream, 16.0 GB download and 17.0 GB peak unified memory.
A 52,740-token answer sustained 32.4 tok/s over 27.2 minutes; long reasoning at xhigh ran 35.7 and 32.0 tok/s. The same task in the MTPLX Mac app ran 64.4 tok/s, oMLX 0.5.7 ran 63.3 tok/s and LM Studio ran 17.40 tok/s on the long answer. KL divergence to the bf16 checkpoint is 0.0376.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen 3.8 27B at 208.1 tok/s average on a Radeon AI PRO R9700, with a 227.8 tok/s peak request.
Setup is the Lucebox engine with UD-IQ4_XS GGUF weights and a DFlash2 Q8_0 drafter at block size 16, q8_0 KV cache, single card.
The figure is a HumanEval average; the repository also lists 110.6 tok/s for Qwen 3.8 27B on an RTX 5090 and 79.7 tok/s on 2x RTX 3090 with NVLink.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 21.05 tokens/s engine-reported output decode on a Strix Halo 128GB (Ryzen AI MAX+ 395 / Radeon 8060S).
Setup is Gufo with UD-Q4_K_XL GGUF weights, configured context capacity 131584 tokens, one slot, MTP off, thinking off, temperature 0.
The run read an actual input of 127525 tokens and produced 420 output tokens; reported prefill duration was 93.24 seconds and whole-request wall time 113.47 seconds, with no derived prefill rate submitted because raw counter semantics were not independently verified. One cold sample; cached repeats excluded.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingcreative-writing
User reports a fork of the Splash engine, called Splish, optimised for the M5 Max. On a 40-core M5 Max with 4-bit models, Swift-1.5 went from 74.7 to 89.8 tok/s at a single request with tuned kernels.
Setup uses the Splash engine with 4-bit models and kernel choices tuned for the 40-core M5 Max. The fork is roughly 1.25x faster at a single request and up to 1.5x faster at 2-4 requests.
In real-world short story prompts, speed went from about 45-51 tok/s to 56-64 tok/s. Qwen3.6-35B-A3B saw speedups of +5% / +18% / +22% / +18% at 1-4 requests. Whole-file edits for coding agents improved by +24% to +42%.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports Qwen3.8-Flash-Next at 83 tok/s on code and 45-50 tok/s on prose on a single RX 7900 XTX 24GB with 60GB DDR5.
Setup is TabbyAPI with EXL3 3.05bpw, 128K context and 8-bit KV cache, 166 of 512 experts per layer on the GPU and 12 CPU threads, MTP drafting with 2 tokens.
A 28.7K-token code prompt prefills at 505 tok/s cold and decodes at 51-70 tok/s; without drafting the model runs 53 tok/s on code. The 4.05bpw variant reaches 68 tok/s on code. llama.cpp with IQ4_XS and --n-cpu-moe 36 is cited as a reference at 17.5 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks llama.cpp CUDA patches on two Tesla P100-PCIE-16GB cards, reporting Qwen3.8-27B Q4_K_XL at 24.5 tok/s decode with -sm tensor, up from 22.3 tok/s stock.
Setup is llama.cpp b10660 with patches 0001-0012, CUDA 12.4, F16 KV cache, tensor split across two cards.
The README also reports Llama-3.1-8B Q4_K_M at 79.57 tok/s on one card (2.03x stock), Qwen3.8-27B pp8192 at 378.7 tok/s, and a real 74,919-token request at 20.92 tok/s with F16 KV versus 16.85 with q8_0. Four concurrent agents reach 20.3 tok/s each (72.9 aggregate).