Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports the AtomicChat build of Qwen3.8-Flash-Next running a full 256K context on 96 GB RAM and 40 GB VRAM, at 22 t/s.
The build splits the n-gram table to SSD. The user previously ran unsloth quants, which crashed at around 60K context and were slower at 12-14 t/s.
The AtomicChat build handles a 217K token conversation without crashing. Prefill takes about 18 minutes for 220K context.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports 59 t/s in a one-shot full software run and up to 70 t/s in testing on an existing codebase.
Setup uses an asymmetric Q8 KV cache quant and speculative decoding with an MTP draft at 205k context, with --ctx-size 205000, --parallel 1, --kv-unified, --flash-attn on, -ctk q8_0, -ctv q4_0, --temp 0.6, --top-p 0.95, --top-k 20, --min-p 0.0, --spec-type draft-mtp, --spec-draft-n-max 4, --spec-draft-p-min 0.7, --batch-size 512, --ubatch-size 512, --reasoning-effort xhigh, --reasoning-format auto, --fit off, --cache-prompt, -ngl 99 and --no-mmproj.
The user notes this is faster than mtp2, which had 120k context and 41 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports running a 182B model on an RTX 4080 with 64GB RAM by offloading ngrams to SSD.
User claims it is faster and more intelligent than Qwen3.8 27B and provides the launch command.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports running a model on a 12 GB mid-range Android phone with optimizations and low quantization on the dense part.
The model size is given as 80 GB, likely the total including quantized weights.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingtool-useagentic
User reports 22-30 t/s generation at 140k context.
Setup is llama-server via llama-swap with a UD quant, an MTP draft with spec-draft-p-min 0.70, and a q8_0 KV cache, using about 22.1 GB of VRAM.
The user runs it for coding with opencode.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User compares Flash-Next against a dense 27B model.
Flash-Next is faster and mechanically flawless, with zero failures on strict JSON, injection resistance and SLAs, and wins the high-reasoning spatial and code-gen tier with better failure modes. The dense 27B still wins sustained multi-step symbolic work such as bug-fixing, math proofs and abstract puzzles.
On that symbolic work Flash-Next shows a new failure shape: it promises the deliverable, declares "done", and outputs nothing. Both models use the same reasoning_effort knob with radically different semantics, so Flash-Next is not a drop-in replacement but a conditional promotion.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
visionlong-context
User reports custom kernels for RDNA4 giving roughly 3x generation speedup and roughly 30x prefill speedup over public vLLM-Radiance.
Setup is dual R9700s with 128GB RAM.
User also mentions work on the Muse Glimmer 30B dense model.
3× RTX PRO 6000 64GB DDR5, RTX 5090 64GB DDR5, MacBook Pro 48GB · llama.cpp
generation:
6.4 tokens/s
prompt processing (prefill):
16.5 tokens/s
quant:
BF16
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports full BF16 at 330 GiB with no quantization, running stock llama.cpp master b10680 over RPC on a LAN across 4 devices on 3 boxes: 94 GiB on the RTX PRO, 40 on the Mac's Metal, 30 on the 5090, plus the 5090's DDR5 as a 4th RPC device.
About 39 GiB still spills to local RTX PRO RAM, as the device pool cannot hold all 234 GiB of non-PLE weights.
The user notes a reported single 2×3090 box at 32 t/s on this model at UD-Q4_K_XL, which is 1.5× better byte-for-byte than what the user is achieving, so it is not purely a bytes problem. The next step is GGML_SCHED_DEBUG=2 plus GGML_RPC_STATS to see how much of the 155 ms token is RPC round-trips versus actual work.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 51.56 tokens per second over 5369 generated tokens with MTP on 2x R9700, writing a binary tree in javascript.
Setup is llama.cpp built in the ROCm 10 docker image with unsloth's Q8_K_XL GGUF, tensor split across both cards, 255000 context, f16 KV cache and MTP drafting up to 3 tokens, using 27GB on each card.
Draft acceptance is 0.72147. The user sees 37-50 tg/s generating text, spiking over 60tg/s when writing code.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports hybrid quantization to fit MTP and long context in 16 GB VRAM.
Setup uses MTP draft with spec-type draft-mtp and spec-draft-n-max 3, at 85,000 context, with q4_0 KV cache quantization for both cache types.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Next-Flash at ~240 t/s decode on a single RTX Pro 6000 Max-Q at 300W.
Setup is a patched sglang with the RadixArk/Qwen3.8-Flash-Next-NVFP4 checkpoint, using FP8 quantization of the lm head and layers, kernel tuning, and MTP config tuning.
The user notes a theoretical bandwidth limit of ~280 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8-27B MTP quants on an Apple M5 Max.
The oQ4e-mtp quant reaches 36.6 t/s with a quality score of 86.8.
User calls oQ4e-mtp the best balance of speed and quality and recommends it as the sweet spot.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 7-8 t/s with 2x3060 and RAM offloading, but 4.5-5.5 t/s with 3x3060.
User asks why adding a GPU slows down generation, and notes there is no tensor parallelism in both chats.
User asks about enabling ngram or mlock.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User asks about adding RAM to run a larger quant of Qwen 3.8.
User mentions Q4_K_XS ran fine at 16 tps.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8 Flash Next across VRAM tiers from 8 GB to 96 GB with llama.cpp b10666. CPU-only decode reaches 8.34 tok/s and full 96 GB decode reaches 109.07 tok/s. At 245K context, decode ranges from 14.89 tok/s on 24 GB to 21.61 tok/s on 96 GB.
The model activates 6B params per token (MoE). RAM-resident loading gives 1.87x prefill versus mmap. Non-unified KV reaches 92.0 tok/s total at concurrency 16.
A PLE table on CUDA causes a 55.6x decode slowdown, from 108.5 to 1.95 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B-UD-IQ3_XXS.gguf at 40.68 t/s generation and 778.91 t/s prompt processing at 131,072 context, against 13.36 t/s and 137.87 t/s at 65,536 context.
Setup is llama-server with flash-attn, a q8_0 KV cache, Vulkan0, and speculative decoding with draft-mtp and ngram-mod.
User asks why the smaller context degrades performance.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks TensorSharp against llama.cpp on Qwen 3.8 Flash Next, reporting prompt processing and generation speeds for 1 GPU and 2 GPU layer split configurations.
The promptTps and generationTps fields capture the 1 GPU pp512 and tg64 values respectively.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen 3.8 27B Q4_K_M running on an RTX 4070 Ti Super 12GB and a MacBook M5 Air over RPC with llama.cpp, completing a coding game clone task in 117 minutes.
Setup uses llama.cpp over RPC between the two machines. LM Studio is also mentioned.
The 117 minutes is the total time for the task, not a throughput figure.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports a custom kernel achieving fp32 quality with 0.99997 similarity at int8.
Setup includes optimized prefill and decode speeds.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports a fork optimized for AMD gfx906 (MI50/MI60/Radeon VII) reaching 379.2 t/s prefill versus 332.3 t/s (+14.1%), 252.6 t/s versus 231.1 t/s on a 120k-context fill (+9.3%), and 13.6 t/s versus 13.5 t/s deep-context TG.
DFlash acceptance is 0.691.