Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User compares two llama.cpp setups on the same GPU: Q5 with a Q8 KV cache, 32K context, 4 slots and MTP off at about 35 t/s, against Q4 with a Q4 KV cache, 131K context, 1 slot and MTP on at about 65 t/s.
User notes the trade-offs between the two configurations.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 560/30 t/s on 2x RTX 3090.
Setup is Q8_0 quantization, run fully in VRAM.
The user calls it the local king and praises its quality. Other models mentioned are DeepSeek-V4-Flash, GLM-5.3-Flash, and Qwen3.8-Flash-Next.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
tool-useagentic
User reports Qwen 3.8 27B at 86 tok/s on a single RTX 3090 with vision enabled at 150k context.
The model is locally hosted and run through the DeepSeek harness (dsh).
The user also mentions writing a Gmail plugin.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports 30 t/s decode and 120 t/s prefill on a dual GPU setup.
Setup is llama.cpp with a PR build.
User asks for optimization advice.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
creative-writingcoding
User reports about 20 t/s on an RTX 3090 with 128 GB of DDR5, running Qwen3.8-Flash-Next.
Setup is a custom llama.cpp build from PR #27742, using the new qwen4exp architecture.
The figure is approximate and likely generation speed, and the run is a showcase of a generated 3D voxel demo rather than a formal benchmark.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User reports Qwen3.8-27B at ~70 tok/s on a single RTX 3090 with 24 GB, using 4-bit inference via llama.cpp.
The model is used for trading decisions, with QLoRA training and Ollama mentioned as a potential engine for distributed inference.
User is calling for contributors to an LLM trading experiment.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-context
User compares llama.cpp with a Q4_K_XL quant against vLLM with W4A16-AutoRound on an RTX 3090.
llama.cpp reaches about 45-50 t/s at 150K context, while vLLM reaches about 55-65 t/s at 175K context.
The user prefers vLLM for speed but notes llama.cpp is needed for vision, and asks about the quality difference between the quants for coding.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 55 t/s on 4x RTX 3090 with Qwen3.8-Flash-Next-UD.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User compares DeepSeek-V4-Flash and Qwen3.8-Flash-Next.
Setup is DeepSeek-V4-Flash lossless at 162 GB in Q8_K_XL on 4x RTX 3090 with 192 GB DDR4, reaching about 30 t/s and about 230 pp.
User notes Qwen3.8-Flash-Next has no lossless Q8 available and would need 360 GB, and criticizes benchmark misleadingness due to the size difference.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash Next at 15-19 t/s generation and 550 t/s prompt processing on dual RTX 3090s with a Threadripper Pro.
Setup is llama.cpp with speculative decoding using draft-mtp.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next (125B MoE + 51B n-gram table) at 32 t/s decode and 463 t/s prefill on 2xRTX 3090 + 96GB DDR5.
Setup is llama.cpp with experts offloaded to RAM and the n-gram table offloaded to NVMe, UD-Q4_K_XL, a 640K serving pool (4x160K lanes, q8 KV), and the 51B n-gram/PLE table mmap'd on NVMe. The MoE expert cache (RFC #24528 branch) at explicit --moe-cache 5000 gives +30% decode, while auto grants nothing. Concurrency erases the gain, so the user fans out shallow and serializes deep. Prefix caching reprocesses ~24 tokens on warm chat turns, with checkpoints anchored at turn boundaries and DeltaNet state included. The user found and fixed a gridDim.y overflow crash (deep multi-stream batches, QSA indexer norm) with a 2-line reorder, the same shape as the pending upstream fix in PR #27941. Serving holds 32 t/s while pulling 90GB via cgroup write-caging (systemd-run --scope -p MemoryMax=6G).
In vLLM, the user grafted the unmerged mmap-PLE PR (#54129) onto the qwen38-flash-next image, serving the table from NVMe under W4A16, TP=2, in 53GB where the official recipe wants ≥110GB free RAM. Real-table decode is 15.3 t/s vs 16.0 with the table stubbed, so the NVMe n-gram costs ~4%. Selective experts-only UVA offload is mandatory: naive --cpu-offload-gb recopies the whole offload every step (2.5 t/s), while --cpu-offload-params routed_experts.*_weight_packed gives 16 t/s. MTP speculation hurts when the PCIe bus is the bottleneck (13.9 vs 16.0). vLLM is currently being optimised and tested for concurrency, and single stream is behind llama.cpp right now.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 22 t/s with the Q6_K_XL quant on 4x RTX 3090, compared to 47 t/s for Q4_K_XL.
Setup is llama-server with --split-mode layer, --kv-unified, --flash-attn on, and speculative decoding (ngram-mod). Context is limited to 200k to avoid OOM, with no KV quantization.
User mentions MTP for Qwen 3.8 Flash Next may land upstream, and prefers 27B_Q8 as a daily driver.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports 59 t/s in a one-shot full software run and up to 70 t/s in testing on an existing codebase.
Setup uses an asymmetric Q8 KV cache quant and speculative decoding with an MTP draft at 205k context, with --ctx-size 205000, --parallel 1, --kv-unified, --flash-attn on, -ctk q8_0, -ctv q4_0, --temp 0.6, --top-p 0.95, --top-k 20, --min-p 0.0, --spec-type draft-mtp, --spec-draft-n-max 4, --spec-draft-p-min 0.7, --batch-size 512, --ubatch-size 512, --reasoning-effort xhigh, --reasoning-format auto, --fit off, --cache-prompt, -ngl 99 and --no-mmproj.
The user notes this is faster than mtp2, which had 120k context and 41 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports a custom kernel achieving fp32 quality with 0.99997 similarity at int8.
Setup includes optimized prefill and decode speeds.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen 3.8 Next at ~10.1 t/s on an RTX 3090 capped at 250W, with a Xeon e5 2690v4 and 96GB DDR4 RAM.
Setup replaces the N-gram layer with Q8 precision on top of an IQ4_XS quant.
The user reports no meaningful speed penalty from the change.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports 25-29 t/s decode at short and mid context on 2x RTX 3090 over PCIe 3.0 with dual Xeon E5-2696 v4 and 188 GB usable DDR4-2133 LRDIMM, up from ~17 t/s before.
Setup is llama.cpp with the Qwen3.8-Flash-Next UD-Q6_K_XL GGUF, f16 KV cache, full 261,888 context, all 48 expert layers pinned in host RAM and everything else on GPUs, using PR #27861 GPU-resident LRU expert cache with 135 cache slots per layer.
Decode holds ~17 t/s at 131k depth, prefill is unchanged at ~350 t/s on a 26k prompt, and the cache hit rate is 80-85% on code. Dropping ubatch from 2048 to 512 freed ~5 GB per GPU, and prefill is slower on long prompts. Thread count, poll, CPU masks, q8 KV, lazy PLE, n-gram drafts on prose, MTP at temp 0.7, and more than 2 cache uploads per step did not help. Cache size is ~100 MB per slot per GPU on Q6, and the user plans to test UD-Q4_K_XL next.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User asks for config optimization advice, reporting prompt processing at ~189 t/s and generation at ~15.8 t/s.
The setup splits a MoE model with 42 expert layers on CPU.
User reports Qwen3.8 27B UD Q4_K_XL fits in 24 GB of VRAM at 100k context with a Q8 KV cache.
User runs Qwen3.8-Flash-Next Q4 on dual RTX 3090s and is considering larger models such as MiniMax-M3 for agentic coding.
User calls MiniMax-M2.7 a workhorse but too slow for coding.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports decode improving from 25-29 to 37-41 t/s with the UD-Q4_K_XL quant, expert cache, and MTP.
Prefill runs 180-195 t/s, load time is 2 min, and cache hit rate is 90-92%.
MTP acceptance is 50-58% on reasoning and 94% on code emission. Host RAM is 73 GB pinned plus 28 GB PLE. The branch is available.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-next at 160 t/s prefill and 16 t/s decode.
Setup uses a kvarn5 KV cache quant, with experts offloaded to host RAM and n-grams on disk. MTP works but slows decode.
User also mentions alternative setups for 16 GB VRAM with kvarn4 and vision offloaded, and for 12 GB with ub 512.