Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agenticcoding
User reports Qwen3.8 27B at 144 t/s on an M5 Max MacBook Pro with 36 GB.
Setup is the Inco Splash inference engine, an open-source engine built for Apple silicon, also available as a runtime inside LM Studio. Requirements are M3 or newer, macOS 26.4+, and 36 GB.
The engine is claimed to reach up to 3x the decode speed of Ollama, 2x oMLX, and almost 4x when an agent fans out into sub-agents.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
visionlong-contextsummarization
User reports Qwen3.8 Flash-Next at 5.5 t/s generation and 118 t/s prompt eval at 131,072 context on an RTX A2000 12GB with a Xeon Silver 4314 and 128 GB DDR4.
Setup is llama.cpp master with the unsloth UD-Q4_K_XL GGUF (103.7 GiB), q8_0 KV cache, --n-cpu-moe 99 and -ub 1536, all routed experts in RAM, 11.6 GB VRAM peak.
Generation falls from about 15.4 t/s empty to 8.8 t/s at 58k and 5.5 t/s at 118k. A 3000x2000 image request took 46 s. The same box also ran Qwen3.6-35B-A3B at 26.1 t/s and DeepSeek-V4-Flash at 7.5 t/s at 118k deep.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next Q4 at 7-8 t/s generation and 110 t/s prompt processing on dual RTX 3060 cards.
No engine, context length, or KV cache quant is stated.
User asks whether a used RTX 4070 or RTX 5060 Ti 16GB paired with a 3060 could match RTX 3090 generation speed, and says the current 7-8 t/s is not enough.
User reports Qwen3.8 27B Swift IQ4_XS on an RTX 3090 at 200k context, with 8/TQ3 KV cache.
Setup is LlamAmpere v0.3.1 with custom kernels for EXL and ternary quants plus MTP heads. The release adds support for Turboderp's EXL format and Bonsai ternary 1.75/2bpw quants.
MTP gave about 20% speed up on 1.75bpw and 50%+ on 2bpw in initial 3090 testing. User also tested Qwen Swift on Livecode with xhigh reasoning, performing within 1.5% of original Qwen BF16.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8 Flash-Next at about 35 t/s on a single DGX Spark.
Setup is SGLang with an NVFP4 quant at 262,144 context, used through VSCode Copilot in autopilot mode.
The session produced about 10k lines of code and consumed about 800k tokens over 8 hours of planning, coding, and testing.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 Flash-Next at ~27.6 t/s generation and ~367 t/s prompt processing on an M5 Pro 64GB, with the 95.5 GiB model streamed from SSD.
Setup is a llama.cpp fork with expert streaming, a draft head, depth 3 with a 0.3 confidence floor, sparse attention, and Metal MoE fusion, using a bartowski Q4_0 checkpoint with Q8_0 output layer and unsloth UD-IQ4_XS spliced in.
Generation drops to 18-18.6 t/s with the draft head off and ~20.6 t/s at 29k context in real chat use. Paired 40-chunk perplexity improved from 5.2777 to 4.3148.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User reports Qwen3.8-Flash-Next at ~12 t/s on 2× RTX 3060 12GB at 131,072 context.
Setup is the AtomicChat AD-3.84bpw-IQ4_XS-M64 GGUF (~85GB) with CPU MoE offload, most expert weights partially offloaded to system RAM.
The goal was a quality-first local agent / batch worker rather than maximum chat latency; a 26.5k-token agent prompt was used.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agentictool-uselong-context
User reports Qwen3.8 27B at 57.7 t/s decode on two RTX 3080 20GB cards (40GB pool) at 160k context.
Setup is vLLM in Docker on Unraid with the AWQ-INT4 quant, tensor-parallel-size 2, max-num-seqs 1, max-num-batched-tokens 16384, and GPUs throttled to 250W. Prefill is roughly 2,500 t/s at 10k context with 4.23s TTFT.
Decode drops to about 51 t/s at 42k context and 43 t/s at 110k context; lifetime average across 611 agent requests is 47 t/s. User asks whether speeds can be improved.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B on a V100 at 433.1 t/s prompt processing and 23.19 t/s generation with a Volta-optimized llama.cpp fork.
Benchmark uses 100k cached tokens plus 1k new prompt tokens and 64 generated tokens; the fork improves the FlashAttention path for roughly 40% higher prompt processing.
Upstream llama.cpp gave 306.7 t/s prompt and 23.29 t/s generation on the same setup. The user notes the gain is mostly limited to Qwen-like model shapes and also tested Qwen3.5 122B A10B, Laguna-S-2.1, and GLM-5.2.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 generation dropping from around 10 t/s to 2.8 t/s after 8 hours on a Rust coding task.
Setup is llama.cpp compiled in f16 mode with the maximum context size of 262k tokens, restricted to a single GPU, driven through opencode.
User asks why throughput degrades over a long session and whether it can be fixed.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 25.1 t/s on 4 GPUs at 128k context without speculative decoding.
Setup is llama.cpp with -sm layer across 4 GPUs. Enabling ngram-mod alone drops to 23.3 t/s (−7.1%), draft-mtp alone drops to 22.5 t/s (−10.3%), and MTP plus ngram drops to 20.6 t/s (−17.9%).
MTP acceptance is high at 82.9% but mean accepted length is only 2.8 tokens, so the draft head's expensive MoE forward per candidate is not amortized. ngram-mod has worse acceptance at 23.9% but longer drafts at 5.0 tokens; both spec implementations hurt at 128k.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agentic
User reports Qwen3.8-Flash-Next UD-Q3_K_XL at 43.89 t/s prefill and 7.78 t/s generation on a CPU-only Intel Core Ultra 225 with 64 GB DDR5-6200.
Setup is llama.cpp with N-GRAM SSD offload, RAM pinning, IQ4 expert kernel, AVX2 hyperconnection fusion, and MTP with draft length 3.
Generation reached 11.30 t/s on Pi-Agent workloads; prefill improved from 23.61 to 43.89 t/s and generation from 5.42 to 7.78 t/s through the optimizations.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at about 30 t/s on an RTX A4000 VM.
The model is the DavidAU TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP GGUF. The VM costs $0.15/hour and is shared with a friend; the user offers to move to an RTX A6000 if more people join.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at ~35 t/s decode and ~475 t/s prefill on an RTX 5070 Ti Laptop 12GB with 60 GB system RAM.
Setup is the Flyweight engine with an IQ1_S GGUF, running MoE experts on CPU or split with a hot set cached on the GPU.
Other measurements on the same laptop: Qwen3.8 27B IQ2_XXS at ~40 t/s, DeepSeek-V4-Flash at 6-7 t/s, and Z-Image 1024x1024 in ~15 s with a 35B model loaded alongside.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 16.9 t/s on a single MI50 16GB with the gfx906-16gb-expert-pool llama.cpp fork.
Setup uses the moe-expert-pool branch with IQ4 GGUF weights, 128K context and Q8 KV cache; the expert cache profile of 66 admits 144 pools at roughly 69% hit rate and peaks at 15.24 GiB VRAM.
A stock CPU MoE path without the expert cache reached 11.76 t/s, and up to 19.8 t/s was observed once warm.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
visioncodinglong-context
User reports Qwen3.8-27B (text + vision) as an 11.8 GB GGUF at 3.47 bpw, running at 42.3 tok/s single stream on an RTX 3090, matching Q4_K_M on the same card.
Setup is a patched llama.cpp build (13 files, 471 insertions on llama.cpp 6a1a922d) with the Opti-27B GGUF and mmproj, 16,384 context, full GPU offload. Prefill at 7k tokens is 1,127 tok/s; three concurrent streams give about 24 tok/s each.
Wikitext-2 perplexity is 6.487 versus 6.456 for FP16 and 6.457 for Q4_K_M at 16.8 GB. Four 16k conversations with vision fit in 17 GB; one 65k conversation with vision fits in 15 GB. MMLU-Pro is 53.9 +/- 0.5.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 Q8 at 82 t/s decode on dual RX 7900 XTX cards at 60k context.
Setup uses a fork of llama.cpp optimized for RDNA3 7900 XTX, replacing vanilla llama.cpp with Vulkan, which gave about 28 t/s.
The user notes the work is not theirs and shares the repo so others can push it further.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at around 10 t/s on an RTX 4070 Ti Super in a Threadripper 3975WX system with 256 GB RAM.
Setup is llama.cpp server with the Unsloth Q5 GGUF, run with only --ctx-size and --jinja flags.
User asks whether 10 t/s is the best achievable at this quant without a better GPU.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 27B at ~980 t/s prefill on an M5 Max, and DeepSeek V4 Flash 0731 at ~900 t/s prefill.
Setup is the Draw Things inference engine, described as 1.2x to 1.6x faster on prefill than prior work for supported models.
The engine runs in a sandboxed macOS App Store package with Python, Node.js, Lean, and Bash tools, and can run multiple tasks concurrently on the same laptop.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingcreative-writing
User reports Qwen3.8 Flash-Next at 70 t/s generation and 1.3k t/s prefill on 4x V620 (128GB total VRAM) at 128k+ context.
Setup is a vLLM fork with AutoRound W4A16 quantization and MTP-2 speculative decoding, on an EPYC 7452 with 256GB DDR4. Prose generation runs at 60 t/s.
Power draw is 700-900W during prefill and 500-600W during decode. User was initially disappointed with Qwen3.8 27B speeds but is satisfied with Flash-Next.