Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 27B at 38.4 t/s and Qwen3.6 35B-A3B (MoE) at 55.9 t/s on an RTX 3060.
Setup is llama.cpp with tuning, at 16K context on Ubuntu and 12K context on WSL2.
The user notes editing speeds of about 188 t/s and is frustrated at not reaching 50-60 t/s for the 27B model.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
general-conversation
User reports Qwen3.8 Flash Next at 27-29 t/s generation and 290-340 t/s prefill across 4 GPUs using 52 GB of VRAM.
Setup is the IQ2_M quant with context offloaded at Q8 and an n-gram cache on SSD.
User reports roughly 98% clean Finnish output, but notes safety guardrails cause freezes in grey areas.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagenticreasoning
User reports Qwen 3.8 Flash/Next 125B MoE at about 38.8 t/s decode and about 200 t/s prefill on an RTX 5090 with 64 GB of system RAM.
Setup is ik_llama.cpp with the AP quant at Q4_K_M, using about 29.7 GiB of VRAM.
User notes the run is slower and depends on system RAM bandwidth and CPU offload, and adds that Qwen 3.8 27B is extremely fast but weaker. User asks for advice on models and optimizations.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 15 t/s generation and 100-200 t/s prompt processing at 130k context.
Setup is ROCm with flash-attn, split-mode layer and fit on.
The user notes Vulkan gives similar speed.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Huihui Qwen3.8-27B Abliterated Q4_K_S at an average TG of ~25.58 tok/s on an RTX 3090 24GB at 131072 context, in a conversation of 111797 tokens.
Setup uses a Q4 KV cache with 23.5/24.0 GB dedicated and 0.6/31.8 GB shared GPU memory, on a Xeon E5-2670 v3 with 48GB DDR4-2133, and each run logs a DTA of 64.2% to 70.8%.
Individual runs read 22.02 to 31.94 tok/s. At 196608 context the average TG is 25.80 tok/s with an average DTA of 64.57%.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User compares Qwen 3.8 27B against Qwen 3.6 27B on oMLX. Quality improved from 81.1 to 87.7 (+8%), while speed dropped from 35 to 29 tok/s and runtime increased 5x.
The user notes noticeably better quality but at the cost of tokens and time.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
summarization
User asks whether about 9 t/s is normal for their setup, and is new to local LLMs.
User also asks for model recommendations for PDF summaries on 16 GB of RAM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a custom Windows fork of NInfer running TP2 on two RTX 5060 Ti 16GB cards without P2P.
Setup uses MTP, with speeds of 35.75 t/s at MTP0, 57.71 t/s at MTP1, 63.15 t/s at MTP2, and 66.79 t/s at MTP3.
MTP4 reaches 68.5-68.9 t/s at 512 tokens, 70.6 t/s at 1024 tokens, and 76.65 t/s at 2048 tokens.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User compares MLX and llama.cpp on Qwen3.8-27B. MLX reaches 15.81 t/s generation and 81.76 t/s prompt processing. llama.cpp reaches 9.69 t/s generation and 99.61 t/s prompt processing.
The llama.cpp run uses the UD-Q4_K_M quant.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports PXA, a fork of ik_llama.cpp with a vLLM plugin, at 65.7 t/s plain decode with TP4 on an 8x V100 SXM2 NVLink system.
Setup is PXA as shipped with tensor parallel across four cards and no speculative decoding, measured on 16 GSM8K questions with 192 greedy tokens.
With speculative decoding at k=7 the same TP4 setup reaches 159.6 t/s, and prefill at 3k is 2273 t/s with a TTFT of 1.4 s. 1Cat vLLM on the same system reads 61.0 t/s plain decode at TP4.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 27B at Q4 running at 50 t/s on 16 GB VRAM with 200K context.
Setup uses adaptive-kv streaming to fit 196,608 tokens, with Q8_0 K and Q4_0 V cache quantization. The user pruned non-ASCII characters from the embedding table and LM head to save 700 MB, offloaded the embedding table to save 270 MB, and disabled MTP.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports Qwen3.8-27B-Escha-W2 at about 59 t/s short context and about 50.5 t/s at 60K context on a 16 GB RTX 4080 Super, pushed to 98K context.
Setup is Escha's SGLang build with a 2.469 bpw quant, FP8 KV cache and BF16 SSM state.
MTP4 gave about 67.7 t/s at 64K context, but the user chose no speculation for maximum context. The user finds quality surprisingly good despite the aggressive quant.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-contextagenticcoding
User reports a fork of NInfer with a custom NVFP4 KV cache, YaRN context extension to 555k, multi-level prefix reuse with a host KV safety net, tool calling improvements, and monitoring.
Benchmarks show decode at 117 tok/s at 400k+ context, cold prefill of 260s for 414k tokens at 1600 tok/s, H2D restore in 0.4s, and 30GB of host KV.
Quality results show LongBench matching int8, AIME at 96.7%, and needle-in-haystack at 100%.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen 3.8 Uncensored Q8 at 256k context on the AI PC (Strix Halo) at ~10 t/s.
Setup is a custom AI framework running across multiple devices. A Qwen3.6 35B MOE runs on an RTX 5090 at ~200 t/s, and a Gemma 4 model runs on a MacBook Air for security.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports CPU-offloaded inference with 128 GB of system RAM, running 3.2x faster prefill and 2x faster decode than llama.cpp.
User also tested GLM 5.3 Flash with 3.05 EXL3, which ran 2x slower in decode than llama.cpp.
Decode speeds warm up over time.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
long-context
User reports median decode improving from 30.2 to 33.3 t/s with radix-selection top-k fallback.
A quality screen scored 238/240 correct against 235/240 for the control, with no regression detected.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingsummarization
User asks about a Mac Studio M5 Max 128GB for local Qwen3.8-flash-next.
The user mentions the oQ4e quant at ~55 tok/s and also the oQ5e quant.
The user is debating whether to buy now or wait for the M7 Ultra, for private documents, notes, coding and general assistant work.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User runs Qwen3.8 27B IQ4_XS at 131k context on two RTX 2060 12GB cards, 24 GB total, and gets about 45 t/s.
Setup is llama.cpp.
User asks whether an upgrade to an RX 6800 16GB plus an RX 6800 XT 16GB, 32 GB total, would improve performance and how ROCm and Vulkan support compare.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
long-contextcoding
User reports about 40 t/s on prose and 75 t/s on coding at roughly 760k context.
Setup uses 8-bit dense layers and 4-bit expert layers, with peak memory around 117 GB and a requirement of iogpu.wired_limit_mb=120000. Model weights are at ddalcu/Qwen3.8-Flash-Next-MLX-Serve-mixed-4-8bit.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-context
User compares llama.cpp, SGLang, and FreeToken on Qwen3.8-Flash-Next.
At full context, TTFT is 35.4s for SGLang, 80.4s for FreeToken, 210.2s for llama.cpp with MTP, and 258.4s for llama.cpp baseline. Decode at full context is 126.9 t/s for SGLang, 87.5 t/s for FreeToken, 52.6 t/s for llama.cpp with MTP, and 20.3 t/s for llama.cpp baseline.
MTP improves decode 1.63x at 8K and 1.69x at 32K. Accuracy is 95.22-95.75% on GSM8K and 92.20-93.00% on MATH-500. Startup takes 16s for llama.cpp, 108s for SGLang, and 126s for FreeToken.