Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen 3.8 Flash-Next at about 25 t/s on an RTX 5090 with 64 GB of RAM.
Setup is the Q4_K_XL quant, with the n-gram table unloaded to an SSD to fit in memory.
User advises that the n-gram table can be offloaded to an SSD with minimal speed cost.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports ROCm 7.14 on a Radeon 780M with llama.cpp is fast but unstable and crashes.
Setting AMD_SERIALIZE_KERNEL=3 stabilizes it but drops prompt processing to about 100 t/s, down from 200-300 t/s without the workaround.
User asks for advice.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User reports Qwen3.8-27B at ~70 tok/s on a single RTX 3090 with 24 GB, using 4-bit inference via llama.cpp.
The model is used for trading decisions, with QLoRA training and Ollama mentioned as a potential engine for distributed inference.
User is calling for contributors to an LLM trading experiment.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-contextvision
User reports a custom llama.cpp Vulkan backend patch for Qwen3.8-Flash-Next improves prompt processing from 346 to 448 t/s and generation from 13.3 to 14.4 t/s at 2.7k context.
At 131k context, prompt processing improves from 195 to 263 t/s and generation from 9.1 to 10.4 t/s.
The patch fixes a top-k fallback to CPU in sparse attention. Greedy outputs are identical, the needle is found at 131k, and images are read correctly.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports running unsloth/Qwen3.8-Flash-Next-GGUF on an RTX 5090.
The user asks how it runs on the 5090 and hopes for more benchmarks.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports a run on a 64GB M1 Max.
No drafter was used.
User mentions n-gram and stripping embeddings as potential optimizations.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports over 200k context on 16 GB of VRAM with Qwen 3.8 27B at the IQ3_XXS quant.
Setup is a laptop with Thunderbolt 4 and an Aorus 5060 Ti AI Box eGPU on Windows 11, with the KV cache quantized to q5_1.
The user previously ran UD-Q3_K_XL at 140k context. Prompt processing dropped from 700-800 tk/s to 400 tk/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-context
User compares llama.cpp with a Q4_K_XL quant against vLLM with W4A16-AutoRound on an RTX 3090.
llama.cpp reaches about 45-50 t/s at 150K context, while vLLM reaches about 55-65 t/s at 175K context.
The user prefers vLLM for speed but notes llama.cpp is needed for vision, and asks about the quality difference between the quants for coding.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 55 t/s on 4x RTX 3090 with Qwen3.8-Flash-Next-UD.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks a custom llama.cpp branch with faster Metal inference and n-gram SSD offload at pp512 530.18 t/s and tg128 31.17 t/s with resident n-gram.
The model is 68.37 GiB and 125.74 B params.
With SSD read mode, pp512 is 197.92 t/s and tg128 is 28.47 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User compares Qwen3.8 27B to Qwen3.6 35B-A3B for daily coding use.
User reports Qwen3.8 27B is verbose and overthinks, while Qwen3.6 35B-A3B is faster and more concise.
The comparison is on a Radeon AI PRO R9700, with an iGPU comparison raised as hypothetical.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports 46 t/s generation and 700 t/s prefill with tensor parallelism enabled, against 28 t/s generation and 1000+ t/s prefill without it.
Setup uses MTP with max draft tokens 6 and probability 0.88.
Draft acceptance is 0.93757 with a mean length of 5.10.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash Next at 15-19 t/s generation and 550 t/s prompt processing on dual RTX 3090s with a Threadripper Pro.
Setup is llama.cpp with speculative decoding using draft-mtp.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8-27B at 503.7 t/s prefill and 27.7 t/s decode on dual RTX 3060 12GB cards.
Setup is a Q4_K_M GGUF with a Q8_0 KV cache at 131k context, BF16 vision projector on CPU, xHigh reasoning, on a Ryzen 5 5500 with 64GB DDR4-3200 and GPUs limited to 130W each.
With speculation the user measured about 503 t/s prefill and about 29 t/s decode.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports a 125B MoE model with 6B active on 2x RTX 3060 12GB in llama.cpp, with prefill improving from 36 to 303 t/s and decode at 12 t/s.
Setup uses -sm layer and -ub 2048.
ik_llama.cpp was also tested at 407 t/s prefill but uses ~108GB RAM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 6-7 t/s on Qwen 3.8 flash next with a 1-bit quant, running llama.cpp on 6 GB VRAM and 16 GB system RAM.
User calls the result fire for a 1-bit quant.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User asks about Intel Arc B60 or B65 for Qwen 3.8 27B, currently getting 3 t/s on an RTX 5070.
User mentions a custom vLLM fork for Intel Arc reaching about 20 t/s on the larger B70.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports a 50% performance boost from 20 t/s to 30 t/s by offloading hot experts to VRAM in a llama.cpp fork.
The result was tested on coding workloads only.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
creative-writing
User reports average prefill and token generation throughput with an MTP draft model.
User finds quality superior to Qwen3.8-27B for architectural tasks.