Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen2.5 32B at 37 tokens per second on an RTX 5090.
No engine, quant, context length or KV cache settings are stated.
The user is an amateur and has not collected full benchmarks yet, and may post more results later.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.5-27B Q4_K_M on an RTX 5090 at 30.09 t/s generation and 1515.83 t/s prompt processing with llama-bench (pp1024/tg400, 5 reps).
Setup is llama.cpp b10741 with CUDA 13.3 on Windows 11, -ngl 99 and flash attention on, 15.65 GiB GGUF.
A Qwen3 MoE 30B-A3B control on the same rig reaches 134.18 t/s tg, ruling out a global Windows/driver overhead. The user notes native Linux with MTP reaches about 76% of roofline and ~1.7x speedup, while the Windows/Ollama path runs 1.5-1.6x slower at every draft depth. PCIe Gen5 x1 vs x4 gave 27.95 vs 30.09 t/s, so decode is not host-link-bound.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports 64.8 t/s generation and 1973 t/s prompt processing.
Source evidence (verbatim):
Source: https://github.com/adrienbrault/glm5.3-flash-2x-rtx5090
# GLM-5.3-Flash (321B MoE, 2.05bpw EXL3) on 2x RTX 5090 + 64 GB DDR5, ExLlamaV3 + TabbyAPI
Hardware: two NVIDIA RTX 5090 (32 GB each, PCIe 5.0 x8), AMD Ryzen 7 9800X3D, 60 GB usable of 2x32 GB DDR5-6000. Engine: ExLlamaV3 dev 5783a93 served through TabbyAPI. Model: GLM-5.3-Flash by zai-org, 321B parameters with about 18B active, turboderp 2.05 bpw EXL3 quant (76 GB), with experts offloaded to system RAM (96 and 102 CPU-resident experts per MoE layer out of 288) and 8-bit KV cache at 262144 tokens. Single-stream decode with MTP draft enabled (DRAFT=1, DRAFT_N=1): 64.8 tokens/s median of ten requests, 1,024 forced output tokens at temperature 0, median of three boots. Cold-prompt prefill measured at 1,973 tokens/s at 32,738 prompt tokens. Without the draft, decode with 2, 3 and 4 concurrent streams was 74.4, 92.0 and 98.0 tokens/s in total.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports 26.3 t/s generation and 2165 t/s prompt processing.
Source evidence (verbatim):
Source: https://dev.to/seppegadeyne/run-glm-53-flash-321b-locally-with-project-maya-and-exl3-on-one-rtx-5090-24pj
# Run GLM-5.3-Flash (321B) locally with Project Maya and EXL3 on one RTX 5090
Hardware: one RTX 5090 32 GB (also driving the desktop), Ryzen 9 9950X3D, 128 GB DDR5 at 5,600 MT/s, Arch Linux, NVIDIA driver 615.78, CUDA 13.4. Engine: Project Maya 1.1.11 (a Strata fork) compiling llama.cpp at a pinned commit. Model: GLM-5.3-Flash 321B MoE with about 18B active, turboderp EXL3 3.05 bpw (125.3 GB), with the expert cache and KV streaming offloading to system RAM. Benchmark: streaming client, 8 runs, greedy decoding, 512 output tokens, median reported. At 128K context with the default reserve: decode 26.3 tokens/s (30.4 on Dutch prose, 21.9 on code) and prefill 2,165 tokens/s at 126K. At 262K context: decode 24.1 tokens/s and prefill 2,039 tokens/s at 254K.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8-Flash-Next on an RTX 5090 32 GB, reporting 155.3 t/s decode at 4K context with the Strata NVFP4 stack and 134.0 t/s with Infernix Dense8.
Setup is Strata 0.1.42-nvfp4.2 with an NVFP4/GPTQ pack, int8 KV cache, MTP speculative decoding (--spec 6), 15 CPU expert workers and 128 GB DDR5 at 6000 MT/s; one request at a time, 256 output tokens, greedy sampling.
At 32K Strata gives 154.7 t/s and Infernix 144.2 t/s; at 128K Infernix is faster at 147.3 t/s against 142.1 t/s. Infernix answers sooner (1.02 s vs 1.42 s at 4K). A retained Unsloth UD-Q4_K_XL run on the same Strata binary reached 130.5 t/s at 4K. The user notes the three stacks use different model representations and that this is not an intelligence ranking.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentictool-use
User benchmarks eight models on 33 DevOps agent tasks with taracode 3.3.1 on taraengine 0.3.1, at temperature 0 and 32k context, one run per model. On an RTX 5090 (32 GB), qwen3.8:27b leads at 97% pass rate and 126 tok/s, using 23.2 GiB of VRAM.
Setup is taraengine with Q4_K_M weights and MTP speculative decoding, which the engine pairs with seven of the eight models. The same catalog on an M4 Max (48 GB) runs 3.0 to 4.4 times slower, with qwen3.8:27b at 39 tok/s and 94% pass rate.
Against an earlier Ollama 0.35 board on the same 5090, gemma4:31b rose from 64 to 164 tok/s, gemma4:12b from 138 to 308 and qwen3.6:35b from 226 to 388, with every pass rate within four tasks. The user notes one run per row, rows within four tasks are a tie, and 33 tasks is small.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next running natively in NVFP4 under Basalt on an RTX 5090 at 400W with 128 GB of DDR5 RAM, giving 304 tok/s on structured output and 183 tok/s on prose.
Setup is Basalt with a .basalt pack that keeps stock NVIDIA experts and stores the dense layers as Q6_K; adding a peer RTX 5060 Ti raises the figures to 479 tok/s structured and 239 tok/s prose. Prefill reaches 8,582 tok/s on a 64K prompt, 7.6s to first token.
Against Infernix on the same NVFP4 weights and box the user claims up to 1.3x on decode and 1.6x on prefill. Quality versus Q8_0 over 100 chunks: PPL 3.308 (Q8_0 3.286), KLD 0.068, 91.5% same top-1.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports 73.2 t/s generation.
Source evidence (verbatim):
Source: https://gist.github.com/PierpaoloPernici/f1d1382f8e357b4faffb1a9f584cc1df
# vLLM vs NInfer: Qwen3.8-27B benchmark on an RTX 5090
Hardware: NVIDIA GeForce RTX 5090 32 GB, driver 610.57.04, CachyOS Linux. Engine vLLM 0.27.1, weights NVFP4 (Qwen3.8-27B NVFP4 for RTX 5090, modelopt), fp8 KV cache, max-num-seqs 16, tool-eval-bench harness, PP=2048 TG=128. Plain run without MTP, cold, no prefix caching, concurrency 1: client decode 73.2 tok/s at context depth 0, 62.7 at 4K, 54.0 at 8K, 40.4 at 16K, 24.4 at 32K. vLLM server-side prefill is about 6,000 to 18,000 tok/s. With MTP speculative decoding @3 at concurrency 1: 117.3 tok/s at depth 0. At concurrency 4 without MTP: 217.3 aggregate tok/s at depth 0.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 2,788 tokens/s on a single RTX 5090 with LLaDA-2.1-mini + ORL in NVFP4, using the dlmrt inference engine.
All numbers are for single-request decoding (batch size 1), not aggregate throughput. The 2,788 t/s figure is a peak result on a specific benchmark, not an average across arbitrary prompts.
The post also reports 7,708 tokens/s on 8x B200 with DiffusionGemma 26B MoE + ORL, up to 6.9x higher end-to-end throughput on QwenDLM-4B, and 1,298 tokens/s on a single H100 versus 746 tokens/s for DFlash and 405 tokens/s for EAGLE3.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at ~150 tok/s generation and ~4k prefill on two RTX 5090s.
Setup is Strata with Q5_K_XL weights.
User is considering selling one 5090 and buying four R9700s for roughly the same price, doubling VRAM, and asks how that would affect which models can be run and their performance.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 146.5-152.5 t/s generation and 8,535 t/s prompt processing on an RTX 5090 32GB with 128GB RAM.
Setup is Strata with NVFP4 GPTQ 4-bit weights and 128K context; the model is a 125B-A6B MoE with speculative decoding and expert offload to RAM and SSD.
The figures are a range over three runs; an Unsloth UD-Q4_K_XL run on the same box gave 132.7-138.8 t/s generation and 6,255 t/s prompt processing. The post is a beginner guide and also cites official and community numbers for other hardware.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingmath
User compares Qwen3.8-Flash-Next NVFP4 GPTQ against Unsloth UD-Q4_K_XL on an RTX 5090 32GB with 128GB DDR5-6000 RAM and an Intel Core Ultra 7 270K Plus, running native Strata inference on Windows.
Setup is Strata 0.1.42-nvfp4.2 with 128K context capacity, INT8 KV cache, MTP 6, 15 CPU workers and a 1600MiB VRAM reserve. NVFP4 reached 162.9 t/s at 4K, 161.3 t/s at 32K and 154.1 t/s at 128K; Unsloth UD-Q4_K_XL reached 130.5, 131.0 and 127.6 t/s respectively. Time to first visible answer was 1.42/3.90/14.90 seconds for NVFP4 and 1.99/5.24/20.41 seconds for Unsloth.
Each figure is the median of three runs on identical synthetic code-review prompts with 256 output tokens, temperature 0 and reasoning disabled. NVFP4 figures come from its best complete session of four, which the user notes favours NVFP4. In quality testing Unsloth passed 73/82 selected tasks versus NVFP4's 70/82, mainly through instruction following; maths and coding tied.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-contextagentic
User reports Qwen3 8B at 89 t/s decode and 6.7k tok/s prefill on an RTX 5090 using PulsarLM, a new inference engine that evicts low-attention token pages to keep the context window bounded.
Setup is PulsarLM with custom attention kernels that accumulate attention mass per token page and remove low-mass pages at intervals; RoPE embeddings are re-rotated to keep positions contiguous. Context snapshots avoid re-prefills at the cost of extra RAM.
Batched decode reaches 1.2k tok/s at 16 concurrent requests. The engine is early-stage, tested only on Linux x86_64, and supports only smaller/older models; the author removed a BM25+attention recall mechanism and recommends pairing with external RAG.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 136.7 t/s generation.
Source evidence (verbatim):
Source: https://kgptalkie.com/tutorials/llm-benchmarking/qwen-3-8-27b-llama-cpp-speed-settings
# Qwen 3.8 27B Speed Settings on llama.cpp
Hardware: NVIDIA RTX 5090 32GB. Engine: llama.cpp b10448. Model: Qwen 3.8 27B Q4_K_M. Baseline with MTP off (no speculative decoding, flash attention on, f16 KV cache, 17966 MB VRAM): decode 73.6 tok/s, prompt 3036 tok/s (the article does not explicitly tie the prompt figure to this run). Best MTP run (speculative decoding on: --spec-type draft-mtp, --spec-draft-n-max 2, q4_0 K and V cache, flash attention on, context 32768, -np 1, -ngl 999): decode 136.7 tok/s, 17412 MB VRAM. Best draft-depth-only run n=3 with f16 cache: 133.6 tok/s, reverse-order rerun 129.7 tok/s. MTP reduces prompt processing by roughly 5 to 8 percent.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next GSQ-RCO IQ3_S at a median 161.3 tok/s generation on an RTX 5090 with 128GB DDR5-6000, measured on three 240,000-token inputs.
Setup is Strata 0.1.43.1 with INT8 KV cache, 32,768 resident KV cells, spec 4, draft floor 0.50, 12 pool workers, a 55 GiB resident-expert budget and a 700 MiB VRAM reserve, under a 262,144-token context ceiling.
Median prompt processing was 5537.8 tok/s, median first visible answer 43.475 s and median total request time 45.071 s; the generation range was 159.6-203.4 tok/s. The same nine requests at a 131K ceiling gave 173.5, 181.2 and 183.0 tok/s at 4,096, 32,768 and 128,000 input tokens, and the 262K ceiling gave 171.2, 175.0 and 179.2 tok/s. Unsloth UD-IQ4_XS reached a 121.2 tok/s short-task median after tuning, against 105.0 before, with 3/3 exact recall at 100,510 and 128,510 tokens. The user notes the RAM upgrade comparison is not a RAM-only A/B test and that no statistical significance was established.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-Flash-Next 125B-A6B at a median 116 t/s decode on an RTX 5090 32GB.
Setup is the strata-nvfp4 fork with NVFP4 experts and Q8_0 dense weights, int8 KV cache, MTP draft head with spec 5, and 63 GiB of experts pinned in system RAM.
Decode was median 116 t/s, mean 103, peak 171; prefill averaged 3,332 t/s with a 6,623 t/s peak. The 33m 45s run produced 124,443 output tokens over 87 steps with a 99.0% cache hit rate and no compactions.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.6-35B-A3B at 389.0 t/s decode and 5,310 t/s prefill on a single RTX 5090.
Setup is the NInfer engine (ninfer-ext fork) with no quant code stated, one model at a time, no Python in the inference path.
Decode figures are aggregate rates for the stated number of lanes in flight, not per-request latency; the C=1 prefill figure is a true per-request rate while C=2 and above are point aggregates that include each request's decode time. MTP rows draft 3 tokens with the full proposal head and reach 650.3 t/s at C=1, 958.0 at C=2, 1,163.9 at C=4 and 1,368.0 at C=8. The same chart also covers Gemma 4 26B-A4B, Gemma 4 31B IT M1, Qwen3.6-35B-A3B nvfp4, Qwen3.8-27B groupwise-int, Qwen3.8-27B nvfp4 and Qwen3.8-Flash-Next nvfp4.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8-Flash-Next (125B MoE, 6B active) at ~95-105 tok/s on a single RTX 5090, with three lanes giving ~130 tok/s aggregate.
Setup is the ignis engine with a trellis 2.5-bit expert quant (67 GB file), expert cache in VRAM (~17 GB) and experts in pinned host RAM (~38 GB) over PCIe 3.0.
Quality: MMLU-Pro proxy 70.1% vs 73.7% for BF16, top-1 agreement with BF16 82-98% by domain.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 27B at 701.2 t/s on one NVIDIA RTX 5090, the median of three repeats (695.3, 709.0, 701.2) for a copy/edit task from a 27,135-token prompt with 400 completion tokens.
Setup is llama.cpp b11060 CUDA with UD-Q4_K_XL, q8_0 key and value cache, Flash Attention, greedy sampling, and draft-mtp plus ngram-mod speculation with draft maximum 16 and n-gram range 96-192.
The 131,072-token context is the server limit, not the measured prompt length, and the figure is not general chat or novel-code speed. At 3,038 prompt tokens the same profile returned a 784.1 t/s median (range 768.0-814.3).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-context
User reports running Qwen3.8 Flash-Next IQ3_XXS at a 700K context window on an RTX 5090 32GB with 64GB system RAM, completing a project in one continuous session without context compaction.
Generation speed is given as approximately 100-120 tokens/sec, with vision disabled. The model ran with CPU offload given the 125B size on a single 32GB card.
User notes possible quality degradation at extreme context lengths and estimates 1M context might be possible at around 80 tokens/sec, though untested.