Speeds are as each source reported them. GPU count, offloading and concurrent requests change them, and prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing two reports.
- generation:
- 300-500 tokens/s
- quant:
- NVFP4 (NVFP4)
tool-useagentic
User reports SenseNova-Loop-9B running at 300-500 tok/s on a laptop RTX 5090 in NVFP4.
The model is a looped transformer with a small drafter, and the user says its architecture translates well to existing kernels.
The user cannot yet confirm it holds up against Qwen3.8 27B NVFP4 and suspects their setup may be broken despite a calibration run; in their tool-eval-bench runs SNL-9B scored 92 vs 98 on the standard set and 73 vs 87 on hard mode, with 0 on tenant separation and policy application.
- generation:
- 107.0 tokens/s
- prompt processing (prefill):
- 3706.5 tokens/s
- quant:
- Q4_K_M (GGUF)
- kv:
- Q8_0
codingagentic
User reports Qwen3-Coder-30B-A3B-Instruct at 107.0 tok/s generation and 3,706.5 tok/s prefill on an RTX 5090 Laptop 24GB in an MSI Raider A18.
Setup is llama.cpp with CUDA, Q4_K_M GGUF (17.28 GiB), Q8_0 KV cache, FlashAttention on, 65,536-token context, all layers on the GPU, one slot.
The figures come from a repository-analysis workload with an 8,638-token prompt and 371 generated tokens; the user notes the screenshot captures one strong interval, not an average across the audit.
- generation:
- 110.3 tokens/s
- prompt processing (prefill):
- 3254.0 tokens/s
- quant:
- MLP-NVFP4 hybrid (GGUF)
agentic
User reports Muse Glimmer 30B at 110.3 tok/s greedy decode and 3,254 tok/s prefill on an RTX 5090 Laptop 24GB card.
Setup is a patched llama.cpp with an MLP-NVFP4 hybrid quant, DFlash 2 speculative decoding at depth 15, and 128K context.
A baseline llama.cpp build with unsloth UD-Q4_K_XL and Meta DFlash reached 80.6 tok/s decode and 1,430 tok/s prefill on the same 6,880-token prompt. A 100,554-token prompt prefilled at 2,367 tok/s, and an agent turn with a cached 46K prompt took 0.89 s. KLD vs Q8_0 rose to 0.053/0.030/0.034 (agent/English/Swedish) from 0.033/0.012/0.015.
- generation:
- 30.0 tokens/s
- quant:
- NVFP4
codinglong-context
User reports about 30 t/s running Qwen3.8 Flash-Next in NVFP4 at 256k context on a laptop with a 16 GB Nvidia GPU and 96 GB DDR5 RAM.
Setup details beyond the quant and context length are not stated.
The figure is the user's current machine, cited while asking whether to buy a 24 GB VRAM / 128 GB RAM laptop or a 16 GB VRAM / 192 GB RAM laptop for local coding.
- generation:
- 125.5 tokens/s
- prompt processing (prefill):
- 2993.2 tokens/s
- quant:
- IQ2_XS (GGUF)
User reports Qwen3.8 Swift-1.5 Flash-Next at 125.5 tok/s generation and 2,993.2 tok/s prefill on an RTX 5090 Laptop 24 GB.
Setup is PolyStrata with an IQ2_XS GGUF of about 63.5 GiB, configured for 131,072 tokens of context, with 128 GB of system RAM.
The model is far larger than the card's 24 GB, so weights ran partly from system memory.
- generation:
- 165.6 tokens/s
- prompt processing (prefill):
- 9555.9 tokens/s
- quant:
- Q4_K_M (GGUF)
User reports 165.64 t/s generation and 9555.89 t/s prompt processing.
Source evidence (verbatim):
Source: https://blog.majid.info/llama-bench/golgotha.txt
# NVIDIA GeForce RTX 5090 Laptop GPU 24GB: original Gemma4 E4B Q4_K_M llama-bench log
First-hand author log linked from https://blog.majid.info/llama-bench/ (April 13, 2026 article). Measured hardware: NVIDIA GeForce RTX 5090 Laptop GPU 24GB. Gemma4 E4B Q4_K_M GGUF, llama.cpp e3f542d8 (4236), backend CUDA. pp512 9555.89 ±101.42 tokens/s; tg128 165.64 ±0.19 tokens/s. One device, ngl -1; 23983 MiB visible VRAM, not resident use. Laptop GPU, not desktop RTX5090. CPU and system RAM unknown. Command names --hf-repo unsloth/gemma-4-E4B-it-GGUF; exact tested quantization repository https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF ; tested revision unknown. Log reports 4.62 GiB model size and 7.52B counted parameters; size is not resident memory. Context depth, KV quantization, measured power, driver version and explicit speculation/concurrency declarations unreported. Raw source: https://blog.majid.info/llama-bench/golgotha.txt .
- generation:
- 37.0 tokens/s
- quant:
- Q4_K_XL (GGUF)
codingagentic
User reports Qwen3.8-27B with vision at about 37 tok/s on an RTX 5090 Laptop 24GB with 64GB RAM.
Setup is Q4_K_XL at 100K context, tested with the opencode and pi harnesses.
User also ran a two-laptop cluster of RTX 5080 Laptop 16GB machines: Qwen3.8-27B Q6_K fine-tuned for coding agents at 67 to 77 tok/s, Qwen3.8-27B Q5_K_XL at 53 tok/s, Qwen3-Coder-Next Q3_K_XL at 48 tok/s, and Qwen3.8-Flash-Next IQ4_XS at 19 tok/s. On the single 5090 laptop, Qwen3-Coder-Next Q3_K_XL reached 55 tok/s and Qwen3.8-Flash-Next with vision IQ4_XS at 262K context reached 20 tok/s. User says none of the local models can complete even minor tasks, filling context without output or producing nonsense, and that hosted GPT-6 and Opus 5.5 handle the same tasks without issue.
- generation:
- 30.0 tokens/s
- quant:
- Q4_K_M (GGUF)
- kv:
- 8bit
coding
User reports Qwen3.8 27B at a steady 30 t/s on an RTX 5090 Laptop 24GB.
Setup is llama.cpp with Q4_K_M and 8-bit KV cache at 65k context, using 21 GB of VRAM.
The user also tested Unsloth UD_Q5_K_XL at 27 t/s, and NInfer models reaching 82.54 t/s (8-bit, 132k context), 92.4 t/s (NVFP4, 32k context), and 86.63 t/s (NVFP4, 4-bit, 64k context). The user says the laptop is too slow for coding and recommends a DGX Spark instead.
User reports a DIY Jev-like inference setup using unmodified open weight LLMs, evaluated on a 32,235-example benchmark.
Qwen3.6 35B-A3B reached 75.5% accuracy at ~5.3 req/s on a laptop RTX 5090 24GB, with Qwen3 27B at 75.3% ~2.9 req/s and Qwen3-4B at 65.0% ~27 req/s.
The approach uses boolean verification of candidate answers via true/false logits, batched through llama.cpp, with no NLI fine-tuning or classifier head. User notes the benchmark is not perfectly apples-to-apples and expresses skepticism about Jev hype.
- generation:
- 3.6 tokens/s
- quant:
- INT4_SYM (OpenVINO IR)
User reports Qwen3-14B Heretic at 3.56 t/s on an Intel AI Boost NPU37XX in an Acer Predator Helios 16 AI laptop with 32 GB RAM and an RTX 5090 Laptop 24GB.
Setup is OpenVINO GenAI with the machine-made-Fibre INT4_SYM OpenVINO IR export, group_size=-1, all_layers=true, loaded directly via openvino_genai.LLMPipeline on NPU with no conversion. Load took 33.52 s and 96 output tokens took 26.98 s; peak RAM was about 15.6 GB with a 9.7 GB working set.
User also benchmarked Qwen3-8B INT4 SYM on the same NPU at 6.72 t/s with 6.15 s load, and found Qwen3-30B-A3B MoE impractical due to host memory pressure during OpenVINO preparation.
- quant:
- IQ4_XS (GGUF)
- kv:
- Q8
- rating:
- 5/5
codingtool-use
User reports Qwen 3.6 27B is excellent for pyspark/python and data transformation debugging, running on an ASUS ROG Strix SCAR 18 with an RTX 5090 laptop (24 GB VRAM) and 64 GB DDR5 RAM.
Setup is llama.cpp with the IQ4_XS quant at 200k context and a Q8_0 KV cache. The user initially tried q4_k_m at q4_0.
No tokens/sec is reported. The user is cancelling cloud subscriptions due to local performance.