Local model performance on your hardware

generation:

165.6 tokens/s

prompt processing (prefill):

9555.9 tokens/s

quant:

Q4_K_M (GGUF)

User reports 165.64 t/s generation and 9555.89 t/s prompt processing. Source evidence (verbatim): Source: https://blog.majid.info/llama-bench/golgotha.txt # NVIDIA GeForce RTX 5090 Laptop GPU 24GB: original Gemma4 E4B Q4_K_M llama-bench log First-hand author log linked from https://blog.majid.info/llama-bench/ (April 13, 2026 article). Measured hardware: NVIDIA GeForce RTX 5090 Laptop GPU 24GB. Gemma4 E4B Q4_K_M GGUF, llama.cpp e3f542d8 (4236), backend CUDA. pp512 9555.89 ±101.42 tokens/s; tg128 165.64 ±0.19 tokens/s. One device, ngl -1; 23983 MiB visible VRAM, not resident use. Laptop GPU, not desktop RTX5090. CPU and system RAM unknown. Command names --hf-repo unsloth/gemma-4-E4B-it-GGUF; exact tested quantization repository https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF ; tested revision unknown. Log reports 4.62 GiB model size and 7.52B counted parameters; size is not resident memory. Context depth, KV quantization, measured power, driver version and explicit speculation/concurrency declarations unreported. Raw source: https://blog.majid.info/llama-bench/golgotha.txt .

generation:

30.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

8bit

User reports Qwen3.8 27B at a steady 30 t/s on an RTX 5090 Laptop 24GB. Setup is llama.cpp with Q4_K_M and 8-bit KV cache at 65k context, using 21 GB of VRAM. The user also tested Unsloth UD_Q5_K_XL at 27 t/s, and NInfer models reaching 82.54 t/s (8-bit, 132k context), 92.4 t/s (NVFP4, 32k context), and 86.63 t/s (NVFP4, 4-bit, 64k context). The user says the laptop is too slow for coding and recommends a DGX Spark instead.

User reports a DIY Jev-like inference setup using unmodified open weight LLMs, evaluated on a 32,235-example benchmark. Qwen3.6 35B-A3B reached 75.5% accuracy at ~5.3 req/s on a laptop RTX 5090 24GB, with Qwen3 27B at 75.3% ~2.9 req/s and Qwen3-4B at 65.0% ~27 req/s. The approach uses boolean verification of candidate answers via true/false logits, batched through llama.cpp, with no NLI fine-tuning or classifier head. User notes the benchmark is not perfectly apples-to-apples and expresses skepticism about Jev hype.

quant:

IQ4_XS (GGUF)

kv:

Q8

rating:

5/5

User reports Qwen 3.6 27B is excellent for pyspark/python and data transformation debugging, running on an ASUS ROG Strix SCAR 18 with an RTX 5090 laptop (24 GB VRAM) and 64 GB DDR5 RAM. Setup is llama.cpp with the IQ4_XS quant at 200k context and a Q8_0 KV cache. The user initially tried q4_k_m at q4_0. No tokens/sec is reported. The user is cancelling cloud subscriptions due to local performance.