Local model performance on your hardware

reported speed:

30.0 tokens/s generation

quant:

Q4_K_M (GGUF)

kv:

8bit

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8 27B at a steady 30 t/s on an RTX 5090 Laptop 24GB. Setup is llama.cpp with Q4_K_M and 8-bit KV cache at 65k context, using 21 GB of VRAM. The user also tested Unsloth UD_Q5_K_XL at 27 t/s, and NInfer models reaching 82.54 t/s (8-bit, 132k context), 92.4 t/s (NVFP4, 32k context), and 86.63 t/s (NVFP4, 4-bit, 64k context). The user says the laptop is too slow for coding and recommends a DGX Spark instead.

reported speed:

3.6 tokens/s generation

quant:

INT4_SYM (OpenVINO IR)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3-14B Heretic at 3.56 t/s on an Intel AI Boost NPU37XX in an Acer Predator Helios 16 AI laptop with 32 GB RAM and an RTX 5090 Laptop 24GB. Setup is OpenVINO GenAI with the machine-made-Fibre INT4_SYM OpenVINO IR export, group_size=-1, all_layers=true, loaded directly via openvino_genai.LLMPipeline on NPU with no conversion. Load took 33.52 s and 96 output tokens took 26.98 s; peak RAM was about 15.6 GB with a 9.7 GB working set. User also benchmarked Qwen3-8B INT4 SYM on the same NPU at 6.72 t/s with 6.15 s load, and found Qwen3-30B-A3B MoE impractical due to host memory pressure during OpenVINO preparation.