Local model performance on your hardware

reported speed:

8.0 tokens/s generation

quant:

Q2 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 8 t/s with Qwen3.5 35B Q2 on an RTX 4060 Ti 8GB, with no CUDA device selected. Setup is llama.cpp with a Q2 GGUF quant; the user notes VRAM usage was 3000MB/8k and that the run had no CUDA device selected. The user is troubleshooting a cuBLAS crash and asks how to cleanly uninstall and reinstall llama.cpp with CUDA 12, and whether CUDA would improve generation speed.

reported speed:

13.0 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports a 26B-A4B Gemma 4 uncensored MoE running at about 13 tok/s on an RTX 4060 Ti 8GB, with experts offloaded to CPU. Setup is LM Studio with a Jev decision-model router that picks between a 3B, a 4B, a 12B and the 26B MoE; only one model fits in VRAM at a time, so a misroute costs a 15 to 45 second model swap. The router agreed with the user's labels 16 out of 16 on labelled prompts, with median latency 530ms and p95 about 643ms; a post-reply judge caught refusals 15 out of 15 on synthetic pairs and 5 of 6 on real replies.