llamaperf

NVIDIA RTX 4060 Ti 8GB reports

3 reports. These are individual setups, not a controlled benchmark. NVIDIA RTX 4060 Ti 8GB speeds and models →

Speeds are as each source reported them. GPU count, offloading and concurrent requests change them, and prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing two reports.

Tone: positiveNVIDIA hardware
generation:
52-65 tokens/s
quant:
Q4_K_XL (GGUF)
agentic

User reports Qwen3.6-35B-A3B at 52-65 tok/s on an RTX 4060 Ti 8GB with 64 GB system RAM. Setup is llama.cpp with Q4_K_XL at 131k context, experts offloaded to system RAM and the rest in VRAM, on headless Linux. The user compares download defaults (~25 tok/s), tuned Windows (39-45 tok/s), and tuned headless Linux (52-65 tok/s). They also report Qwen3.8-Flash-Next 125B at 17-19 tok/s and Ternary Bonsai 27B at 36 tok/s.

Oct 6, 2026
Tone: mixedNVIDIA hardware
generation:
8.0 tokens/s
quant:
Q2 (GGUF)

User reports 8 t/s with Qwen3.5 35B Q2 on an RTX 4060 Ti 8GB, with no CUDA device selected. Setup is llama.cpp with a Q2 GGUF quant; the user notes VRAM usage was 3000MB/8k and that the run had no CUDA device selected. The user is troubleshooting a cuBLAS crash and asks how to cleanly uninstall and reinstall llama.cpp with CUDA 12, and whether CUDA would improve generation speed.

Sep 29, 2026
Tone: positiveNVIDIA hardware
generation:
13.0 tokens/s

User reports a 26B-A4B Gemma 4 uncensored MoE running at about 13 tok/s on an RTX 4060 Ti 8GB, with experts offloaded to CPU. Setup is LM Studio with a Jev decision-model router that picks between a 3B, a 4B, a 12B and the 26B MoE; only one model fits in VRAM at a time, so a misroute costs a 15 to 45 second model swap. The router agreed with the user's labels 16 out of 16 on labelled prompts, with median latency 530ms and p95 about 643ms; a post-reply judge caught refusals 15 out of 15 on synthetic pairs and 5 of 6 on real replies.

Sep 18, 2026