llamaperf

NVIDIA RTX 5060 8GB reports

6 reports. These are individual setups, not a controlled benchmark. NVIDIA RTX 5060 8GB speeds and models →

Speeds are as each source reported them. GPU count, offloading and concurrent requests change them, and prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing two reports.

Tone: positiveNVIDIA hardware
generation:
57.0 tokens/s
quant:
Q4_K_M (GGUF)
codingagentictool-use

User reports Qwen3-8B at about 57 tok/s on an RTX 5060, the only machine where the full verification was run. Setup is llama.cpp with Q4_K_M weights; the tool picked the model, quant, context size and KV cache for the hardware. The figure comes from smoke tests, not a quality benchmark: 9 of 9 checks passed and tool calling works. AMD, Apple and the other supported runtimes have had much less testing.

Nov 9, 2026
NVIDIA hardware
generation:
20.0 tokens/s
agenticcoding

User asks for advice on upgrading a local stack with a €1000 budget, currently a Ryzen 5 9600, RTX 5060 8GB and 32GB DDR5. They report running Ornith 1.5 35B A3B with offload at an average of 20 tk/s, alongside Ornith 1.5 9B, and embeddings, TTS and STT on system RAM. Options considered are a used RTX 3090 at 600-700, a second GPU such as an RTX 3060, or a 16GB Mac Mini M6 at 850; no benchmark of those parts is given.

Oct 19, 2026
Tone: mixedNVIDIA hardware
generation:
30.0 tokens/s
quant:
Q4_K_XL (GGUF)
kv:
Q8_0
codingagentic

User reports Qwen3.6 35B-A3B at close to 30 t/s on an RTX 5060 8GB with a Ryzen 5 3600 and 32GB RAM. Setup is LM Studio with Q4_K_XL weights and a Q8_0 KV cache, reaching 98k context by moving some layers out of VRAM; total VRAM use stays below 7.5GB and RAM around 28-30GB. The user is asking for a better agentic coding model for this hardware and doubts Qwen3.8 27B would fit, and has not yet tested the 2-bit Ternary Bonsai 2.

Oct 18, 2026

Qwen3.8 27B

NVIDIA RTX 5060 8GB · llama.cpp · 8,192 ctx

NVIDIA hardware
generation:
30.1 tokens/s
quant:
UD-IQ2_XXS (GGUF)
kv:
q4_0

User reports Qwen3.8-27B at 30.10 tok/s median decode on an RTX 5060 8GB, over five runs at an occupied 8192-token prompt. Setup is llama.cpp with UD-IQ2_XXS weights and q4_0 KV cache, 65/65 layers on CUDA, peak VRAM 7767 MiB. A short-context FULL_GPU decode of 31.39 tok/s and a ctx512 control of 18.11 tok/s are also given; quality gate and uncensored checkpoint are not done.

Oct 7, 2026
NVIDIA hardware
generation:
2.4 tokens/s
quant:
FP8 (safetensors)

User reports DeepSeek-V4.1-Flash (552B backbone + 196B Engram, 384 experts top-6, FP8 + FP4) running on a single RTX 5060 8GB at ~1.6 tokens/s from disk only and ~2.4 tokens/s with a 16 GB RAM cache. Setup streams experts and Engram rows from NVMe using DeepSeek's reference code with a modified storage layer; only the dense part lives on the GPU (cap --vram_gb 7.3), with 240 experts (4.2 GiB) read per token. Batch size 1, text only, context limit 8192, prefill max 700 tokens. Live chat via the OpenAI-compatible server with a 16 GB RAM cache reaches 2.43 tokens/s; peak VRAM 7.06 GiB. A 311-token prompt takes 17.9 s. The user notes the speed-ups changed nothing in output (136/136 tokens identical) and that DeepSeek's untouched reference could not be run side by side because it cannot load the model in 8 GB.

Oct 5, 2026
NVIDIA hardware
generation:
40.0 tokens/s
prompt processing (prefill):
500.0 tokens/s
quant:
Q4_K_XL (GGUF)
kv:
Q8

User asks whether speculative decoding (MTP) is still worth enabling for a 35B A3B MoE model offloaded across an 8GB RTX 5060 and 32GB of system RAM. Current llama-server setup uses a Q4_K_XL GGUF with a Q8 KV cache, flash attention on, 4096 batch and ubatch, 16 CPU cores, 40 MoE layers on CPU and 99 GPU layers, yielding about 40 t/s generation and 500 t/s prompt processing without MTP. User recalls earlier reports that MTP hurt prompt processing and wants to know if that is still the case and how others configure llama-server.

Sep 18, 2026