llamaperf

NVIDIA RTX 5060 8GB

NVIDIA · 8GB · 6 reports

Is this your card?

On the NVIDIA RTX 5060 8GB, Qwen3 8B at 4-bit, the model most run on it, generates a median of 57.0 tokens per second from one plain run (community reports, as of 22 Nov 2026).

As of 22 Nov 2026, the models most run on the NVIDIA RTX 5060 8GB, each one size at one quant level, with the median generation speed of its plain runs (one device, one request, no speculative decoding):

Bars are medians on one scale, 0 to 57.0 tokens/s. Press ▶ to watch text arrive at a row's speed.

Engines people use on it: llama.cpp 3 (reports) · LM Studio 1 (reports) · ds4 1 (reports)

Run models on your NVIDIA RTX 5060 8GB? Add your numbers.

Your coding agent can run a speed test on your machine and send it here with a free account. Or paste a result you've already posted.

Reported performance on the NVIDIA RTX 5060 8GB

These reports mix GPU counts and memory setups, so a model listed here may have needed extra cards or CPU offloading. The calculator estimates which models fit in 8 GB of VRAM. Prompt processing is how fast the input is read, not how long you wait for the first word. How to measure time to first token.

Speeds are as each source reported them. GPU count, offloading and concurrent requests change them, and prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing two reports.

Tone: positiveNVIDIA hardware
generation:
57.0 tokens/s
quant:
Q4_K_M (GGUF)
codingagentictool-use

User reports Qwen3-8B at about 57 tok/s on an RTX 5060, the only machine where the full verification was run. Setup is llama.cpp with Q4_K_M weights; the tool picked the model, quant, context size and KV cache for the hardware. The figure comes from smoke tests, not a quality benchmark: 9 of 9 checks passed and tool calling works. AMD, Apple and the other supported runtimes have had much less testing.

Nov 9, 2026
NVIDIA hardware
generation:
20.0 tokens/s
agenticcoding

User asks for advice on upgrading a local stack with a €1000 budget, currently a Ryzen 5 9600, RTX 5060 8GB and 32GB DDR5. They report running Ornith 1.5 35B A3B with offload at an average of 20 tk/s, alongside Ornith 1.5 9B, and embeddings, TTS and STT on system RAM. Options considered are a used RTX 3090 at 600-700, a second GPU such as an RTX 3060, or a 16GB Mac Mini M6 at 850; no benchmark of those parts is given.

Oct 19, 2026
Tone: mixedNVIDIA hardware
generation:
30.0 tokens/s
quant:
Q4_K_XL (GGUF)
kv:
Q8_0
codingagentic

User reports Qwen3.6 35B-A3B at close to 30 t/s on an RTX 5060 8GB with a Ryzen 5 3600 and 32GB RAM. Setup is LM Studio with Q4_K_XL weights and a Q8_0 KV cache, reaching 98k context by moving some layers out of VRAM; total VRAM use stays below 7.5GB and RAM around 28-30GB. The user is asking for a better agentic coding model for this hardware and doubts Qwen3.8 27B would fit, and has not yet tested the 2-bit Ternary Bonsai 2.

Oct 18, 2026

Qwen3.8 27B

NVIDIA RTX 5060 8GB · llama.cpp · 8,192 ctx

NVIDIA hardware
generation:
30.1 tokens/s
quant:
UD-IQ2_XXS (GGUF)
kv:
q4_0

User reports Qwen3.8-27B at 30.10 tok/s median decode on an RTX 5060 8GB, over five runs at an occupied 8192-token prompt. Setup is llama.cpp with UD-IQ2_XXS weights and q4_0 KV cache, 65/65 layers on CUDA, peak VRAM 7767 MiB. A short-context FULL_GPU decode of 31.39 tok/s and a ctx512 control of 18.11 tok/s are also given; quality gate and uncensored checkpoint are not done.

Oct 7, 2026
NVIDIA hardware
generation:
2.4 tokens/s
quant:
FP8 (safetensors)

User reports DeepSeek-V4.1-Flash (552B backbone + 196B Engram, 384 experts top-6, FP8 + FP4) running on a single RTX 5060 8GB at ~1.6 tokens/s from disk only and ~2.4 tokens/s with a 16 GB RAM cache. Setup streams experts and Engram rows from NVMe using DeepSeek's reference code with a modified storage layer; only the dense part lives on the GPU (cap --vram_gb 7.3), with 240 experts (4.2 GiB) read per token. Batch size 1, text only, context limit 8192, prefill max 700 tokens. Live chat via the OpenAI-compatible server with a 16 GB RAM cache reaches 2.43 tokens/s; peak VRAM 7.06 GiB. A 311-token prompt takes 17.9 s. The user notes the speed-ups changed nothing in output (136/136 tokens identical) and that DeepSeek's untouched reference could not be run side by side because it cannot load the model in 8 GB.

Oct 5, 2026
NVIDIA hardware
generation:
40.0 tokens/s
prompt processing (prefill):
500.0 tokens/s
quant:
Q4_K_XL (GGUF)
kv:
Q8

User asks whether speculative decoding (MTP) is still worth enabling for a 35B A3B MoE model offloaded across an 8GB RTX 5060 and 32GB of system RAM. Current llama-server setup uses a Q4_K_XL GGUF with a Q8 KV cache, flash attention on, 4096 batch and ubatch, 16 CPU cores, 40 MoE layers on CPU and 99 GPU layers, yielding about 40 t/s generation and 500 t/s prompt processing without MTP. User recalls earlier reports that MTP hurt prompt processing and wants to know if that is still the case and how others configure llama-server.

Sep 18, 2026

Get a weekly email of new NVIDIA RTX 5060 8GB reports and newly released models that fit it.

Email me new reports

Reported configurations

Individual observations from this page, not expected speeds or a ranking. Select a model to open its report, with the source and offloading conditions. Reported context may be a limit rather than actual input length.

Model / hardwareQuant / engineReported contextGeneration
Qwen3 8B
NVIDIA RTX 5060 8GB
Q4_K_M
llama.cpp
Not reported57.0 tokens/s
Ornith1.5 35B (3B active)
NVIDIA RTX 5060 8GB
Not reported
Engine not reported
Not reported20.0 tokens/s
Qwen3.6 35B (3B active)
NVIDIA RTX 5060 8GB
Q4_K_XL
LM Studio
98,00030.0 tokens/s
Qwen3.8 27B
NVIDIA RTX 5060 8GB
UD-IQ2_XXS
llama.cpp
8,19230.1 tokens/s
DeepSeek V4.1 Flash 552B (16B active)
NVIDIA RTX 5060 8GB
FP8
ds4
8,1922.4 tokens/s
Qwen3.6 35B (3B active)
NVIDIA RTX 5060 8GB
Q4_K_XL
llama.cpp
Not reported40.0 tokens/s

Frequently asked

How fast is the NVIDIA RTX 5060 8GB for local LLMs?

On the NVIDIA RTX 5060 8GB, Qwen3 8B at 4-bit, the model most run on it, generates a median of 57.0 tokens per second from one plain run (community reports, as of 22 Nov 2026). Also reported: Qwen3.8 27B at 2-bit (30.1 t/s, one plain run).

What is the biggest model reported running on the NVIDIA RTX 5060 8GB?

The largest model with a plain generation run on the NVIDIA RTX 5060 8GB is Qwen3.8 27B at 2-bit (30.1 t/s, one plain run), as of 22 Nov 2026. Plain means one device, one request, no speculative decoding, so a run split across several cards does not count.