NVIDIA RTX 5060 Ti 16GB reports

generation:

9.5-10 tokens/s

prompt processing (prefill):

12-38 tokens/s

quant:

Q3_K_XL (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 9.5-10 t/s decode and 12-38 t/s prefill on dual RTX 5060 Ti 16GB cards with 96GB DDR4 system RAM. Setup is Unsloth Studio (llama-server backend) with UD-Q3_K_XL quant, q4_0 KV cache, 65k context, experts offloaded via --cpu-moe -ngl 99 -b 1024 -ub 512 -fa on, and MTP speculative decoding with an ~87% draft acceptance rate. User asks whether ditching MTP, bypassing mmap with --load-mode none, or building a custom llama.cpp with the LRU expert cache would help on DDR4, or whether the setup is bandwidth-capped.

generation:

100.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Occamy 1.0 at around 100 tok/s on two RTX 5060 Ti cards, with enough memory for two independent ~175K context pools for concurrent subagents. Setup uses llama.cpp on the desktop worker box; the cards cost about $400 each. A 96 GB M5 Ultra Mac Studio is planned as the primary inference appliance running Qwen 3.8 Next Flash through oMLX, with DeepSeek V4.1 Flash via API filling in until it arrives.

generation:

132.0 tokens/s

prompt processing (prefill):

4000.0 tokens/s

quant:

NVFP4

kv:

FP8

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B NVFP4 on dual RTX 5060 Ti 16GB (32GB total VRAM) at 132 t/s decode for structured output and 72 t/s for prose, with ~4,000 tok/s prefill. Setup is stock vLLM 0.30.0 on headless Ubuntu with FP8 KV cache, full 262,144-token context, and MTP speculative decoding. GPUs are power limited to 150W with no throughput cost. Video input is disabled; the user notes this runs without an RTX 5090.

generation:

50.0 tokens/s

quant:

exl3-5.0bpw (EXL3)

kv:

Q8

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at about 50 t/s on an RTX 5060 Ti 16GB plus RTX 3060 12GB, with MTP enabled the rate varies between 40 and 60 t/s. Setup is ExLlamaV3 through TabbyAPI with the turboderp exl3-5.0bpw weights, 102,400 tokens of context, a Q8 KV cache and Q8 draft cache, tensor parallelism across the two cards, and 32GB of system RAM. Without MTP the rate drops to 22 t/s. The user notes 1.3GB free on the 3060 and 0.6GB on the 5060 and is trying to rebalance the GPU split.

generation:

100-110 tokens/s

prompt processing (prefill):

2000.0 tokens/s

quant:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Ornith-1.5-35B at around 100-110 t/s generation and ~2000 t/s prefill on 2x RTX 5060 Ti 16GB (32GB total). Setup uses Q4 quantization; no engine or context length specified. Previously ran Qwen 27B Q4_K_M at around 50 t/s. User asks for fast model recommendations for bulk text extraction and whether a community leaderboard tracks local inference speeds.

generation:

5.3 tokens/s

quant:

IQ4 (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at ~5.3 t/s on an RTX 5060 Ti 16GB with 16GB single-channel system RAM. Setup is llama.cpp with IQ4 weights, 110k context, Q8 KV cache, and partial GPU/CPU offload; CPU and GPU each sit around 50% utilization. User estimates Q8 weights plus 256k context would need 38-40GB total and drop to roughly 2 t/s, and asks whether quant level, context, or throughput should be prioritized for a local coding-agent worker.

generation:

18.3 tokens/s

prompt processing (prefill):

25.4 tokens/s

quant:

Q4 (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next-REAP-320 at 18.3 t/s generation and 25.4 t/s prefill on an RTX 5060 Ti 16GB with 32GB system RAM. Setup is llama.cpp with a Q4 GGUF, 64k context, q4_0 KV cache, --n-cpu-moe 34, --ngl 48, and lazy mmap for the 29.48 GB n-gram embedding. User notes the model is smarter than Qwen3.8 27B but keeps the 27B as daily driver due to 5x faster prompt processing.

generation:

40.0 tokens/s

quant:

Q3 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 40 t/s on a dual-GPU PC with an RTX 5060 Ti 16GB and an RTX 3060 Ti 8GB. Setup uses a Q3 quant and reaches around 160k context window. User asks what the next meaningful step would be for local agentic coding, considering replacing the 3060 Ti with a second 5060 Ti or moving to twin DGX Sparks running a Flash model.

generation:

45.0 tokens/s

prompt processing (prefill):

300.0 tokens/s

quant:

IQ3_XXS (GGUF)

kv:

kvarn4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B running on an RTX 5060 Ti 16GB with vision and 85K context, leaving 1.5GB VRAM headroom. Setup is beellama, a llama.cpp fork, with MTP speculative decoding, the IQ3_XXS-mtp quant from ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, and a kvarn4 KV cache quant. The user notes mmproj could be moved to CPU for more VRAM.

generation:

37.2 tokens/s

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks multiple models on 16 GB of VRAM. Qwen3.6-35B-A3B-APEX-I-Quality reaches 96.7% HumanEval pass@1, 100% agentic success and 37.2 tok/s. Ornith-1.0-35B IQ4_NL runs at 38 tok/s. Other models tested are Qwen3.8-27B variants, gemma-4-26B-A4B, KAT-Coder-V2.5-Dev and Qwen3.5-9B. Ornith 1.0 35B A3B wins overall.

generation:

73.9 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q4

rating:

3/5

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports that adding Mempalace as an MCP to LM Studio did not work properly, fetching memory only sometimes. The same setup used via Copilot Chat in VS Code Insiders may stop responding or modify only a small part of the code.

generation:

76.7 tokens/s

quant:

NVFP4

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports a custom Windows fork of NInfer running TP2 on two RTX 5060 Ti 16GB cards without P2P. Setup uses MTP, with speeds of 35.75 t/s at MTP0, 57.71 t/s at MTP1, 63.15 t/s at MTP2, and 66.79 t/s at MTP3. MTP4 reaches 68.5-68.9 t/s at 512 tokens, 70.6 t/s at 1024 tokens, and 76.65 t/s at 2048 tokens.

generation:

46.0 tokens/s

prompt processing (prefill):

1050.0 tokens/s

quant:

Q4_K_M

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen 27B Q4_K_M on Ollama at 100k context, running on an RTX 5060 Ti and 5070 Ti, with ~1050 t/s prompt processing and ~46 t/s generation on a ~35k token prompt. Setup is Open WebUI with Open Terminal integration on Unraid, with inference on a separate machine. The use case is legal document research and creation.

generation:

18.0 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports a model fits on a single RTX 5060 Ti 16GB at 131k context with a Q4 KV cache. Setup uses GGUF weights only, with no dflash or mmproj loaded. A Q8 KV cache allows about 90k context.

generation:

40.0 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B MoE at about 40 t/s on an RTX 5060 Ti 16GB with mmproj enabled, and just over 50 t/s without mmproj and with --n-cpu-moe 24. Setup is llama.cpp server in Docker with --cache-type-k/v q8_0, --cache-ram 0, --no-mmap, and --n-cpu-moe 19, or 24 without vision. The user experienced an OOM shutdown before adding --cache-ram 0 and is seeking advice on the setup.

generation:

11.0 tokens/s

prompt processing (prefill):

200.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports llama.cpp running on 4x RTX 5060 Ti 16GB with DDR4 3200 RAM in 4-channel. Setup uses -ub/-b at 4096.

generation:

29.8 tokens/s

quant:

IQ3_XXS

kv:

q8

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports a recommended single-card preset for Qwen3.8 27B on an RTX 5060 Ti 16GB, with sustained decode at 45.9K prompt tokens. Two retrieval and two sustained checks passed. User also mentions a 2x RTX 5060 Ti setup with the Q6_K quant at 131K context, reaching 38.6 t/s decode and 597.9 t/s prefill.

generation:

67.3 tokens/s

prompt processing (prefill):

952.5 tokens/s

quant:

NVFP4

kv:

FP8

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User compares NVFP4 on vLLM against Q6_K on llama.cpp, both on 2x RTX 5060 Ti. NVFP4 reaches 952.5 t/s prefill and 67.3 t/s decode with 0.696 MTP acceptance. Q6_K reaches 597.9 t/s prefill and 38.6 t/s decode. On quality, NVFP4 scores 65/68 correct, 64/68 exact format and 61/68 both, while Q6_K scores 66/68, 68/68 and 66/68. Both score 10/10 on long-context retrieval. NVFP4 is the default endpoint, with Q6_K kept for stricter output.

generation:

50-60 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 50-60 t/s on 4x RTX 5060 Ti 16GB. Setup is tensor parallelism with MTP. User asks about memory overclocking risks.

generation:

50-60 tokens/s

quant:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B at 50-60 t/s on two RTX 5060 Ti 16GB cards. Setup is LM Studio with Q4 quantization and MTP. A comparison to Qwen 3.6 gave 30 t/s with MTP, with an OS change from Windows to Linux. The user also downloaded NVFP4 but did not try it in vLLM.