RTX 5060 Ti 16GB

reported speed:

18.3 tokens/s generation · 25.4 tokens/s prompt processing

quant:

Q4 (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next-REAP-320 at 18.3 t/s generation and 25.4 t/s prefill on an RTX 5060 Ti 16GB with 32GB system RAM. Setup is llama.cpp with a Q4 GGUF, 64k context, q4_0 KV cache, --n-cpu-moe 34, --ngl 48, and lazy mmap for the 29.48 GB n-gram embedding. User notes the model is smarter than Qwen3.8 27B but keeps the 27B as daily driver due to 5x faster prompt processing.

reported speed:

40.0 tokens/s generation

quant:

Q3 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8 27B at 40 t/s on a dual-GPU PC with an RTX 5060 Ti 16GB and an RTX 3060 Ti 8GB. Setup uses a Q3 quant and reaches around 160k context window. User asks what the next meaningful step would be for local agentic coding, considering replacing the 3060 Ti with a second 5060 Ti or moving to twin DGX Sparks running a Flash model.

reported speed:

45.0 tokens/s generation · 300.0 tokens/s prompt processing

quant:

IQ3_XXS (GGUF)

kv:

kvarn4

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8-27B running on an RTX 5060 Ti 16GB with vision and 85K context, leaving 1.5GB VRAM headroom. Setup is beellama, a llama.cpp fork, with MTP speculative decoding, the IQ3_XXS-mtp quant from ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, and a kvarn4 KV cache quant. The user notes mmproj could be moved to CPU for more VRAM.

reported speed:

37.2 tokens/s generation

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User benchmarks multiple models on 16 GB of VRAM. Qwen3.6-35B-A3B-APEX-I-Quality reaches 96.7% HumanEval pass@1, 100% agentic success and 37.2 tok/s. Ornith-1.0-35B IQ4_NL runs at 38 tok/s. Other models tested are Qwen3.8-27B variants, gemma-4-26B-A4B, KAT-Coder-V2.5-Dev and Qwen3.5-9B. Ornith 1.0 35B A3B wins overall.

reported speed:

73.9 tokens/s generation

quant:

Q4_K_M (gguf)

kv:

Q4

rating:

3/5

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports that adding Mempalace as an MCP to LM Studio did not work properly, fetching memory only sometimes. The same setup used via Copilot Chat in VS Code Insiders may stop responding or modify only a small part of the code.

reported speed:

76.7 tokens/s generation

quant:

NVFP4

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports a custom Windows fork of NInfer running TP2 on two RTX 5060 Ti 16GB cards without P2P. Setup uses MTP, with speeds of 35.75 t/s at MTP0, 57.71 t/s at MTP1, 63.15 t/s at MTP2, and 66.79 t/s at MTP3. MTP4 reaches 68.5-68.9 t/s at 512 tokens, 70.6 t/s at 1024 tokens, and 76.65 t/s at 2048 tokens.

reported speed:

46.0 tokens/s generation · 1050.0 tokens/s prompt processing

quant:

Q4_K_M

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen 27B Q4_K_M on Ollama at 100k context, running on an RTX 5060 Ti and 5070 Ti, with ~1050 t/s prompt processing and ~46 t/s generation on a ~35k token prompt. Setup is Open WebUI with Open Terminal integration on Unraid, with inference on a separate machine. The use case is legal document research and creation.

reported speed:

18.0 tokens/s generation

quant:

Q4_K_XL (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports a model fits on a single RTX 5060 Ti 16GB at 131k context with a Q4 KV cache. Setup uses GGUF weights only, with no dflash or mmproj loaded. A Q8 KV cache allows about 90k context.

reported speed:

40.0 tokens/s generation

quant:

Q4_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B MoE at about 40 t/s on an RTX 5060 Ti 16GB with mmproj enabled, and just over 50 t/s without mmproj and with --n-cpu-moe 24. Setup is llama.cpp server in Docker with --cache-type-k/v q8_0, --cache-ram 0, --no-mmap, and --n-cpu-moe 19, or 24 without vision. The user experienced an OOM shutdown before adding --cache-ram 0 and is seeking advice on the setup.

reported speed:

11.0 tokens/s generation · 200.0 tokens/s prompt processing

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports llama.cpp running on 4x RTX 5060 Ti 16GB with DDR4 3200 RAM in 4-channel. Setup uses -ub/-b at 4096.

reported speed:

29.8 tokens/s generation

quant:

IQ3_XXS

kv:

q8

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports a recommended single-card preset for Qwen3.8 27B on an RTX 5060 Ti 16GB, with sustained decode at 45.9K prompt tokens. Two retrieval and two sustained checks passed. User also mentions a 2x RTX 5060 Ti setup with the Q6_K quant at 131K context, reaching 38.6 t/s decode and 597.9 t/s prefill.

reported speed:

67.3 tokens/s generation · 952.5 tokens/s prompt processing

quant:

NVFP4

kv:

FP8

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User compares NVFP4 on vLLM against Q6_K on llama.cpp, both on 2x RTX 5060 Ti. NVFP4 reaches 952.5 t/s prefill and 67.3 t/s decode with 0.696 MTP acceptance. Q6_K reaches 597.9 t/s prefill and 38.6 t/s decode. On quality, NVFP4 scores 65/68 correct, 64/68 exact format and 61/68 both, while Q6_K scores 66/68, 68/68 and 66/68. Both score 10/10 on long-context retrieval. NVFP4 is the default endpoint, with Q6_K kept for stricter output.

reported speed:

55.0 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8-27B at 50-60 t/s on 4x RTX 5060 Ti 16GB. Setup is tensor parallelism with MTP. User asks about memory overclocking risks.

reported speed:

55.0 tokens/s generation

quant:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen 3.8 27B at 50-60 t/s on two RTX 5060 Ti 16GB cards. Setup is LM Studio with Q4 quantization and MTP. A comparison to Qwen 3.6 gave 30 t/s with MTP, with an OS change from Windows to Linux. The user also downloaded NVFP4 but did not try it in vLLM.

reported speed:

68.3 tokens/s generation · 41.8 tokens/s prompt processing

quant:

Q6_K (gguf)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8-27B at about 68 t/s generation on dual RTX 5060 Ti. Setup is speculative decoding with a Q8 KV cache at 100k context. Vision is enabled but not used.

reported speed:

55.0 tokens/s generation · 850.0 tokens/s prompt processing

quant:

Q6_K (gguf)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8 27B generating a GTA-style game, with prompt processing of ~850-890 t/s for 25k tokens and generation typically 50-55 t/s, peaking at 197 t/s with ngram speculative decoding. Setup is llama.cpp with ngram and MTP draft, a Q8 KV cache, and 128k context. User is enthusiastic about the results.

reported speed:

31.3 tokens/s generation · 738.7 tokens/s prompt processing

quant:

IQ4_XS (gguf)

kv:

Q4

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8-27B on an RTX 5060 Ti 16GB with llama.cpp, comparing the jpetrina IQ4_XS-pure and Unsloth UD-IQ4_XS quants. Short-context generation runs at roughly 45-47 tok/s, dropping to about 31 tok/s after 55K prefill. The Unsloth quant has better fidelity (KLD 0.018 vs 0.0236), while jpetrina wins the multi-turn agent eval (40/80 vs 38/80). Vision works but needs a separate profile. A Q8 baseline was used as a quality control.

reported speed:

35.0 tokens/s generation

quant:

Q4_K_M (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports a speed gap between LM Studio at 30-40 tok/s and raw llama-server at 50-55 tok/s on the same model and settings. The model is a Q6_K quant. User also asks about reasoning effort dropdown differences between lmstudio-community and Unsloth GGUFs.

reported speed:

20.0 tokens/s generation · 1000.0 tokens/s prompt processing

quant:

q8 (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8-27B q8 GGUF at 20 t/s tg and 1000 pp on a Threadripper rig with 4x RTX 5060 Ti 16GB, VRAM only. User also reports DeepSeek V4 Flash 0731 GGUF at 11 t/s tg and 200 pp with RAM offload. User asks about vLLM tensor parallelism, NVFP4, a CPU upgrade, and other models.

reported speed:

400.0 tokens/s prompt processing

quant:

IQ3_XXS (gguf)

kv:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports over 200k context on 16 GB of VRAM with Qwen 3.8 27B at the IQ3_XXS quant. Setup is a laptop with Thunderbolt 4 and an Aorus 5060 Ti AI Box eGPU on Windows 11, with the KV cache quantized to q5_1. The user previously ran UD-Q3_K_XL at 140k context. Prompt processing dropped from 700-800 tk/s to 400 tk/s.

reported speed:

46.0 tokens/s generation · 700.0 tokens/s prompt processing

quant:

Q4_K_XL (gguf)

kv:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 46 t/s generation and 700 t/s prefill with tensor parallelism enabled, against 28 t/s generation and 1000+ t/s prefill without it. Setup uses MTP with max draft tokens 6 and probability 0.88. Draft acceptance is 0.93757 with a mean length of 5.10.

reported speed:

38.0 tokens/s generation · 1300.0 tokens/s prompt processing

quant:

NVFP4 (gguf)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8-27B-NVFP4-MTP-GGUF at 210k context on 3x RTX 5060 Ti 16GB in a Dell 5820 workstation with 128GB DDR4 RAM, with prompt processing at 1300-700 t/s and generation at 29-71 t/s, mostly 38-44. Setup is llama.cpp with a Q8 KV cache and vision enabled. The user added a third GPU but reports cooling and stability issues, and is considering a second workstation with subagents for Mixture of Agents. The user also tested Ornith 1.5 35B (MXFP) and Gemma 4 26B NVFP on 16GB VRAM. Use cases are web hosting, office documents, and prompt generation for image and video models.

reported speed:

52.2 tokens/s generation · 608.0 tokens/s prompt processing

quant:

Q8

kv:

F16

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 608 t/s cold prefill and 52.2 t/s decode on 4x RTX 5060 Ti 16GB at 256K context. Setup is a Vast AI instance with the Q8 quant, an FP16 KV cache, and MTP enabled. User considers this excellent for $2K hardware.

reported speed:

52.0 tokens/s generation

quant:

float8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 50-54 t/s with a custom CUDA/C++ engine, a 50% improvement over llama.cpp at 33-34 t/s.

reported speed:

19.0 tokens/s generation

quant:

IQ4_XS (gguf)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports 19 t/s peak and 14 t/s during long generation at 65k context with the KV cache offloaded to RAM via -nkvo, which allows the whole model on GPU with an f16 KV cache. Speed remained similar at 128k context with 63 layers on GPU. Quantizing the KV cache to RAM did not improve performance.

reported speed:

22.0 tokens/s generation · 760.0 tokens/s prompt processing

quant:

IQ4_XS (gguf)

kv:

Q8

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.6 27B IQ4_XS at 22 t/s generation and 760 t/s prompt processing on an RTX 5060 Ti 16GB. Setup is llama.cpp with TheTom's TurboQuant fork, a 75k context window, and a turbo4/turbo2 KV cache. The user also tested BF16, Q8_0, Q6_K, Q5_K_XL, Q4_K_XL, IQ4_XS, IQ3_XXS, Q3_K_XL, Q3_K_M and Q2_K_XL on an L40S or RTX 5060 Ti, comparing quality with a chess board SVG generation task, and recommends IQ4_XS as the minimum.