NVIDIA RTX 5070

generation:

4.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B at about 4 tokens/s on an RTX 5070 in a laptop with a Core Ultra 9 275HX and 32GB DDR5. Setup is llama.cpp with Q4_K_M, launched with -ngl auto; anything else crashes. The run uses 100% of system RAM, about 25% of the iGPU and 0-3% of the RTX 5070, so the model is not running on the discrete GPU.

generation:

53.0 tokens/s

prompt processing (prefill):

1620.0 tokens/s

quant:

IQ3_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 53 tokens/s generation and 1,620 tokens/s prompt processing on an RTX 5070 12GB with 64GB RAM. Setup is the Strata engine with IQ3_S quantization at 128K context; the model's experts are split across GPU, RAM and SSD. The page also lists Q2_0 at 94/2,650, IQ2_XS at 79/2,090, IQ3_XXS at 62/1,750 and Coder at 55/2,180 tokens/s on the same card, and RX 9070 XT 16GB figures of 60/1,160 (Q2_0), 52/1,110 (IQ2_XS) and 44/1,420 (Coder). Unsloth UD-Q4_K_XL is given as 7-8.5 tokens/s on a 64GB PC.

generation:

11.5 tokens/s

quant:

UD-IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next 177B running at 11-15 tok/s on a single RTX 5070 12GB with 32GB DDR4-2400 RAM and a Ryzen 5 5600GT on Windows. Setup is llama.cpp with UD-IQ3_XXS quant and expert streaming from system RAM, using one file handle per worker and a page-locked hot-expert tier. Benchmark is about 11.5 tok/s, up from roughly 7 tok/s on the inherited setup. Normal conversations run 14-15 tok/s, and a long coding prompt generated 4,892 tokens at 10.15 tok/s. A video shows around 16 minutes for 10k tokens at 10.41 tok/s.

generation:

44.8 tokens/s

prompt processing (prefill):

414.0 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 44.8 t/s generation and 414 t/s prompt processing on a 12GB RTX 5070 at 128K context. Setup is a custom Strata inference engine with the IQ3_XXS GGUF quant, 64GB DDR5-5600 and a Ryzen 5 7600 on Windows. The engine is CUDA-only and requires 47GB minimum in RAM+VRAM for this quant. A Q2_0 quant reached 65.1 t/s generation and 543 t/s prompt processing, and IQ2_XS reached 52.0 t/s generation and 472 t/s prompt processing. The user previously measured 15 t/s output and 100-120 t/s prompt processing with llama.cpp on the same hardware.

prompt processing (prefill):

1800.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports that after AUR llama.cpp-cuda package deletion, binary packages (ggml-cuda and llama.cpp) are ~10% speed, with model loaded to GPU but processing on CPU. Custom build achieves 1800 t/s prefill vs 150 t/s. GPU is RTX 5070.

generation:

35.0 tokens/s

prompt processing (prefill):

1000.0 tokens/s

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.6 35B-A3B at 35 t/s generation and 1000 t/s prompt processing on an RTX 5070 12GB with an i5-10400F and 32GB RAM. Setup is LM Studio with a 64k context, Q8 KV cache, flash attention, about 30 experts offloaded to CPU, and retain-thinking enabled. The run used about 10GB VRAM and 28GB system RAM at roughly 58K context. User asks whether more performance can be squeezed from the rig and reports inconsistent context reporting between Cline and OpenCode.

generation:

15.0 tokens/s

prompt processing (prefill):

104.7 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 15 t/s on an RTX 5070 12GB, with prompt processing at 100-120 t/s. Setup is llama.cpp with a 3 bpw IQ3_XXS GGUF (76GB total, 47GB sharded into VRAM+RAM, rest on SSD) at up to 128K context. Generation held 13.8 t/s at 1K, 14.6 t/s at 8K, 14.0 t/s at 20K, and 11.3 t/s at 90-100K context. User rates output quality as matching Unsloth Q6-Q8.

generation:

3.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User asks about Intel Arc B60 or B65 for Qwen 3.8 27B, currently getting 3 t/s on an RTX 5070. User mentions a custom vLLM fork for Intel Arc reaching about 20 t/s on the larger B70.

generation:

22.0 tokens/s

prompt processing (prefill):

25.0 tokens/s

quant:

IQ1_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 22.0 t/s generation and 24.96 t/s prompt processing on a single RTX 5070 with 12 GB of VRAM. Setup is llama.cpp with the IQ1_S quant at 10,000 tokens of context.