NVIDIA RTX 5070 Ti reports

generation:

111.7 tokens/s

prompt processing (prefill):

4326.8 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at 111.72 tok/s decode and 4,326.76 tok/s prompt eval on an RTX 5070 Ti 16GB. Setup is llama.cpp with Q4_K_M GGUF, F16 KV cache, 65,536 context, cache 132 and ub=8192, single request. Peak VRAM was 14,369 MiB. At NP4 the same model reached 247.79 tok/s aggregate decode (62.00-67.78 per request) and 1,142.11-2,243.00 tok/s prefill per request. Gemma-4-26B-A4B-it Q4_0 runs are also listed, reaching 104.58 tok/s at NP1 and 175.56 tok/s aggregate at NP16.

generation:

54.9 tokens/s

quant:

IQ2_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 54.9 tok/s decode on an RTX 5070 Ti 16 GB, with 33 GiB of experts kept in RAM. Setup is EuLLM (llama.cpp b11370 + PR #29887) with IQ2_XS, 2048-token micro-batch, and a GPU expert cache; the model is 125B total with about 6B active per token. The same PC reaches 85.8 tok/s with Strata and MTP, and prompt processing peaks at 1,500–1,743 tok/s with prefetch versus 2,320 tok/s for Strata. MTP loses 27% on this model with 52% draft acceptance, and gains only +7% prose / +16% code on Qwen3.6-35B-A3B.

generation:

28.0 tokens/s

prompt processing (prefill):

400-600 tokens/s

quant:

Q4 (GGUF)

kv:

q8_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next 125B-A6B abliterated at around 28 t/s generation and 400-600 pp on an RTX 5070 Ti 16GB plus RTX A4000 16GB with 128GB system RAM. Setup is stock llama.cpp with Q4 GGUF weights, q8_0 KV cache, 32768 context, 48 GPU layers, tensor-split 64,18 and ncmoe 35, so part of the model runs from system RAM. User is happy with the speed but asks whether the Strata engine can use multiple GPUs and whether it would improve on the 60 t/s others report on 12GB laptops.

generation:

180.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Ornith 1.5 35B-A3B at about 180 tok/s on two RTX 5070 Ti 16GB cards, roughly 3x the 55-70 tok/s of the Qwen3.8 27B it replaced. Setup is Ollama on Windows with Q4_K_M at 128K context, 24.4GB fully on the two cards; the agent runs on pi in WSL. Two runs gave 176 and 183 tok/s with a short prompt and thinking off. At 256K about 1.2GB spills to system RAM and speed drops to about 139 tok/s, so 128K stays the default. Both of the user's agent tests max out at 9/9 and 10/10, which the user notes only shows it is not worse than the 27B, not that it is smarter. Vendor numbers of 79 on SWE-bench Verified and 68.5 on Terminal-Bench 2.1 are cited but untested.

generation:

53.5 tokens/s

quant:

IQ3_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 53.5 tok/s with a cached prefix on an RTX 5070 Ti 16 GB, up from 17.2 tok/s before tuning. Setup is Strata with IQ3_S at 262K context, KV cache in RAM streaming to a 32K VRAM window, speculative decoding with depth 6 and min-p 0.70, and 13 pool workers on a 14700KF with 96 GB RAM. Cold run was 43 tok/s; the 53.5 figure is with the prefix cached, which the user says is what agent sessions actually do. The user also reports the calibrator's own curve at 42.1 tok/s at --pcie-frac 0.0 down to 20.7 at 0.75.

generation:

180-290 tokens/s

quant:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User compares local model choices by memory budget, recommending Qwen3.8-27B at Q4 for 32 GB or more of VRAM and Qwen3.8-Flash-Next on Strata otherwise. On 2x RTX 5070 Ti 16 GB with 96 GB RAM, the user measures Qwen3.8-27B under ninfer-tp2 with MTP/DFlash at roughly 180-290 t/s. Other reported figures include a single 5090 at about 150-210 t/s with MTP3 and 260-290 t/s on code with DFlash2, a single R9700 at about 50-65 t/s under llama.cpp or vLLM with MTP, and 2x R9700 at about 185 t/s under vLLM MXFP4 with speculative decoding. For Qwen3.8-Flash-Next on Strata the user measures 62-72 t/s decode on one GPU and 80-90 t/s on two, with prefill around 2,400 and 3,000 t/s at 16k.

generation:

53.0 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User compares Qwen3.8 27B base, Swift 1.5, and Bonsai 2 on a 16GB RTX 5070 Ti. Swift 1.5 IQ3_XXS at 112K context is judged the best all-round option, nearly matching base on coding and scoring higher on math. Bonsai 2 generated about 1.9x faster (99.6 t/s vs roughly 53 t/s) and supported 256K context but solved fewer coding problems. Base and Swift passed the same 2/4 Next.js tasks; Bonsai passed none.

generation:

108-110 tokens/s

prompt processing (prefill):

1683-4323 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3 30B-A3B at 108-110 t/s decode and 1683-4323 t/s prefill on an RTX 5070 Ti 16 GB. Setup is Ollama 0.35.0 with Q4_K_M at 8192 context, median of 2 runs, measured idle. The model produced no compilable code (0/2), failed one-word obedience with 899 characters, and returned 0 characters on the reasoning test after burning its 4000-token budget on 15088 characters of thinking. User notes prefill swings 2x with background load and that the fastest model here measures verbosity.

generation:

82.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports about 82 tokens/s average decode speed from 8K through 72K context on an RTX 5070 Ti 16GB. Setup is a llama.cpp fork with Adaptive KV Streaming V2 and MTP draft length 3, keeping an effective decode KV pool of around 2.2 GiB even at 256K context. Decode speed drops to about 35 t/s at 144K, 30 t/s at 192K, and 19 t/s at 256K. The user notes MTP requires the same K/V quantization as the main model.

generation:

220.0 tokens/s

prompt processing (prefill):

4956.0 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 220.0 tok/s decode with MTP3 structured output on 2x RTX 5070 Ti, matching an RTX 5090's 219.8 tok/s on the same official NVFP4 weights. Setup is the NInfer tensor-parallel fork v0.2.3 with NVFP4 weights, 7,680-token context, C=1, uncapped clocks, no P2P over PCIe 5.0 x8/x8. Without speculation decode is 70.9 tok/s and prefill 4,956 tok/s (59% of the 5090's 8,340). Against llama.cpp on the same two cards: cold prefill 4,577 vs 2,197 tok/s, decode at 184k context 122.9 vs 64.0 tok/s, and a 16-turn agent session to ~94k context 46.5 s vs 82.8 s. QUASAR-QAT weights reach 254.5 tok/s decode but are lighter (17.0 vs 20.9 GiB).

generation:

140.5 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 140.5 tok/s median on 2x RTX 5070 Ti 16GB, up from 69.7 tok/s without speculation. Setup is llama.cpp b10990 with UD-Q4_K_XL GGUF, 32768 context, F16 KV cache, tensor split 1:1, and ungated MTP n-max 6. Combined generation rate across three workloads is 126.74 tok/s (1.82x), aggregate acceptance 0.440, peak GPU memory 20.20 GiB, mean power 482 W. The four-request MTP result is 231.0 aggregate tok/s.

generation:

20.4 tokens/s

prompt processing (prefill):

129.3 tokens/s

quant:

UD-IQ3_XXS (GGUF)

kv:

Q4_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 20.44 t/s on an RTX 5070 Ti 16GB. Setup is llama.cpp (unsloth build) with UD-IQ3_XXS GGUF and Q4_0 KV cache, 262144 context, 49 layers offloaded, CPU MoE, and ngram-mod speculative decoding. Prompt processing measured at 129.29 t/s over 2409 tokens.

generation:

104.7 tokens/s

prompt processing (prefill):

2030.0 tokens/s

quant:

IQ3_S (GGUF)

kv:

Q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B abliterated at 104.7 tok/s decode and 2,030 tok/s prefill on a 15,694-token prompt on an RTX 5070 Ti. Setup is a patched llama.cpp build with IQ3_S GGUF, Q4_0 KV cache, 131,072-token context, and embedded MTP speculative decoding with up to 3 draft tokens. At 128K a 128,794-token prompt prefilled at 1,576 tok/s and decoded at 91.3 tok/s; short-context decode reached 130.0-131.6 tok/s. The patch is not upstream and was written only for this model on this GPU.

generation:

40.0 tokens/s

quant:

Q8

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User asks whether Qwen3.8 Flash-Next can run on a mixed RTX 5070 Ti 16GB plus Radeon AI PRO R9700 32GB box with 96GB DDR5 system RAM. They report Qwen3.6 27B already runs on both cards with layer split at 40 tok/s using Q8 weights and Q8 KV cache at 256k context. They also mention losing two days to an AMD Pro driver unload bug that freezes the system, and ask for quant, tok/s, context size, launch flags, and mixed NVIDIA/AMD advice.

generation:

82.2 tokens/s

prompt processing (prefill):

6488.4 tokens/s

quant:

FP8 (FP8)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8 27B FP8 on a quad RTX 5070 Ti 16GB node, reporting 82.21 t/s generation and 6488.44 t/s prompt processing at depth 0. Setup is vLLM 0.30 on Ubuntu 24.04 with CUDA 13.2, tensor-parallel-size 4, 65536 max model length, 0.85 GPU memory utilization, and no speculative decoding. GPUs are connected through a PLX 88096 PCIe switch with P2P enabled within the node. A second quad RTX 5060 Ti 16GB node reached 43.35 t/s generation and 3218.53 t/s prompt processing on the same model. Throughput scales roughly linearly with GPU tier, and the user notes results are stock with no overclock or undervolt.

generation:

95.1 tokens/s

prompt processing (prefill):

1729.5 tokens/s

quant:

Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B peaking at 95.1 t/s generation and 1729.5 t/s prompt processing on an RTX 5070 Ti plus RTX 4070 Super, with means of 73.8 t/s and 845.1 t/s. Setup is llama.cpp with the Unsloth UD-Q4_K_XL GGUF and speculative decoding via draft-dflash plus ngram-mod; Dflash2 with ngram ran 20% faster than MTP with ngram. MTP acceptance was 0.48108 (674 accepted of 1401 generated). User praises the model for coding and vision tasks but notes reasoning makes it slow and token-hungry, with up to 50% of context consumed by reasoning and poisoned tool calls looping at 100k+ context. Caching was finicky and default settings did not work.

generation:

50.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at roughly 50 t/s on an RTX 5070 Ti 16GB with 32GB of DDR5 system RAM, at about 26,000 tokens of context. Setup is llama.cpp with the unsloth UD-IQ4_XS GGUF, a q8_0 KV cache, flash attention on, and an MTP draft model (mtp-Qwen3.8-27B-Q4_0.gguf) for speculative decoding. User is happy with the speed but feels the context window is too small and asks for ways to get a larger one.

generation:

97.0 tokens/s

quant:

APEX Compact (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports a 35B MoE agentic model that beats Qwen3.6 and DeepSeek V4 Pro in some scenarios. The model is available in the SC117/Agents-A1-Uncensored-MTP-APEX-GGUF repository. A 4B dense variant is also mentioned but is not attractive.

generation:

50.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

kv:

Q8_0 K, Q4_0 V

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at Q4 running at 50 t/s on 16 GB VRAM with 200K context. Setup uses adaptive-kv streaming to fit 196,608 tokens, with Q8_0 K and Q4_0 V cache quantization. The user pruned non-ASCII characters from the embedding table and LM head to save 700 MB, offloaded the embedding table to save 270 MB, and disabled MTP.

generation:

72.8 tokens/s

prompt processing (prefill):

1647.0 tokens/s

quant:

Q3_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at Q3_K_XL on an RTX 5070 Ti 16GB with llama.cpp, reaching about 73 t/s at 32k context with MTP on and fully GPU-resident. At-depth generation is 64.5 t/s at 24.1k depth, with prefill at 1647 t/s at depth. MTP off gives about 50 t/s. The practical context ceiling is 40k, or 49k with MTP off. The user also compares Q3_K_XL against Q2_K_XL and finds Q3 faster than its size predicts.