Local model performance on your hardware

generation:

124.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks five quants of Qwen3.8-27B on 2x RTX 5070 Ti 16GB, finding standard evals (GSM8K, MMLU-Pro, IFEval) cannot distinguish them within ±2 points. The UD-Q4_K_XL quant reaches 124 t/s decode with MTP and 245,760 tokens of context, versus 93 t/s and 32,768 tokens for Q8_0. Setup is llama.cpp with tensor split, MTP draft and q8_0 KV cache; NVFP4 runs use vLLM TP=2. Q4_K_XL picks a different token than Q8_0 once every 28 tokens by top-1 agreement. On the user's own agentic coding bench (112 runs per model, OpenCode, hidden tests), 27B NVFP4 solves 77% and Flash-Next NVFP4 98%; on the 6 hardest tasks it is 1/12 versus 12/12. SWE-rebench agrees at 69% versus 82% (paired p ≈ 0.007). Flash-Next Q3 on Strata matches NVFP4 on 11 of 12 discriminating tasks but always misses one causality bug.

generation:

66.8 tokens/s

quant:

NVFP4 (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Gemma 4 31B IT NVFP4 at 66.80 tok/s aggregate effective batch throughput on an RTX 5070 Ti 16GB. Setup is llama.cpp with NVFP4 GGUF weights, a Gemma 4 31B Assistant/MTP draft model, --spec-type draft-mtp, --spec-draft-n-max 5, flash attention on, and all layers offloaded to GPU. The benchmark used mtp-bench across 13 mixed tasks with 2,016 predicted tokens in 30.18 s and a 60.2% draft acceptance rate. Aggregate throughput rose from 53.10 tok/s at n_max 2 to 66.80 tok/s at n_max 5, while acceptance fell from 78.2% to 60.2%. Per-task speeds ranged from 35.8 tok/s on creative writing to 104.6 tok/s on repeated patterns. The author notes these are aggregate figures and that no non-speculative baseline was measured.

generation:

111.7 tokens/s

prompt processing (prefill):

4326.8 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at 111.72 tok/s decode and 4,326.76 tok/s prompt eval on an RTX 5070 Ti 16GB. Setup is llama.cpp with Q4_K_M GGUF, F16 KV cache, 65,536 context, cache 132 and ub=8192, single request. Peak VRAM was 14,369 MiB. At NP4 the same model reached 247.79 tok/s aggregate decode (62.00-67.78 per request) and 1,142.11-2,243.00 tok/s prefill per request. Gemma-4-26B-A4B-it Q4_0 runs are also listed, reaching 104.58 tok/s at NP1 and 175.56 tok/s aggregate at NP16.

generation:

28.0 tokens/s

prompt processing (prefill):

400-600 tokens/s

quant:

Q4 (GGUF)

kv:

q8_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next 125B-A6B abliterated at around 28 t/s generation and 400-600 pp on an RTX 5070 Ti 16GB plus RTX A4000 16GB with 128GB system RAM. Setup is stock llama.cpp with Q4 GGUF weights, q8_0 KV cache, 32768 context, 48 GPU layers, tensor-split 64,18 and ncmoe 35, so part of the model runs from system RAM. User is happy with the speed but asks whether the Strata engine can use multiple GPUs and whether it would improve on the 60 t/s others report on 12GB laptops.

generation:

82.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports about 82 tokens/s average decode speed from 8K through 72K context on an RTX 5070 Ti 16GB. Setup is a llama.cpp fork with Adaptive KV Streaming V2 and MTP draft length 3, keeping an effective decode KV pool of around 2.2 GiB even at 256K context. Decode speed drops to about 35 t/s at 144K, 30 t/s at 192K, and 19 t/s at 256K. The user notes MTP requires the same K/V quantization as the main model.

generation:

140.5 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 140.5 tok/s median on 2x RTX 5070 Ti 16GB, up from 69.7 tok/s without speculation. Setup is llama.cpp b10990 with UD-Q4_K_XL GGUF, 32768 context, F16 KV cache, tensor split 1:1, and ungated MTP n-max 6. Combined generation rate across three workloads is 126.74 tok/s (1.82x), aggregate acceptance 0.440, peak GPU memory 20.20 GiB, mean power 482 W. The four-request MTP result is 231.0 aggregate tok/s.

generation:

20.4 tokens/s

prompt processing (prefill):

129.3 tokens/s

quant:

UD-IQ3_XXS (GGUF)

kv:

Q4_0

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 20.44 t/s on an RTX 5070 Ti 16GB. Setup is llama.cpp (unsloth build) with UD-IQ3_XXS GGUF and Q4_0 KV cache, 262144 context, 49 layers offloaded, CPU MoE, and ngram-mod speculative decoding. Prompt processing measured at 129.29 t/s over 2409 tokens.

generation:

104.7 tokens/s

prompt processing (prefill):

2030.0 tokens/s

quant:

IQ3_S (GGUF)

kv:

Q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B abliterated at 104.7 tok/s decode and 2,030 tok/s prefill on a 15,694-token prompt on an RTX 5070 Ti. Setup is a patched llama.cpp build with IQ3_S GGUF, Q4_0 KV cache, 131,072-token context, and embedded MTP speculative decoding with up to 3 draft tokens. At 128K a 128,794-token prompt prefilled at 1,576 tok/s and decoded at 91.3 tok/s; short-context decode reached 130.0-131.6 tok/s. The patch is not upstream and was written only for this model on this GPU.

generation:

95.1 tokens/s

prompt processing (prefill):

1729.5 tokens/s

quant:

Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B peaking at 95.1 t/s generation and 1729.5 t/s prompt processing on an RTX 5070 Ti plus RTX 4070 Super, with means of 73.8 t/s and 845.1 t/s. Setup is llama.cpp with the Unsloth UD-Q4_K_XL GGUF and speculative decoding via draft-dflash plus ngram-mod; Dflash2 with ngram ran 20% faster than MTP with ngram. MTP acceptance was 0.48108 (674 accepted of 1401 generated). User praises the model for coding and vision tasks but notes reasoning makes it slow and token-hungry, with up to 50% of context consumed by reasoning and poisoned tool calls looping at 100k+ context. Caching was finicky and default settings did not work.

generation:

50.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at roughly 50 t/s on an RTX 5070 Ti 16GB with 32GB of DDR5 system RAM, at about 26,000 tokens of context. Setup is llama.cpp with the unsloth UD-IQ4_XS GGUF, a q8_0 KV cache, flash attention on, and an MTP draft model (mtp-Qwen3.8-27B-Q4_0.gguf) for speculative decoding. User is happy with the speed but feels the context window is too small and asks for ways to get a larger one.

generation:

97.0 tokens/s

quant:

APEX Compact (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports a 35B MoE agentic model that beats Qwen3.6 and DeepSeek V4 Pro in some scenarios. The model is available in the SC117/Agents-A1-Uncensored-MTP-APEX-GGUF repository. A 4B dense variant is also mentioned but is not attractive.

generation:

50.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

kv:

Q8_0 K, Q4_0 V

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at Q4 running at 50 t/s on 16 GB VRAM with 200K context. Setup uses adaptive-kv streaming to fit 196,608 tokens, with Q8_0 K and Q4_0 V cache quantization. The user pruned non-ASCII characters from the embedding table and LM head to save 700 MB, offloaded the embedding table to save 270 MB, and disabled MTP.

generation:

72.8 tokens/s

prompt processing (prefill):

1647.0 tokens/s

quant:

Q3_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at Q3_K_XL on an RTX 5070 Ti 16GB with llama.cpp, reaching about 73 t/s at 32k context with MTP on and fully GPU-resident. At-depth generation is 64.5 t/s at 24.1k depth, with prefill at 1647 t/s at depth. MTP off gives about 50 t/s. The practical context ceiling is 40k, or 49k with MTP off. The user also compares Q3_K_XL against Q2_K_XL and finds Q3 faster than its size predicts.

generation:

18-20 tokens/s

quant:

IQ4_XS (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at ~18-20 t/s generation speed on 16 GB VRAM and 64 GB system RAM. Setup uses FFN offloading to keep attention on the GPU, a q8_0 KV cache at 65k context, MTP speculative decoding, and thread pinning.

generation:

36.0 tokens/s

prompt processing (prefill):

560.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports llama.cpp RPC across a 5070 Ti and a 1080 Ti over gigabit ethernet, running Qwen 3.8 27B at UD-Q4_K_XL with a Q8 KV cache and 96k context. Best generation was 36 t/s with MTP enabled and GPU order RPC0,CUDA0. A prefill-focused config gave 560 pp and 19 tg. UD-Q5_K_XL with a full KV cache gave 380 pp and 15 tg. User notes MTP tanks prefill speeds and uses VRAM, and that enabling KV quant for the draft model reduces context. User recommends jumbo frames and careful GPU ordering.

generation:

22.3 tokens/s

prompt processing (prefill):

1003.9 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 22.30 t/s generation and 1003.85 t/s prefill on an RTX 5070 Ti Laptop 12GB plus an RTX 5060 Ti 16GB eGPU. Setup is llama.cpp with layer-split across the two cards. Q4_K_XL measures 1003.85/22.30 t/s pp512/tg128, Q5_K_XL 923.21/19.11, and Q6_K 901.18/18.23. MTP speculative decoding raises generation to 32-38 t/s. The context ceiling is 49,152 for Q6_K and 73,728 for Q5_K_XL. The user also notes tensor-split syntax differences and eGPU power interruption.

generation:

22.0 tokens/s

quant:

Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports the AtomicChat build of Qwen3.8-Flash-Next running a full 256K context on 96 GB RAM and 40 GB VRAM, at 22 t/s. The build splits the n-gram table to SSD. The user previously ran unsloth quants, which crashed at around 60K context and were slower at 12-14 t/s. The AtomicChat build handles a 217K token conversation without crashing. Prefill takes about 18 minutes for 220K context.

generation:

60.0 tokens/s

prompt processing (prefill):

1500.0 tokens/s

quant:

IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports 60 t/s decode and 1500 t/s prefill with Qwen3.8-27B-IQ4-XS-MTP across an RTX 5070 Ti and an RTX 5060 Ti. Setup uses tensor parallelism. User is considering adding a 3090 FE for more VRAM but is concerned about mixing Ampere with Blackwell.

generation:

115.0 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-27B on dual RTX 5070 Ti 16GB cards. Setup is llama.cpp with the Q4_K_XL quant, MTP3 speculative decoding, and 220k context. The user also tests vLLM and NInfer, and praises llama.cpp for its simplicity and performance.

generation:

26.0 tokens/s

quant:

IQ3_XSS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 26t/s on an RTX 5070 Ti 16GB with 64GB of DDR5 6000MHz system RAM. Setup is llama.cpp with the IQ3_XSS quant, using mmap to fit the model into 16G of VRAM plus 64G of RAM, on a 9800X3D CPU. The user says this is faster than a non-MoE 30B model, which only reaches 10 t/s.