Local model performance on your hardware

generation:

60.8 tokens/s

prompt processing (prefill):

1104.0 tokens/s

quant:

Q4_0 (GGUF)

kv:

q8_0

User reports Ornith 1.5 35B-A3B at 60.8 t/s decode and 1104 t/s prefill at zero depth on a single Tesla P40 24GB. Setup is a custom llama.cpp fork with Q4_0 weights, q8_0 KV cache, full 262k context, on Windows over PCIe x1; the llama-bench table is without MTP. Decode falls to 46.0 t/s at 32k, 29.1 t/s at 100k and 18.8 t/s at 190k. With MTP (draft n=2) the real server gives 65-74 t/s on short prompts, and a 101k-token prompt reads at ~415 t/s and generates at 35-39 t/s. ECC off added 1.5 GB VRAM and ~9% decode; q4 KV gave 30.6 t/s versus 42.2 t/s for q8 at 32k.

generation:

5.8 tokens/s

prompt processing (prefill):

10.7 tokens/s

quant:

Q2_K_XL (GGUF)

kv:

f16

User reports Kimi K2.5 at 5.85 t/s generation and 10.67 t/s prompt processing on eight Tesla P40 24GB cards. Setup is llama.cpp with a Q2_K_XL GGUF (349.01 GiB, 2.92 BPW) at 262144 context, f16 KV cache, flash attention on, experts offloaded to CPU. User says smaller MoE models reach over 20 t/s and Kimi K2.5 gives 5+ t/s; a second request measured 5.47 t/s generation and 7.48 t/s prompt processing.

generation:

34.8 tokens/s

prompt processing (prefill):

501.0 tokens/s

quant:

IQ2_XS (GGUF)

User benchmarks Qwen3.8 Flash-Next at 34.8 tok/s decode on two Tesla P40 24GB cards with a layer split, and 19.6 tok/s on a single P40. Setup is Strata v0.1.39 with IQ2_XS weights at 32,768 context, experimental CUDA 12 engine built locally; the two-card run splits layers across both GPUs (K=24, all 24576 profiled expert pairs resident). Prompt reading was 353 / 501 tok/s at 4K / 16K on both cards and 348 / 388 tok/s on one; figures are medians of 3 runs per cell, with the first run of each configuration slower due to expert-cache warm-up. A 69-task correctness suite ran twice, and two greedy runs of identical prompts gave 49 of 69 identical outputs.

generation:

23.7 tokens/s

quant:

IQ4_XS (GGUF)

kv:

turbo3

User reports Qwen 3.6 27B at 23.66 tok/s decode on a Tesla P40 24GB. Setup is llama.cpp with IQ4_XS GGUF, turbo3 KV cache, MTP self-speculative decoding (N=2, p_min=0.5), 65536 context, single card. MTP accept rate 83.7%, a +28.0% gain over the non-MTP Q4_K_M baseline at 18.48 tok/s. A cache sweep found turbo3 best at 22.43 tok/s and 18.1 GB VRAM, while q8_0 broke MTP with 0% draft acceptance.

generation:

9.7 tokens/s

quant:

int4 gs=64

User reports Qwen3.6-35B-A3B at 9.67 tok/s decode on a single Tesla P40 24GB, with a cold run at 9.63 tok/s. Setup is colibri's CUDA backend built for sm_61, int4 gs=64 weights, 200 tokens greedy, single stream, whole model resident in VRAM (10240/10240 experts, 100% hit, 0 CPU misses). The P40 is slower than the P100 once warm (9.67 vs 12.02 tok/s) on ~346 GB/s GDDR5 versus ~732 GB/s HBM2; the cold-to-warm gap is 0.4% versus the P100's 61%. Probe-off control gives 9.62 tok/s and byte-identical output.

generation:

37.9 tokens/s

prompt processing (prefill):

982.0 tokens/s

quant:

Q3_K_XL (GGUF)

User reports Qwen3.6-35B-A3B at Q3_K_XL generating 37.9 tok/s on a single Tesla P40 24GB under WSL2 with driver 582.70 in the MCDM driver model. Setup is llama.cpp built with CUDA arch 61, GGML_CUDA_FORCE_MMQ=ON and GGML_CUDA_F16=OFF; llama-bench pp512 measures 982 tok/s prefill. With MTP speculative decoding at --spec-draft-n-max 2 generation rises to 51.6 tok/s (n-max 3: 48.3, n-max 4: 41.9). Prefill falls to about 790 tok/s at 6K tokens, 422-609 at 25K and about 216 at 78K. With MTP n-max 2, 4096 context with f16 KV uses 21710 MiB at 51.45 tok/s, 32768 with q8_0 KV uses 22086 MiB at 50.79 tok/s, and 131072 with q8_0 KV uses 22782 MiB at 50.54 tok/s.

generation:

51.6 tokens/s

prompt processing (prefill):

982.0 tokens/s

quant:

UD-Q3_K_XL (GGUF)

kv:

Q8_0

User reports Qwen3.6-35B-A3B at 51.6 tok/s on a Tesla P40 24GB under WSL2. Setup is llama.cpp built for sm_61 with UD-Q3_K_XL (16.85 GB, 17.23 GB with MTP), full GPU offload, MTP speculative decoding at --spec-draft-n-max 2, and Q8_0 KV cache at 131072 context. Baseline without speculation is 37.9 tok/s; MTP with speculation disabled lands at 38.3 tok/s, confirming the speedup is real. Prefill is 982 tok/s (pp512). Context scaling from 4096 to 131072 costs about 1 GB VRAM and about 2% throughput. Needle-in-a-haystack passed 9 of 9 at 10/50/90 percent depth, with 6315-token prompts at about 790 tok/s prefill and 25111-token prompts at 422-609 tok/s prefill. All generation figures are single-stream.

generation:

45.0 tokens/s

prompt processing (prefill):

450.0 tokens/s

User reports Qwen3.8 27B at up to 45 t/s generation and 450 t/s prefill on 2x Tesla P40 with fresh context. At 150K+ context prefill falls to around 120 t/s and generation to 12-16 t/s. User notes that running two agents concurrently breaks prefix caching and KV cache sharing, and asks whether a single-supervisor harness with sequential subagent handoffs exists.

generation:

91.0 tokens/s

quant:

Q8_K_XL (GGUF)

User reports Qwen 3.8 27B at up to 91 t/s with MTP on 3x RTX 3090 plus a Tesla P40 and 128 GB of system RAM. Setup is the UD-Q8_K_XL quant. Thinking time varies by setting: low about 3s, medium about 3min, xHigh about 15min. The user compares it to Qwen 3.6 27B and DeepSeek V4 Flash, and says Qwen 3.8 produced a much better Galaga recreation with detailed sprites, sound effects, and a capture system. The user notes the high thinking time may be an issue for slower setups.

generation:

46.0 tokens/s

prompt processing (prefill):

444.4 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

User reports about 46 t/s on typical short interactive decode in use on dual P40s, up from roughly 15 t/s without speculative decoding. Setup is MTP speculative decoding with draft-mtp and ngram-simple, draft maximum 6, Qwen reasoning medium and the F16 vision projector loaded. A synthetic 128-token decode reached 48 t/s, long-context interactive decode runs at about 20 t/s, and prefill is 444 t/s at pp512. The model is a fine-tune of Qwen3.8 27B.