Local model performance on your hardware

generation:

20.7 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at 20.7 t/s at 127k context on an Intel Arc Pro B70 32GB, down from 70.7 t/s at depth 0. Setup is llama.cpp b10235 (Vulkan) with Q4_K_M, -ngl 999 -ub 512 -fa 0, single card in a PCIe-passthrough VM with 6 vCPU and 16 GB RAM. The user attributes the depth collapse to a memory-latency-bound decode attention kernel, measuring a constant ~21-25 ns per KV position per full-attention layer across backends, models and quants; vLLM XPU on the same card stays flat to 127k. Also reports a silent CPU fallback for prefill score tensors, a SYCL ~16.5 GiB weight-buffer limit, and a ~19% MoE decode regression from b8183 to b10235.

generation:

16.7 tokens/s

prompt processing (prefill):

1047.0 tokens/s

quant:

UD-Q8_K_L (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B-UD-Q8_K_L at 16.66 t/s generation and 1046.98 t/s prompt processing on an Intel Arc Pro B70 GPU. Setup is llama.cpp with the SYCL backend, FP16 build, flash attention off, single GPU, on Windows 11 with a Ryzen 9 5950X host. With flash attention on the same setup gives 16.63 t/s generation and 572.10 t/s prompt processing; FP32 builds give 16.64 t/s and 335.62 t/s without FA, and 16.60 t/s and 300.36 t/s with FA.

generation:

31.2 tokens/s

prompt processing (prefill):

713.7 tokens/s

quant:

Q2_0 (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 31.2 t/s generation and 713.7 t/s prompt processing on an Intel Arc Pro B70 32GB, after overclocking DDR5 memory from 3600MHz to 4800MHz. Setup is llama.cpp with a customized SYCL backend, Q2_0 GGUF quant, q8_0 KV cache, 262144 context, MTP speculative decoding, and 34 MoE layers offloaded to CPU. Overclocking gave about 10% higher prefill and 5% higher generation overall; the post compares base and overclocked llama-benchy numbers across context depths.

generation:

77.7 tokens/s

prompt processing (prefill):

1172.1 tokens/s

quant:

Q4_K_M (GGUF)

kv:

FP16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3 8B at 77.70 t/s generation and 1172.08 t/s prompt processing on an Intel Arc Pro B70 32GB external GPU. Setup is llama.cpp b10852 with Q4_K_M GGUF, FP16 KV cache, flash attention, 512 batch, 10 CPU threads, full GPU offload via SYCL backend on Windows. SYCL repeat measured 83.51 t/s generation; Vulkan and CUDA backends were also tested. A separate 49B capacity test with Llama 3.3 Nemotron Super 49B Q4_K_M achieved 15.99 t/s generation at depth 0 with all 81 layers offloaded.

generation:

54.7 tokens/s

prompt processing (prefill):

615.0 tokens/s

quant:

UD-Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen 3.6-35B-A3B at 54.7 t/s generation and 615 t/s prompt processing on a single Intel Arc Pro B70. Setup is llama.cpp with the SYCL backend, F16 accumulation, and UD-Q4_K_M quant (20.6 GiB). The post is a benchmark table from a GitHub repository; the flagship row is the single-card recommendation. The same table also lists Qwen 3.5-35B-A3B at 54.5 t/s, Gemma 4 26B-A4B at 52.6 t/s, and Qwen 3.5-27B at 20.4 t/s.

generation:

76.0 tokens/s

prompt processing (prefill):

1225.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.6-35B-A3B at 76.0 t/s single-stream decode and 1,225 t/s prefill on one Intel Arc Pro B70 32GB. Setup is llama.cpp with the Vulkan backend, Q4_K_M quant and q4_0 KV cache, no speculative decoding. The host has 32 GB RAM. With --ubatch-size 2048 prefill reaches 1,824 t/s at pp2048 and 8-stream aggregate decode is 624 t/s. The SYCL backend gives 77.3 t/s decode and 979 t/s prefill at pp2048. A dense Qwen3.6-27B Q4_K_XL comparison runs at 20 t/s single stream.

generation:

25.6 tokens/s

prompt processing (prefill):

512.1 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8 27B at 25.6 t/s generation and 512.14 t/s prompt processing on an Intel Arc Pro B70 32GB. Setup is llama.cpp with Q4_K_M weights on Windows 11, comparing SYCL and Vulkan backends; the reported figures are the Vulkan llama-bench results (tg128 25.60 t/s, pp128 512.14 t/s). Vulkan was roughly 3x faster at prompt processing and about 25% faster at generation than SYCL in llama-bench. In a real long business plan generation, Vulkan gave 517.15 t/s prompt processing and 16.17 t/s generation versus SYCL's 238.78 t/s and 17.39 t/s, but the user notes the Vulkan run processed 11,359 prompt tokens against 1,842 on SYCL, so those chat numbers are not apples to apples. The user also finds SYCL 10% to 35% faster on Linux.

generation:

35.6 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 35.6 t/s decode on an Intel Arc Pro B70 (32 GB) with Vulkan, dropping to 4.5 t/s at 64k context. Setup is llama.cpp b10883 with Q4_K_M and q8_0 KV cache, single GPU, full offload, MTP3 draft (Q8_0, n-max 3, p-min 0.10). Decode collapses ~8x from 1k to 64k on Vulkan while SYCL degrades only ~2.4x (38.5 to 15.8 t/s); prefill falls 562 to 101 t/s on Vulkan. An f16-KV control collapses identically, and MTP gain goes negative at depth on Vulkan. A third-party RX 7900 XTX run holds ~58-60 t/s at 28-30k depth.

generation:

63.7 tokens/s

prompt processing (prefill):

977.4 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at 63.66 t/s sustained generation on an Intel Arc Pro B70. Setup is llama.cpp commit 210a657 with SYCL backend, Q4_K_XL quant and q8_0 KV cache, 262144 context. Short bursts peak at ~65 t/s; sustained generation over 1000+ tokens holds at 62-64 t/s. llama-bench on the same machine gives tg128 70.54 t/s and pp512 977.40 t/s.

generation:

70.5 tokens/s

prompt processing (prefill):

977.4 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports creating a poker game without issues. User mentions trying Intel's vLLM fork previously.

generation:

63.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.