Local model performance on your hardware

generation:

51.0 tokens/s

prompt processing (prefill):

838.0 tokens/s

quant:

Q8_0 (GGUF)

User reports Qwen3.8-27B at 51 t/s average decode and 838 t/s prompt processing across about 1,030 real agentic coding requests on 2x Arc Pro B70 32GB. Setup is llama.cpp SYCL with tensor parallel across both cards, Q8_0 weights, a DFlash2 drafter at Q4_K_M, 200K context and temp 1.0 on Windows 11. Decode ranged from 47 t/s at 40-80K context to 61 t/s at 160K+, rising at high context because more drafts get accepted; a fresh 94K prompt prefills at about 1,070 t/s. The user also reports a oneAPI 2026.1 runtime leak that pushed private memory to 92 GB and slowed decode from 64 to 200 ms per cycle, fixed with one SYCL context per GPU.

generation:

20.0 tokens/s

prompt processing (prefill):

450.0 tokens/s

quant:

FP8 (FP8)

mtp (multi-token prediction):

on

User reports Qwen3.8 Flash-Next FP8 with MTP at 450 t/s prefill and 20 t/s generation on 4x Intel Arc Pro B70 (128GB VRAM total). Setup is vLLM in a container with 128GB system RAM, using the qfn-38-fp8-4xb70-mtp configuration. The figures come from LLM-run tests and the user notes it is slower than some benchmarks found elsewhere; prior manual vLLM attempts on multiple B70 cards crashed for almost every model until 2^32 overflow fixes were backported.

generation:

70.0 tokens/s

prompt processing (prefill):

2800.0 tokens/s

quant:

W4A16-AutoRound (AutoRound)

kv:

FP8

User reports Qwen3.8 Flash-Next at ~70 t/s generation and ~2800 t/s prefill on 4x Intel Arc Pro B70 GPUs. Setup is vLLM with Intel's W4A16-AutoRound quant, full 262K FP8 context, prefix caching, and concurrency=2. The user notes an alternative setup reached ~100 t/s generation but only ~1400 t/s prefill and 16-bit context, and considers the doubled prefill and context worth the ~30 t/s generation loss.

generation:

91.2 tokens/s

prompt processing (prefill):

2421.0 tokens/s

quant:

EXL3 4.00bpw (EXL3)

kv:

fp8

User reports Qwen3.8-27B EXL3 4.00bpw at 91.2 t/s decode (C1, prose) on one Intel Arc Pro B70 32GB. Setup is vLLM 0.26.1 XPU with EXL3 trellis quantization, fp8 KV cache, 262,144 context, MTP speculative decoding k=3, int8 prefill and oneDNN fused attention. Aggregate throughput scales to 365.2 t/s at C16; cold prefill is 2,421 t/s at 4K and 911 t/s at 254K. The same card with tuned llama.cpp SYCL Q4_K_M reaches 25.0 t/s at C1 and 999 t/s prefill at 4K.

generation:

20.7 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3.6-35B-A3B at 20.7 t/s at 127k context on an Intel Arc Pro B70 32GB, down from 70.7 t/s at depth 0. Setup is llama.cpp b10235 (Vulkan) with Q4_K_M, -ngl 999 -ub 512 -fa 0, single card in a PCIe-passthrough VM with 6 vCPU and 16 GB RAM. The user attributes the depth collapse to a memory-latency-bound decode attention kernel, measuring a constant ~21-25 ns per KV position per full-attention layer across backends, models and quants; vLLM XPU on the same card stays flat to 127k. Also reports a silent CPU fallback for prefill score tensors, a SYCL ~16.5 GiB weight-buffer limit, and a ~19% MoE decode regression from b8183 to b10235.

generation:

16.7 tokens/s

prompt processing (prefill):

1047.0 tokens/s

quant:

UD-Q8_K_L (GGUF)

User reports Qwen3.8-27B-UD-Q8_K_L at 16.66 t/s generation and 1046.98 t/s prompt processing on an Intel Arc Pro B70 GPU. Setup is llama.cpp with the SYCL backend, FP16 build, flash attention off, single GPU, on Windows 11 with a Ryzen 9 5950X host. With flash attention on the same setup gives 16.63 t/s generation and 572.10 t/s prompt processing; FP32 builds give 16.64 t/s and 335.62 t/s without FA, and 16.60 t/s and 300.36 t/s with FA.

generation:

31.2 tokens/s

prompt processing (prefill):

713.7 tokens/s

quant:

Q2_0 (GGUF)

kv:

q8_0

User reports Qwen3.8-Flash-Next at 31.2 t/s generation and 713.7 t/s prompt processing on an Intel Arc Pro B70 32GB, after overclocking DDR5 memory from 3600MHz to 4800MHz. Setup is llama.cpp with a customized SYCL backend, Q2_0 GGUF quant, q8_0 KV cache, 262144 context, MTP speculative decoding, and 34 MoE layers offloaded to CPU. Overclocking gave about 10% higher prefill and 5% higher generation overall; the post compares base and overclocked llama-benchy numbers across context depths.

generation:

77.7 tokens/s

prompt processing (prefill):

1172.1 tokens/s

quant:

Q4_K_M (GGUF)

kv:

FP16

User benchmarks Qwen3 8B at 77.70 t/s generation and 1172.08 t/s prompt processing on an Intel Arc Pro B70 32GB external GPU. Setup is llama.cpp b10852 with Q4_K_M GGUF, FP16 KV cache, flash attention, 512 batch, 10 CPU threads, full GPU offload via SYCL backend on Windows. SYCL repeat measured 83.51 t/s generation; Vulkan and CUDA backends were also tested. A separate 49B capacity test with Llama 3.3 Nemotron Super 49B Q4_K_M achieved 15.99 t/s generation at depth 0 with all 81 layers offloaded.

generation:

54.7 tokens/s

prompt processing (prefill):

615.0 tokens/s

quant:

UD-Q4_K_M (GGUF)

User benchmarks Qwen 3.6-35B-A3B at 54.7 t/s generation and 615 t/s prompt processing on a single Intel Arc Pro B70. Setup is llama.cpp with the SYCL backend, F16 accumulation, and UD-Q4_K_M quant (20.6 GiB). The post is a benchmark table from a GitHub repository; the flagship row is the single-card recommendation. The same table also lists Qwen 3.5-35B-A3B at 54.5 t/s, Gemma 4 26B-A4B at 52.6 t/s, and Qwen 3.5-27B at 20.4 t/s.

generation:

175.3 tokens/s

prompt processing (prefill):

9698.5 tokens/s

quant:

MXFP4 (MXFP4)

User reports Ornith-1.5-35B-A3B at 175.25 tok/s generation and 9,698.55 tok/s prompt processing at concurrency 1 on an Intel Arc Pro B70. Setup is GRIMOIRE, a native C++/SYCL inference engine, with MXFP4 weights and 4096-token prompts. At concurrency 8 the engine reaches 391.04 tok/s total generation and 10,154.18 tok/s total prompt processing. A separate v1.5 build measured 193.6 tok/s at c1 and 464.2 tok/s total at c8, with pp512 at c8 measuring 9,547 tok/s.

generation:

76.0 tokens/s

prompt processing (prefill):

1225.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q4_0

User benchmarks Qwen3.6-35B-A3B at 76.0 t/s single-stream decode and 1,225 t/s prefill on one Intel Arc Pro B70 32GB. Setup is llama.cpp with the Vulkan backend, Q4_K_M quant and q4_0 KV cache, no speculative decoding. The host has 32 GB RAM. With --ubatch-size 2048 prefill reaches 1,824 t/s at pp2048 and 8-stream aggregate decode is 624 t/s. The SYCL backend gives 77.3 t/s decode and 979 t/s prefill at pp2048. A dense Qwen3.6-27B Q4_K_XL comparison runs at 20 t/s single stream.

generation:

25.6 tokens/s

prompt processing (prefill):

512.1 tokens/s

quant:

Q4_K_M (GGUF)

User benchmarks Qwen3.8 27B at 25.6 t/s generation and 512.14 t/s prompt processing on an Intel Arc Pro B70 32GB. Setup is llama.cpp with Q4_K_M weights on Windows 11, comparing SYCL and Vulkan backends; the reported figures are the Vulkan llama-bench results (tg128 25.60 t/s, pp128 512.14 t/s). Vulkan was roughly 3x faster at prompt processing and about 25% faster at generation than SYCL in llama-bench. In a real long business plan generation, Vulkan gave 517.15 t/s prompt processing and 16.17 t/s generation versus SYCL's 238.78 t/s and 17.39 t/s, but the user notes the Vulkan run processed 11,359 prompt tokens against 1,842 on SYCL, so those chat numbers are not apples to apples. The user also finds SYCL 10% to 35% faster on Linux.

generation:

35.6 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q8_0

User reports Qwen3.8 27B at 35.6 t/s decode on an Intel Arc Pro B70 (32 GB) with Vulkan, dropping to 4.5 t/s at 64k context. Setup is llama.cpp b10883 with Q4_K_M and q8_0 KV cache, single GPU, full offload, MTP3 draft (Q8_0, n-max 3, p-min 0.10). Decode collapses ~8x from 1k to 64k on Vulkan while SYCL degrades only ~2.4x (38.5 to 15.8 t/s); prefill falls 562 to 101 t/s on Vulkan. An f16-KV control collapses identically, and MTP gain goes negative at depth on Vulkan. A third-party RX 7900 XTX run holds ~58-60 t/s at 28-30k depth.

generation:

63.7 tokens/s

prompt processing (prefill):

977.4 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

q8_0

User reports Qwen3.6-35B-A3B at 63.66 t/s sustained generation on an Intel Arc Pro B70. Setup is llama.cpp commit 210a657 with SYCL backend, Q4_K_XL quant and q8_0 KV cache, 262144 context. Short bursts peak at ~65 t/s; sustained generation over 1000+ tokens holds at 62-64 t/s. llama-bench on the same machine gives tg128 70.54 t/s and pp512 977.40 t/s.

generation:

52.2 tokens/s

prompt processing (prefill):

763.0 tokens/s

quant:

INT4 (GPTQ)

kv:

FP8

User reports Qwen3.8-27B at a median 52.2 tok/s decode and 763 tok/s prefill at 111.8K tokens on an Intel Arc Pro B70 32GB. Setup is vLLM XPU with the GPTQ INT4 quant, MTP2 speculative decoding, an FP8 KV cache, and 128K context. Vision, tool calling, and agent tests pass. vLLM beats llama.cpp SYCL by about 1.8x. An MTP plus concurrency crash was fixed with max-num-seqs 1, and a prefix caching bug was patched.

generation:

70.5 tokens/s

prompt processing (prefill):

977.4 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q8

User reports creating a poker game without issues. User mentions trying Intel's vLLM fork previously.

generation:

63.0 tokens/s