Local model performance on your hardware

generation:

48.6 tokens/s

prompt processing (prefill):

1480.0 tokens/s

quant:

ternary (GGUF)

kv:

q4_0

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports ternary Bonsai 2 27B at 48.6 t/s plain decode on an Intel Arc B580 12GB. Setup is a custom llama.cpp SYCL branch with ternary weights, q4_0 KV cache and 128K context, with no speculation for this figure. With MTP, n-gram speculation and a trimmed draft head the model reaches 85 t/s writing code, 79 t/s in chat and 89 t/s at 115K tokens of history, with prompt reading at ~1,480 t/s. The user's own draft head was used for the video, so public results may be a few percent lower.

generation:

42.3 tokens/s

prompt processing (prefill):

895.0 tokens/s

quant:

PTQ1_0 (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Bonsai 2 27B (PTQ1_0 ternary) at 42.3 t/s plain generation and 895 t/s prompt reading on an Intel Arc B580 12GB. Setup is llama.cpp's llama-bench (tg128 and pp512) on the SYCL backend with PTQ1_0 GGUF weights. With speculation, fresh code answers reach 90 t/s, renames 365 t/s, edits 258 t/s. At 48K context new code runs 63 t/s and edits 34 t/s; at 115K, 44 and 27 t/s. The user also compares against the Vulkan backend and PrismML's own fork, and reports KL divergence of 0.00022 against reference logits.

generation:

21.3 tokens/s

quant:

Q4_0 (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports GLM-4 9B at 21.25 t/s on an Intel Arc B580 12GB, with GPU utilization near 40%. Setup is llama.cpp (ipex-llm[cpp] build 2024.12.17) with Q4_0 GGUF and F16 KV cache, 2048 context, all 41 layers offloaded to the GPU. The user considers the performance low for the hardware. The prompt eval figure of 41.47 t/s is over only 5 tokens and is not reported as a prefill rate.

generation:

12.7 tokens/s

prompt processing (prefill):

260.5 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3 4B at 260.52 t/s prompt processing and 12.71 t/s generation on an Intel Arc B580. Setup is llama.cpp (IPEX-LLM build) with Q4_K_M GGUF, all 99 layers offloaded to the GPU via SYCL. Compared against stock Ollama on the same model, which reached 14.23 t/s prompt and 6.20 t/s generation, making the Arc GPU about 18.3x faster at prompt processing and 2.05x faster at generation.