Local model performance on your hardware

generation:

45.0 tokens/s

quant:

ternary

User reports Bonsai-2 27B running at around 45 t/s on an Intel Arc B580 12GB. Setup is PrismML's llama.cpp fork with custom SYCL kernels and fusions, which the user wrote and PrismML merged (PR #356, about 7,700 lines). The models use ternary weights (-1, 0, +1). The user also trained small ternary models in cloud data centres that run at home on the B580: SmolLM2-360M shrunk from 690 MB to 162 MB at 1.4x the original perplexity and about 94% of benchmark scores, and SmolLM2-135M from 257 MB to 73 MB at about 1.5x and about 91%. They say these are not yet Bonsai quality, whose models keep about 99%. The models are not released yet.

prompt processing (prefill):

4454.0 tokens/s

quant:

Q4_0 (GGUF)

User reports Llama 2 7B Q4_0 prompt processing at 4,454 t/s on an Intel Arc B580 with llama.cpp's SYCL backend, ahead of the RTX 5060 Ti's 4,196 t/s on llama.cpp's scoreboard. Prompt processing is 3.3x to 6.6x faster than llama.cpp master across Llama 2, Llama 3.1, Qwen3.5 and Gemma 4, with long chats 1.5x faster at 32K and 1.6x at 64K. Short batches on K-quant models are up to 2.5x faster and still in progress; perplexity is unchanged. The changes are on the user's GitHub branches and submitted for upstreaming.

generation:

48.6 tokens/s

prompt processing (prefill):

1480.0 tokens/s

quant:

ternary (GGUF)

kv:

q4_0

mtp (multi-token prediction):

on

User reports ternary Bonsai 2 27B at 48.6 t/s plain decode on an Intel Arc B580 12GB. Setup is a custom llama.cpp SYCL branch with ternary weights, q4_0 KV cache and 128K context, with no speculation for this figure. With MTP, n-gram speculation and a trimmed draft head the model reaches 85 t/s writing code, 79 t/s in chat and 89 t/s at 115K tokens of history, with prompt reading at ~1,480 t/s. The user's own draft head was used for the video, so public results may be a few percent lower.

generation:

21.0 tokens/s

quant:

IQ2_XS (GGUF)

kv:

INT8

User reports Qwen3.8-Flash-Next at 21.0 tok/s on an Intel Arc B580 12GB. Setup is Strata v0.1.40.4 with IQ2_XS quantization and INT8 KV cache, 256K context, 2808 resident expert slots, speculative decoding with MTP. Long-context stress test at 74,213 prompt tokens sustained 17.8 tok/s for 237.7 seconds. User compares to llama.cpp/Ollama offload at 1.5-3.0 tok/s and previous Strata version at 17.1 tok/s.

generation:

42.3 tokens/s

prompt processing (prefill):

895.0 tokens/s

quant:

PTQ1_0 (GGUF)

kv:

q4_0

User reports Bonsai 2 27B (PTQ1_0 ternary) at 42.3 t/s plain generation and 895 t/s prompt reading on an Intel Arc B580 12GB. Setup is llama.cpp's llama-bench (tg128 and pp512) on the SYCL backend with PTQ1_0 GGUF weights. With speculation, fresh code answers reach 90 t/s, renames 365 t/s, edits 258 t/s. At 48K context new code runs 63 t/s and edits 34 t/s; at 115K, 44 and 27 t/s. The user also compares against the Vulkan backend and PrismML's own fork, and reports KL divergence of 0.00022 against reference logits.

generation:

21.3 tokens/s

quant:

Q4_0 (GGUF)

kv:

F16

User reports GLM-4 9B at 21.25 t/s on an Intel Arc B580 12GB, with GPU utilization near 40%. Setup is llama.cpp (ipex-llm[cpp] build 2024.12.17) with Q4_0 GGUF and F16 KV cache, 2048 context, all 41 layers offloaded to the GPU. The user considers the performance low for the hardware. The prompt eval figure of 41.47 t/s is over only 5 tokens and is not reported as a prefill rate.

generation:

12.7 tokens/s

prompt processing (prefill):

260.5 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3 4B at 260.52 t/s prompt processing and 12.71 t/s generation on an Intel Arc B580. Setup is llama.cpp (IPEX-LLM build) with Q4_K_M GGUF, all 99 layers offloaded to the GPU via SYCL. Compared against stock Ollama on the same model, which reached 14.23 t/s prompt and 6.20 t/s generation, making the Arc GPU about 18.3x faster at prompt processing and 2.05x faster at generation.

generation:

0.7 tokens/s

quant:

UD-Q2_K_XL (GGUF)

User reports Qwen3.8 27B UD-Q2_K_XL at 0.65 tok/s decode on an Intel Arc B580 12GB. Setup is the FreeToken-Intel fork with PyTorch XPU plus native SYCL packed matvecs, context 128 and KV cache 256 tokens; TTFT 37.58 s and model load 47.39 s, with 9.47 GB model state. The run used a fixed 16-token forced decode length and is presented as a development baseline, not a general performance claim; the same table lists Llama 3.2 1B Q4_K_M at 18.29 tok/s, Gemma 4 12B Q4_K_XL at 2.85 tok/s and Ternary-Bonsai 27B PQ2_0 at 2.10 tok/s.