Intel Arc B580 12GB reports

generation:

45.0 tokens/s

quant:

ternary

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Bonsai-2 27B running at around 45 t/s on an Intel Arc B580 12GB. Setup is PrismML's llama.cpp fork with custom SYCL kernels and fusions, which the user wrote and PrismML merged (PR #356, about 7,700 lines). The models use ternary weights (-1, 0, +1). The user also trained small ternary models in cloud data centres that run at home on the B580: SmolLM2-360M shrunk from 690 MB to 162 MB at 1.4x the original perplexity and about 94% of benchmark scores, and SmolLM2-135M from 257 MB to 73 MB at about 1.5x and about 91%. They say these are not yet Bonsai quality, whose models keep about 99%. The models are not released yet.

prompt processing (prefill):

4454.0 tokens/s

quant:

Q4_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Llama 2 7B Q4_0 prompt processing at 4,454 t/s on an Intel Arc B580 with llama.cpp's SYCL backend, ahead of the RTX 5060 Ti's 4,196 t/s on llama.cpp's scoreboard. Prompt processing is 3.3x to 6.6x faster than llama.cpp master across Llama 2, Llama 3.1, Qwen3.5 and Gemma 4, with long chats 1.5x faster at 32K and 1.6x at 64K. Short batches on K-quant models are up to 2.5x faster and still in progress; perplexity is unchanged. The changes are on the user's GitHub branches and submitted for upstreaming.

generation:

48.6 tokens/s

prompt processing (prefill):

1480.0 tokens/s

quant:

ternary (GGUF)

kv:

q4_0

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports ternary Bonsai 2 27B at 48.6 t/s plain decode on an Intel Arc B580 12GB. Setup is a custom llama.cpp SYCL branch with ternary weights, q4_0 KV cache and 128K context, with no speculation for this figure. With MTP, n-gram speculation and a trimmed draft head the model reaches 85 t/s writing code, 79 t/s in chat and 89 t/s at 115K tokens of history, with prompt reading at ~1,480 t/s. The user's own draft head was used for the video, so public results may be a few percent lower.

generation:

21.0 tokens/s

quant:

IQ2_XS (GGUF)

kv:

INT8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 21.0 tok/s on an Intel Arc B580 12GB. Setup is Strata v0.1.40.4 with IQ2_XS quantization and INT8 KV cache, 256K context, 2808 resident expert slots, speculative decoding with MTP. Long-context stress test at 74,213 prompt tokens sustained 17.8 tok/s for 237.7 seconds. User compares to llama.cpp/Ollama offload at 1.5-3.0 tok/s and previous Strata version at 17.1 tok/s.

generation:

42.3 tokens/s

prompt processing (prefill):

895.0 tokens/s

quant:

PTQ1_0 (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Bonsai 2 27B (PTQ1_0 ternary) at 42.3 t/s plain generation and 895 t/s prompt reading on an Intel Arc B580 12GB. Setup is llama.cpp's llama-bench (tg128 and pp512) on the SYCL backend with PTQ1_0 GGUF weights. With speculation, fresh code answers reach 90 t/s, renames 365 t/s, edits 258 t/s. At 48K context new code runs 63 t/s and edits 34 t/s; at 115K, 44 and 27 t/s. The user also compares against the Vulkan backend and PrismML's own fork, and reports KL divergence of 0.00022 against reference logits.

generation:

21.3 tokens/s

quant:

Q4_0 (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports GLM-4 9B at 21.25 t/s on an Intel Arc B580 12GB, with GPU utilization near 40%. Setup is llama.cpp (ipex-llm[cpp] build 2024.12.17) with Q4_0 GGUF and F16 KV cache, 2048 context, all 41 layers offloaded to the GPU. The user considers the performance low for the hardware. The prompt eval figure of 41.47 t/s is over only 5 tokens and is not reported as a prefill rate.

prompt processing (prefill):

638-867 tokens/s

quant:

IQ2_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports a 35B-A3B MoE model (3B active) at 47-50 t/s decode and 160-200K context on a single Intel Arc B580 12 GB. Setup is llama.cpp SYCL with IQ2_XXS mixed quant, MoE experts offloaded to CPU via --n-cpu-moe 24, -ub 1024 -b 2048, -fa on, on Windows 11 with a Ryzen 5 9600X and 32 GB DDR5-6000. Decode drops to 43.0 t/s at 43.5K context; prefill at 43.5K is 638-867 t/s depending on quant and n-cpu-moe. A 65K context sweet spot with --n-cpu-moe 8 reaches 63 t/s. The same recipe applies across the Gated-DeltaNet MoE family, including Qwen3.6-35B-A3B, Occamy-1.0, Qwen3.8-35B-Distill and Ornith-1.5-35B-A3B.

generation:

80-90 tokens/s

quant:

2-bit ternary (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Ternary Bonsai 2 27B running on an Intel Arc B580 12GB at 128K context entirely in VRAM, with about 90 t/s writing new code at temperature 0 (about 80 at temperature 0.6), 250 to 370 t/s on code edits with MTP plus n-gram drafts, and 42 t/s plain. Setup is a custom llama.cpp fork with ternary weights repacked to 2 bit, q4_0 KV cache, and decode attention on XMX; the fork is compared against PrismML's basic SYCL path on the same card, model and flags. At 48K tokens of history it still does 63 t/s, and 44 at 115K. Against PrismML's fork: prompt reading 207 vs 895 t/s, plain generation 21 vs 42, generation at 32K depth 8 vs 36, chat with MTP 38 vs 88. Against Gemma 4 12B on the same card, Gemma is a bit better at code (HumanEval+ 152 vs 141 / 164), they tie on GSM8K, and tool calling is close; with Google's QAT assistant as MTP drafter, Gemma 4 12B goes from ~40 to ~100 t/s on new code (edits 64 -> 168).

generation:

12.7 tokens/s

prompt processing (prefill):

260.5 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3 4B at 260.52 t/s prompt processing and 12.71 t/s generation on an Intel Arc B580. Setup is llama.cpp (IPEX-LLM build) with Q4_K_M GGUF, all 99 layers offloaded to the GPU via SYCL. Compared against stock Ollama on the same model, which reached 14.23 t/s prompt and 6.20 t/s generation, making the Arc GPU about 18.3x faster at prompt processing and 2.05x faster at generation.

generation:

0.7 tokens/s

quant:

UD-Q2_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B UD-Q2_K_XL at 0.65 tok/s decode on an Intel Arc B580 12GB. Setup is the FreeToken-Intel fork with PyTorch XPU plus native SYCL packed matvecs, context 128 and KV cache 256 tokens; TTFT 37.58 s and model load 47.39 s, with 9.47 GB model state. The run used a fixed 16-token forced decode length and is presented as a development baseline, not a general performance claim; the same table lists Llama 3.2 1B Q4_K_M at 18.29 tok/s, Gemma 4 12B Q4_K_XL at 2.85 tok/s and Ternary-Bonsai 27B PQ2_0 at 2.10 tok/s.