Local model performance on your hardware

generation:

49.0 tokens/s

prompt processing (prefill):

48.0 tokens/s

quant:

Q6_K (GGUF)

kv:

q4_1

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-9B at 49 t/s generation and 48 t/s prompt processing on an Intel Arc A770 16GB. Setup is llama.cpp b9521 (Vulkan) with Q6_K weights, q4_1 KV cache, 256K context, flash attention, vision mmproj and MTP speculative decoding on a single slot. The same model under WSL2 with Q8_0 KV cache and 128K context reached about 40 t/s without vision. Qwopus3.5-4B-Coder hit 64 t/s generation and 100 t/s prompt at 96K context, and Gemma 4 12B reached 22 t/s generation and 74 t/s prompt at 128K context.

generation:

14.4 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6-27B-A3B-Coder at 14.4 t/s decode on an Intel Arc A770 16GB, scoring 10/10 on a Lua CSV parser acceptance task. Setup is llama.cpp with the SYCL backend and GGUF Q4_K_M weights, a comparison point against arcint's own AWQ IR serving on the same card. The figure is described as a rough bound rather than a directly comparable measurement, since the quantisation and engine differ from arcint's production configuration.

generation:

33.2 tokens/s

quant:

Q8_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3-1.7B at 33.2 tok/s on an Intel Arc A770 16GB using llama.cpp with the SYCL backend. Setup is llama.cpp SYCL with Q8_0 quantisation, single-stream (n_parallel=1). OpenVINO via OVMS reached 65.4 tok/s on the same model. Ten models were tested in total, with OpenVINO faster than SYCL on every model.