Local model performance on your hardware

generation:

70.0 tokens/s

prompt processing (prefill):

2800.0 tokens/s

quant:

W4A16-AutoRound (AutoRound)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at ~70 t/s generation and ~2800 t/s prefill on 4x Intel Arc Pro B70 GPUs. Setup is vLLM with Intel's W4A16-AutoRound quant, full 262K FP8 context, prefix caching, and concurrency=2. The user notes an alternative setup reached ~100 t/s generation but only ~1400 t/s prefill and 16-bit context, and considers the doubled prefill and context worth the ~30 t/s generation loss.

generation:

91.2 tokens/s

prompt processing (prefill):

2421.0 tokens/s

quant:

EXL3 4.00bpw (EXL3)

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B EXL3 4.00bpw at 91.2 t/s decode (C1, prose) on one Intel Arc Pro B70 32GB. Setup is vLLM 0.26.1 XPU with EXL3 trellis quantization, fp8 KV cache, 262,144 context, MTP speculative decoding k=3, int8 prefill and oneDNN fused attention. Aggregate throughput scales to 365.2 t/s at C16; cold prefill is 2,421 t/s at 4K and 911 t/s at 254K. The same card with tuned llama.cpp SYCL Q4_K_M reaches 25.0 t/s at C1 and 999 t/s prefill at 4K.

generation:

40-70 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 40-70 tokens per second on an Intel Arc Pro B70. Setup is vLLM with a Helm chart applying patches from the Intel Arc Pro B70 Inference Cookbook, at 128k context. The user gives the speed as a range depending on workload.

generation:

52.2 tokens/s

prompt processing (prefill):

763.0 tokens/s

quant:

INT4 (GPTQ)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at a median 52.2 tok/s decode and 763 tok/s prefill at 111.8K tokens on an Intel Arc Pro B70 32GB. Setup is vLLM XPU with the GPTQ INT4 quant, MTP2 speculative decoding, an FP8 KV cache, and 128K context. Vision, tool calling, and agent tests pass. vLLM beats llama.cpp SYCL by about 1.8x. An MTP plus concurrency crash was fixed with max-num-seqs 1, and a prefix caching bug was patched.