Local model performance on your hardware

generation:

130.2 tokens/s

prompt processing (prefill):

5790.9 tokens/s

quant:

Q4_0 (GGUF)

flash attention:

on

User reports Llama2 7B at 130.2 t/s generation and 5790.9 t/s prompt processing on a Radeon 8060S plus an RTX 5070 Ti over OCuLink. Setup is llama-bench with Q4_0, -ngl 99, -fa 1, -n 128 and -p 512, split 9/1 between the RTX 5070 Ti and the Radeon 8060S. The 9/1 split is the first interior configuration, not the fastest single-device endpoint; the 10/0 endpoint reached 168.91 t/s generation and 7461.22 t/s prompt processing.

generation:

50.2 tokens/s

prompt processing (prefill):

1493.3 tokens/s

quant:

Q4_0 (GGUF)

flash attention:

on

User reports Llama2 7B at 50.2 t/s generation and 1493.3 t/s prompt processing on an AMD Strix Halo 128GB system. Setup is llama.cpp with Q4_0, run natively on the Radeon 8060S iGPU via ROCm with -ngl 99 and flash attention; an attached RTX 5070 Ti does not participate. ROCm TG128 measured 50.16 +/- 0.07 t/s and PP512 1493.28 +/- 30.20 t/s; prompt processing fell to 958.19 t/s at 8192 tokens of context.