llamaperf
Oct 13, 2026
Throughput
24.9 t/s gen · 396.8 tokens/s prompt processing (prefill)Prompt processing measures input prefill speed; it does not tell you the time to first token.
Quant
UD-Q2_K_XL (GGUF)
KV cache
F16
Flash Attention
on
MTP (Multi-Token Prediction)
off
System RAM
128 GB

Summary

First-party historical kksoftwareag benchmark, inspected 2026-10-13. Published evidence is pinned to e6c7d65d0744db1236b21387bfa6a04085c4c9d5; GitHub currentOid and archive PAX comment agree. No inference rerun or model-weight download. Hardware: AMD Ryzen AI MAX+ 395 / Radeon 8060S (gfx1151), 128 GB shared LPDDR5X, Fedora 44, ROCm/HIP 7.1, from the source machine description. Governor: powersave; tuned: balanced. Source estimates about 3% run noise. No measured power or thermals. Core counts are unreported here; 16 is worker threads, not core count. Shared 128 GB is capacity, not dedicated VRAM or resident model use. One participating GPU is run-backed: raw gpu_info is exactly AMD Radeon 8060S Graphics. At the logged build revision, llama-bench get_gpu_info enumerates every GPU/IGPU and joins all descriptions without deduplication. The record specifies ROCm, devices auto, 99 GPU layers, main_gpu 0, no CPU MoE and layer splitting; only one GPU is enumerated. This proof comes from this benchmark, not another run's device inventory. Runtime: reporter’s HIP comparison build without rocWMMA. Both binaries report build 10685 / 17252c769; the source base is 17252c769a63c1cb650ce98ae309cf4de0da7778. The source-owned build script enables rocWMMA. Patches add Qwen/MTP support and the iGPU host-buffer workaround, but this benchmark has no draft. Commit is not an executable hash; other local changes and compiler version remain unknown. A current default is not substituted for the tested build. Workload: separate synthetic PP512 and TG128 tests, both at depth zero, using random token IDs. This is not one 512-input/128-output natural-language chat. Two measured repetitions per cell; raw nanosecond timings and per-trial rates are retained. Device work is synchronized before stopping each phase timer. PP counts 512 decoded input tokens and TG counts 128 serial single-token calls. The rate is the mean of each trial's tokens/time; standard deviation is the sample SD. Both were independently recalculated from samples_ns. Model load and initial warmup are excluded. No HTTP throughput or TTFT was measured. Active context allocation is not logged; model training capacity is not used as the benchmark allocation. Configuration: batch 2048, microbatch 512, 16 CPU threads, KV K/V f16, Flash Attention enabled, load/lazy modes auto, no_host false, no_kv_offload false, embeddings disabled, no tensor-buffer overrides. No rotation-disable environment is set in this arm. Native results (mean ± sample SD): 07-userhip-q2kxl-f16: microbatch 512; PP 396.813961 ± 1.789682; TG 24.907664 ± 0.020203 tokens/s. UTC PP/TG timestamps: 2026-09-03T10:51:10Z / 2026-09-03T10:51:14Z. The first recorded configuration is the headline. Where present, companion microbatch cells are grouped here rather than submitted as additional campaigns. Each test_time is UTC benchmark metadata, not a webpage publication date. Artifact: https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/824f539b2710e5a9e47af4952cf6578cf5ee8932/UD-Q2_K_XL. Every shard at source snapshot 824f539b2710e5a9e47af4952cf6578cf5ee8932 was verified with an HTTP 200 HEAD response. Combined published disk size is 78869128864 bytes; this is not runtime memory. Measured local weight hashes were not logged, and no weights were downloaded. Raw model_size=78858104320 is llama_model_size tensor storage, not measured process RAM. Raw model_n_params=176943899520 is the engine's 176.94389952B tensor count; source architecture is A3B. Do not multiply malformed MoE size labels or silently substitute another nominal model size. Resident memory and dedicated VRAM remain unknown. Sources: https://github.com/kksoftwareag/strix-halo-qwen-3.8-flash-next/blob/e6c7d65d0744db1236b21387bfa6a04085c4c9d5/bench/results/raw/07-userhip-q2kxl-f16.json https://github.com/kksoftwareag/strix-halo-qwen-3.8-flash-next/blob/e6c7d65d0744db1236b21387bfa6a04085c4c9d5/bench/sweep1-backend.sh https://github.com/kksoftwareag/strix-halo-qwen-3.8-flash-next/blob/e6c7d65d0744db1236b21387bfa6a04085c4c9d5/bench/results/sweep1.log https://github.com/kksoftwareag/strix-halo-qwen-3.8-flash-next/blob/e6c7d65d0744db1236b21387bfa6a04085c4c9d5/docs/measurements.html https://github.com/kksoftwareag/strix-halo-qwen-3.8-flash-next/blob/e6c7d65d0744db1236b21387bfa6a04085c4c9d5/engine/fetch.sh https://github.com/kksoftwareag/strix-halo-qwen-3.8-flash-next/blob/e6c7d65d0744db1236b21387bfa6a04085c4c9d5/engine/build.sh https://github.com/ggml-org/llama.cpp/blob/17252c769a63c1cb650ce98ae309cf4de0da7778/tools/llama-bench/llama-bench.cpp#L132-L142 https://github.com/ggml-org/llama.cpp/blob/17252c769a63c1cb650ce98ae309cf4de0da7778/tools/llama-bench/llama-bench.cpp#L2173-L2221 https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/824f539b2710e5a9e47af4952cf6578cf5ee8932/UD-Q2_K_XL Structured rate fields are rounded to the form’s 0.1 token/s precision; exact source rates and separate workloads are retained above.