Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 1821 t/s prompt processing on an AMD Strix Halo 128GB host (GMKtec EVO-X2, Ryzen AI Max+ 395 / Radeon 8060S plus XDNA2 NPU).
Setup is the gufo NPU-enabled fork with UD-IQ4_XS weights, 8192-token prefill chunks, GPU fabric clock held at its top level and NPU performance mode.
The figure is a warmed synthetic-prefill microbenchmark at 16,384-token prefill length, not configured context capacity, and no generation throughput is published. The matched GPU-only baseline is 1698 t/s, a source-reported 7.2% improvement.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 31B at 12.1 t/s generation and 573.6 t/s prefill on an AMD Strix Halo 128GB-class unified-memory system.
Setup is the Gufo Gemma development fork with QAT UD-Q4_K_XL (Q4_0) weights, one warmed HTTP single-session prose request, greedy temperature 0/seed 1, thinking off, autoregressive with draft_n=0.
The row is one sample with no SD, prompt_n 2038, cache_n 0, predicted_n 128; rates are engine-stage, not whole-request throughput, and were not independently recomputed.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 31B at 6.4 t/s generation and 515.7 t/s prompt on an AMD Strix Halo 128GB unified-memory system.
Setup is the Gufo Gemma development fork with UD-Q8_K_XL, one warmed HTTP single-session request, greedy temperature 0/seed 1, thinking off, autoregressive with draft_n=0.
The run used prompt_n 2038, cache_n 0 and predicted_n 128; the rates are engine-stage figures, not prompt/TTFT or whole-request throughput, and come from a single sample.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 31B at 10.95 t/s generation and 544.93 t/s prefill on an AMD Strix Halo 128GB-class unified-memory system.
Setup is the Gufo Gemma development fork with UD-Q4_K_XL, one warmed HTTP single-session prose request, greedy temperature 0/seed 1, thinking off, autoregressive with draft_n=0.
The run used prompt_n 2038, cache_n 0 and predicted_n 128; the rates are engine-stage figures from one sample, not prompt/TTFT or whole-request throughput.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports GLM-5.3 Flash (Maya-S-v2 IQ2_XXS) at 18.3 t/s generation and 320 t/s prompt processing on an AMD Strix Halo 128GB.
Setup is Project Maya 1.0.30 on HIP with the Maya-S-v2-IQ2_XXS mixed quant, one request, 262K configured context, the whole model in unified memory; whether MTP was on is not stated.
Source: https://github.com/mw00/project-maya/issues/102 (LIN-LAB-AI, 2026-10-11; third-party measurement, not our run).
Headline metrics are separate built-in --bench tests: 320 tok/s prefill for 8,192 input tokens; generation 18.3 tok/s reported mean, three 256-token answers (18.3/18.4/18.3), greedy, thinking off. Short/warm decode is not generation following that 8K prefill. Also: 267 tok/s prefill for 2,048 input tokens. Prefill repetition count unspecified. Configured context 262,144, not headline prompt length.
Hardware/runtime: one Radeon 8060S (gfx1151), Ryzen AI Max+ 395, 128GB unified; 96GiB BIOS carve-out. Windows reports 31.65GiB host RAM; HIP reports 107.87GB GPU memory. These are overlapping unified-memory views, not additive capacities. Native Windows 11 build 26100; HIP SDK 7.2/clang21; Maya 1.0.30 c189f7a; --backend hip --gpu 0; NVMe model storage; --prefill auto, PREFILL_SUB=1024, PREFILL_MB=6144.
Separate cold technical-prose sweep asserts reused=0 and uses engine prompt_ms/decode_ms: at 8,632–9,070 input tokens median 315.1 pp / 15.90 tg; at 261,748–261,771 input median 274.5 pp / 14.20 tg. Answers usually ~65 tokens, so not comparable to 256-token headline. No downloadable per-request sweep logs linked. KV precision, Flash Attention, MTP status, clocks/power and exact device manufacturer unspecified.
Verified model identity: peasantsmith/GLM-5.3-Flash-Maya-GGUF, folder Maya-S-v2-IQ2_XXS, three GLM-5.3-Flash-Maya-S-v2-IQ2_XXS-0000{1,2,3}-of-00003.gguf shards. Source reports SHA256 verification against its launcher. Tested-version manifest: https://github.com/mw00/project-maya/blob/v1.0.30/maya.py . It accepts two first-shard headers and uses revision main; exact tested Hugging Face commit is unknown. Quant is mixed: IQ2_XXS gate/up experts, IQ2_S/IQ3_XXS down, Q6_K attention, including an MTP block (presence does not establish use).
Model card/file verification: https://huggingface.co/peasantsmith/GLM-5.3-Flash-Maya-GGUF/tree/main/Maya-S-v2-IQ2_XXS and https://huggingface.co/peasantsmith/GLM-5.3-Flash-Maya-GGUF . Card identifies 321B total/~18B active and 96.5GB text weights. Source 90.90GiB on-disk includes vision files; not peak RAM. Vision was untested. No quality benchmark reported.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticlong-contextvision
User reports Qwen3.8-Flash-Next running on a BOSGAME M5 with AMD Ryzen AI Max+ 395 / Radeon 8060S (Strix Halo) and 128 GB unified RAM, showing ~69 tok/s peak decode and ~1,155 tok/s prefill with peaks around 1,685 tok/s during a real long-running agent workload.
Setup is Halogen 0.17.2 on Ubuntu 26.04 with 262K context configured and vision enabled; decode stays in the 60–70 tok/s range while prefill remains above 1k tok/s for much of the workload.
The figures come from a Grafana dashboard tracking Halogen throughput, KV/context usage, RAM, PSI memory pressure and GPU metrics; context was ~84% full, 82K tokens generated in the last hour, GPU around 62°C and 24 W, with ~121 GB physically occupied but mostly reclaimable cache (~85 GB).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 60.39 t/s generation and 1700.52 t/s prompt processing on a Strix Halo 128GB device (GPD Win5 128G).
Setup is the Gufo engine with Q4_K_XL quant, single user.
The same engine also ran Qwen 27B Q4_K_XL at 70.56 t/s generation and 656.33 t/s prompt with DFlash2, and served 8 users at 162.98 t/s aggregate. User says performance is close to Halogen and better than llama.cpp variants on prefill, and is switching to it for daily use.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticcodingtool-use
User asks how to combine a 128GB Evo-X2, a 48GB MacBook Pro M5 Pro and an RTX 4080 Super desktop for local LLM work.
The Evo-X2 runs a Qwen 35B MoE at about 500-900 tok/s prompt processing and roughly 50 tok/s decode, used for business products rather than as a daily driver.
The user also describes a memory layer and an orchestrator setup with leads and an oracle, and asks whether they are reinventing the wheel.
User reports running Qwen3.8 Flash-Next on a Strix Halo 128GB at 1600-1800 t/s prompt processing and 60 t/s generation for agentic work. The source discusses Halogen and Gufo with Q4_K_XL but does not unambiguously attribute this Flash-Next speed range to one engine/quant combination; structured engine and quantization are therefore unspecified. The 128GB is unified memory on the Framework Desktop, not additional discrete VRAM.
A separate single CMP 170HX 64GB (unlocked from 8GB) runs two concurrent Qwen3.8 27B sessions with 200k bf16 context each, using DFlash 2 speculative decoding. Aggregate performance at low context is 2000 t/s prompt and 150-300 t/s generation. A 4-hour session with 7.2M tokens in and 1.1M tokens out averaged 1747 t/s prompt and 100 t/s generation aggregate across the two 27B sessions.
The CMP 170HX sits on a USB4 eGPU dock limited to PCIe 2x4. The user dropped cloud usage entirely with this setup, which draws about 400W fully loaded and cost around $3000 total.
Hardware attribution corrected: the Flash-Next figures above belong to the Strix Halo. Two concurrent 27B sessions on the separate CMP 170HX are not two GPUs; the author cancelled the second-card purchase. The CMP figures remain a separate workload and are not assigned to this Flash-Next report.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Ornith 1.0 35B-A3B at 74.1 t/s generation and 1,102.5 t/s prompt processing on an AMD Strix Halo 128GB.
Setup is llama.cpp llama-bench build 9590 on Vulkan (RADV) with Q4_K_M, one request at a time, speculative decoding off, the whole model in unified memory.
Source-attributed historical benchmark by axjns, measured August 17, 2026. This is a public third-party llama-bench measurement, not my own hardware or an independent hardware rerun. The model is Ornith 1.0, NOT Ornith 1.5.
RESULT AND ACTUAL WORK
The submitted rounded fields are prefill 1102.5 tokens/s and generation 74.1 tokens/s. Exact source means and standard deviations across three repetitions are:
- Prefill: 1102.529259 +/- 24.069094 tokens/s, n_prompt=512, n_gen=0, depth=0.
- Decode: 74.104028 +/- 0.245648 tokens/s, n_prompt=0, n_gen=128, depth=0.
These are TWO SEPARATE synthetic microbenchmark phases. They are not one 512-input/128-output request, not medians, not client end-to-end throughput or TTFT, and not an application conversation. The context field is left blank rather than presenting an input count as configured context capacity. Depth zero means no prefilled context depth for these headline points; it does not establish cold model loading or a cold filesystem cache.
RAW SAMPLES AND CHECK
The pp512 sample durations are 472523394, 467817691, and 453257747 ns. The tg128 durations are 1728353257, 1732444252, and 1721144960 ns. Arithmetic means of 512e9/ns and 128e9/ns respectively reproduce 1102.529259240522 and 74.10402815760727 tokens/s, agreeing with the published means. The reported standard deviations are the source values. The two records are lines 41 and 42 of the JSONL. Source timestamps for those records are 2026-08-17T13:09:34Z and 2026-08-17T13:09:42Z.
Other separate points from the same model, lines 43 and 44, are pp4096 at depth 0: 1060.246020 +/- 8.507957 tokens/s, and tg128 at depth 4096: 69.904247 +/- 0.369516 tokens/s. These are supplementary observations, not pooled into the headline rates. These additional 4K-scale observations do not establish long-context application performance.
HARDWARE AND MEMORY
One AMD Ryzen AI MAX+ 395 / Radeon 8060S Graphics, gfx1151, with 128 GB unified memory according to the dataset card. GPU count 1 denotes one machine/iGPU. The raw records expose a 64 GiB GPU allocation (68719476736 bytes) and 62 GiB OS-visible system RAM. The 64 GiB carve-out and 62 GiB OS-visible figure are not extra memory to add to the 128 GB total. The separate reported-VRAM form field is deliberately blank to avoid implying independent dedicated VRAM.
For the headline prefill record, source GPU-allocation peak is 23.66 GiB and baseline 3.60 GiB, difference 20.06 GiB. For decode they are 23.15, 3.59, and 19.56 GiB. These are rocm-smi readings sampled at 4 Hz; a sampled maximum is not a guaranteed instantaneous peak, process RSS or total resident system-memory measurement. All four Ornith rows have gpu_contended=false and an empty concurrent_workloads list. Power profile, watts, clock rates and temperature are unreported.
RUNTIME AND BACKEND CONFLICT
Use the raw runtime provenance: llama.cpp llama-bench build 9590, recorded commit d2462f8f7; lb_backends=BLAS,Vulkan and lb_gpu_info=AMD Radeon 8060S Graphics (RADV GFX1151). The dataset card describes ROCm, and the environment contains ROCm/HIP 7.13.26162, but those installed versions do NOT make this a HIP/ROCm inference measurement. The actual recorded benchmark backend is Vulkan/RADV. Mesa version and an independently verified binary hash are unknown.
Fedora release 43, kernel 7.0.14-101.fc43.x86_64. n_batch=2048, n_ubatch=512, n_threads=16, n_gpu_layers=999, n_cpu_moe=0, split_mode=layer, main_gpu=0, no_kv_offload=false, use_mmap=true, use_direct_io=false. K and V cache types are both f16. lb_flash_attn=-1 records AUTO; actual Flash Attention engagement is unverified and the form flag is left unknown. No speculative route or draft model is recorded. No inference-quality or speculation-quality claim is made.
MODEL IDENTITY
Recorded filename: ornith-1.0-35b-Q4_K_M.gguf. Raw model type: qwen35moe 35B.A3B Q4_K - Medium. Raw parameter count is 34660610688; the form uses nominal 35B total / 3B active. The GGUF file is 21166757760 bytes on disk; lb_model_size is 21155768832 bytes. The source parses the quant label from the filename, so Q4_K_M is filename-derived rather than independently verified by inspecting the weights.
The tested model Hugging Face repository, tested model revision, and tested weight-file hash are UNREPORTED. Do not infer a model download identity or immutable weight pin from the local folder name or from a current similarly named model page. The Hugging Face links below point to a BENCHMARK DATASET and its harness, not to the tested model weights. The dataset commit is an evidence snapshot, not the model revision.
PUBLIC PRIMARY SOURCES
Dataset card and methodology:
https://huggingface.co/datasets/axjns/strix-halo-inference-bench
Immutable raw benchmark records (lines 41-44):
https://huggingface.co/datasets/axjns/strix-halo-inference-bench/blob/9eecbcec0c626e9e8340c47752ebee50112c856a/data/results.jsonl
Current raw-record view:
https://huggingface.co/datasets/axjns/strix-halo-inference-bench/blob/main/data/results.jsonl
Published measurement harness:
https://huggingface.co/datasets/axjns/strix-halo-inference-bench/blob/main/strix_bench.py
Verification October 11, 2026: all 44 records were readable in the primary file view; pinned and main snapshots parsed identically. Headline means were independently recomputed from published timing samples. Live catalog dedup checked all 115 Strix Halo reports across six pages. The existing Ornith 1.5 ROCmFP4 report is a different model/version and setup. These checks verify evidence consistency, not independent hardware reproduction. This single-machine, single-build snapshot measures speed only; model quality, real-task usefulness, longer contexts and sustained service behavior were not evaluated.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 29.80 t/s generation and 1193.06 t/s prefill on two AMD Strix Halo 128GB hosts.
Setup is Gufo with two-host RDMA tensor parallelism over USB4 RoCE v2, full Q8_0 GGUF, 262144-token configured context, one session, speculative decoding off.
Measured prompt was 2040 tokens with no cached prefix and 128 predicted tokens; the 262144 context is capacity, not a benchmark length. Source-reported third-party measurement, not independently reproduced.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-context
User reports Qwen3.8 Flash-Next at 42.7 t/s generation and 1294.2 t/s prefill on a 126,000-token prompt on an AMD Strix Halo 128GB.
Setup is Strata at UD-IQ4_XS with speculative decoding on, one request.
tomasreminek's context-131072 request of 2026-10-06. A source-reported measurement, not independently reproduced.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 26B-A4B at 53.2 t/s generation and 3991.2 t/s prefill on a 2,038-token prompt on an AMD Strix Halo 128GB.
Setup is Gufo at UD-Q4_K_XL, one request, speculative decoding off.
franzmoca's depth-zero row of 2026-10-07 from the Gemma 4 Gufo fork. A source-reported maintainer measurement, not independently reproduced.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-context
User reports Qwen3.6 35B-A3B at 53.2 t/s generation and 891.6 t/s prefill on a 29,982-token prompt on an AMD Strix Halo 128GB.
Setup is llama.cpp at Q4_K_M, one request, speculative decoding off.
AGmind Systems Lab's doc-session run of 2026-08-03 (CC BY 4.0). A source-reported measurement, not independently reproduced.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 35 t/s generation and 1701.6 t/s prefill on a 126,000-token prompt on an AMD Strix Halo 128GB.
Setup is UD-IQ4_XS with speculative decoding on, one request.
tomasreminek's context-131072 request of 2026-10-06. A source-reported measurement, not independently reproduced.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next Uncensored at 36.9 t/s generation and 1232.9 t/s prefill on an AMD Strix Halo 128GB.
Setup is Gufo with MTP speculative decoding on, Q4 Mix (Q4_K/Q5_1/IQ4_NL/Q8_0), one request.
Medians of the three mixed-long requests (10,583-token prompts) in vmlinux's bench-gufo-mtp.json. A source-reported measurement, not independently reproduced.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codinglong-contextsummarization
User reports DeepSeek-V4-Flash at 15.4 tok/s decode and 142.30 tok/s prefill at 2,048 tokens of context on a Ryzen AI Max+ PRO 395 (Strix Halo) with 128 GB unified memory.
Setup is the ds4 engine with an IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8 GGUF (81 GB), 32,768-token context, ROCm 7.1.1; decode falls 12% from 2K to 32K while prefill stays flat.
A second model, Qwen3.8-Flash-Next UD-IQ4_XS (88 GB) under Strata 0.1.40, decodes at 39.0 to 40.9 tok/s and prefills a 6,786-token prompt at 343 to 367 tok/s, dropping to 30.4 tok/s each with two requests in parallel. The same engine measured 8.7 tok/s while another model's worker held 41 GiB of VRAM and 23 GB of RAM, against 39.0 tok/s on an exclusive box.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3 30B-A3B MoE at 86.1 t/s token generation on a Strix Halo 128GB (Ryzen AI Max+ 395, Radeon 8060S).
Setup is llama.cpp with Q4_K_M GGUF, Vulkan RADV backend, 120W power profile, single host (GMKtec EVO-X2).
Prompt processing measured at 1142 t/s (pp512). The repo also lists MiniMax M2.5 Q3_K_M at 32.8 t/s, Qwen3-Coder-Next 80B-A3B Q4_K_M at 42.7 t/s, GPT-OSS 120B Q4_K_M at 53.4 t/s and Nemotron-3 Nano 30B-A3B MXFP4 at 61.5 t/s on the same hardware.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentictool-uselong-context
User benchmarks Swift-1.5 Qwen3.8-Flash-Next IQ4_XS on a Strix Halo 128GB (Ryzen AI Max+ 395, Radeon 8060S) at 36.0 t/s decode for prose, 43.2 t/s code and 40.9 t/s reasoning, with 1,634 t/s prefill at 52k context.
Setup is halogen-flash-server 0.16.2 with IQ4_XS weights at 128k context, single stream, temperature 1.0 and reasoning_effort medium.
The same engine with base Flash-Next decoded 41.7/51.3/51.6 t/s and prefilled 1,655 t/s at 52k, while the best llama.cpp Vulkan config (Swift Flash-Next IQ3_XXS with MTP n=3) reached 28.8/44.0/46.5 t/s decode and 275 t/s prefill at 52k. Swift-1.5 finished agentic tasks in 40.4 s each versus 53.3 s for base Flash-Next, scoring 36/36 agentic tasks and 56/58 eval items.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingcreative-writinglong-context
User reports Qwen3.8-27B at 59.7 tok/s cold (c30) on AMD Strix Halo (Ryzen AI Max+ 395, Radeon 8060S, 64GB unified).
Setup is llama.cpp b10435-era with UD-Q4_K_XL GGUF, F16 KV cache, 96k context, draft-mtp + ngram-mod speculation (n-max 12, n-min 24).
Warm c30 reaches 148-163 tok/s but is labeled a repetition artifact; real conversational traffic is 11-24 tok/s (prose) and 30-40 tok/s (code). Prefill is 271-330 tok/s by depth. Time-per-task is 7.9-14.3s (median 11.3).