Summary
User reports Qwen3.8 Flash-Next at 39.0 tok/s per request on two Strix Halo 128GB boxes with tensor parallelism, against 29.8 tok/s on one box.
Setup is a fork of gufo with Q4 weights (111 GB), TP=2 over USB4 RDMA, each box holding 57 GB of the model. Decode is plain autoregressive with no speculation.
At 2 concurrent requests the two-box run gives 31.7 tok/s versus 23.3 on one box, and at 4 concurrent 24.6 versus 18.0. Short-prompt prefill is 1161 tok/s on two boxes and 1155 on one.
Compare this run
Medians of plain runs: one device, one request, no speculative decoding, the whole model in the device's memory. The report above is one setup, so read it beside them and not as one of them.
Qwen3.8 125B · 6B active at 4-bit by hardware
- 129.2 t/s · one run
- 105.5 t/s · 6 runs
- 57.2 t/s · 2 runs
- 55.0 t/s · one run
- 36.7 t/s · one run
- NVIDIA DGX Spark128 GB29.8 t/s · 2 runs
Bars are medians on one scale, 0 to 232.0 tokens/s. Each dot is one run and opens its report. Press ▶ to watch text arrive at a row's speed.
Models most run on the AMD Strix Halo 128GB
- 26.6 t/s · 8 runs
- 60.0 t/s · 7 runs
- 69.9 t/s · 6 runs
- 50.4 t/s · 4 runs
- 52.6 t/s · 4 runs
- 71.7 t/s · 4 runs
Bars are medians on one scale, 0 to 77.0 tokens/s. Each dot is one run and opens its report. Press ▶ to watch text arrive at a row's speed.
Everything reported on the AMD Strix Halo 128GB →Every Qwen3.8 report →Estimate Qwen3.8 on this card →