llamaperf

Qwen3.8 125B (6B active) Flash-Next

on 2× NVIDIA Tesla P40 24GB · Strata · 32,768 ctx

NVIDIA hardware
Oct 9, 2026
Throughput
34.8 t/s gen · 501.0 tokens/s prompt processing (prefill)Prompt processing measures input prefill speed; it does not tell you the time to first token.
Quant
IQ2_XS (GGUF)
VRAM reported
22 GB

Summary

User benchmarks Qwen3.8 Flash-Next at 34.8 tok/s decode on two Tesla P40 24GB cards with a layer split, and 19.6 tok/s on a single P40. Setup is Strata v0.1.39 with IQ2_XS weights at 32,768 context, experimental CUDA 12 engine built locally; the two-card run splits layers across both GPUs (K=24, all 24576 profiled expert pairs resident). Prompt reading was 353 / 501 tok/s at 4K / 16K on both cards and 348 / 388 tok/s on one; figures are medians of 3 runs per cell, with the first run of each configuration slower due to expert-cache warm-up. A 69-task correctness suite ran twice, and two greedy runs of identical prompts gave 49 of 69 identical outputs.