- generation:
- 140.0 tokens/s
- quant:
- NVFP4 (NVFP4)
User reports Edge0-145B-NVFP4, a 145B latent MoE with about 10B active built on Qwen3.8-Flash-Next, at 140 tok/s single-stream on one RTX 5090 32 GB. Setup is a vLLM plugin on the NGC vLLM 0.33 image with NVFP4 routed experts, Q8_0 dense weights, a DFlash2 drafter and hot/cold expert placement: 12.8-16.2 GiB of experts in VRAM, the rest in pinned host RAM, with the CPU computing 55% of cold experts. The 51 GB PLE n-gram table is read from NVMe every step. At c4, c8 and c32 the same config reaches 213, 214 and 214 tok/s. Strata with MTP gets 121, 117 and 113 tok/s at c1, c4 and c8; like for like at c1 the user measures 140.5 vs 122.0 tok/s. Top-1 matches Strata in 257 of 261 teacher-forced positions.