llamaperf

Qwen3.8 125B (6B active) Flash-Next

on NVIDIA DGX Spark · vLLM · 500,000 ctx

NVIDIA hardware
Sep 28, 2026
Throughput
34.0 t/s gen · 2700-4000 tokens/s prompt processing (prefill)Prompt processing measures input prefill speed; it does not tell you the time to first token.
Quant
NVFP4 (NVFP4)
System RAM
128 GB

Use cases

agentictool-uselong-contextcoding

Summary

User reports Qwen3.8-Flash-Next at ~34 tok/s single-stream decode on a single NVIDIA DGX Spark (GB10, 128 GB unified pool). Setup is vLLM v0.30.0 with NVIDIA's NVFP4 checkpoint, hybrid fp8 side layers, MTP=2 speculative decoding, prefix caching, deterministic top-k, and YaRN 500k context. Prefill is ~2,700-4,000 tok/s (8k-32k prompts) with a ~505-520k-token KV pool. The repo patches vLLM to serve the 48 GiB n-gram PLE table from NVMe via mmap, fixing prefix caching and non-deterministic top-k on GB10. A separate concurrency sweep on the same hardware reports 266.8 aggregate tok/s at 48 concurrent streams.