Speeds are as each source reported them. GPU count, offloading and concurrent requests change them, and prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing two reports.
- generation:
- 55-65 tokens/s
- prompt processing (prefill):
- 900-1100 tokens/s
- quant:
- IQ2_XXS GSQ-RCO-GGUF (GGUF)
User reports DeepSeek V4.1 Flash running at 55-65 t/s decode and 900-1100 t/s prefill on a single unlocked CMP 170HX 64GB at 150W.
Setup is Project Maya with an IQ2_XXS GSQ-RCO-GGUF (2.0 bpw) quant.
Both figures are given as ranges. The user plans to retry at 250-300W once a server shroud arrives.
- generation:
- 25-33 tokens/s
- prompt processing (prefill):
- 900-1100 tokens/s
- quant:
- IQ2_XXS (GGUF)
User reports DeepSeek V4.1 Flash at around 25-33 t/s decode and 900-1100 t/s prefill at 128-256K context on an unlocked CMP 170HX 64GB, with prefill dropping to 520 t/s at 1M context.
Setup is Strata (Project Maya) with an IQ2_XXS GSQ-RCO GGUF at 2.0 bpw, on a Dell R740 server with 256GB DDR4-2133 ECC RAM, GPU capped at 150W and running at 85C without a fan.
User asks how others run this card with this model, and says the low-bit DeepSeek V4.1 Flash beats the Qwen3.8-Flash-Next Q6 they ran before on quality, though they avoid expert-preference optimizations that trade quality for speed.
- generation:
- 38.0 tokens/s
- prompt processing (prefill):
- 907.0 tokens/s
- quant:
- Q4_K_M (GGUF)
User reports Qwen3.6 27B at 38 t/s generation on an unlocked NVIDIA CMP 170HX 64GB.
Setup is llama.cpp with Q4_K_M and flash attention, full GPU layer offload, zero context depth.
Prompt processing measured 907 t/s at pp512 and 888 t/s at pp8192, five repetitions each; the model file is about 16GB.
- generation:
- 100.0 tokens/s
- quant:
- Q4_K_XL (GGUF)
agentictool-use
User reports Qwen3.8 Flash-Next at 100 tok/s on a CMP 170HX 64GB and one RTX 3090, with system RAM offload.
Setup is Strata with Q4_K_XL weights; the run used two GPUs plus system RAM, and the user notes about a quarter of the results came from Strata .38, an eighth from .39, and the rest from .40.
Before Strata, the same model ran on three GPUs (CMP 170HX and two RTX 3090s) under Unsloth Studio at 39 tok/s with 84 s average TTFT; the Strata run cut average TTFT to 15.7 s and total LLM time from 5,795m 9s to 1,228m 53s.
- generation:
- 95.6 tokens/s
- prompt processing (prefill):
- 1529.0 tokens/s
- quant:
- 3.05bpw (EXL3)
- kv:
- Q8
codingagenticlong-context
User reports GLM-5.3-Flash at 95.6 t/s decode and 1,529 t/s prefill on an 8K input, running on two CMP 170HX 64GB cards.
Setup is ExLlamaV3 1.5.4 with an EXL3 3.05bpw target, DFlash2 EXL3 6bpw K7 drafter and Q8 KV cache, target weights fully resident in HBM at about 116.6GiB across the two cards.
Decode stays near 90 t/s out to a 385K input (90.1 t/s, 1,535 t/s prefill); DFlash acceptance was around 87%, against roughly 60 t/s on the older MTP d2 path. The user also compared a Qwen3.8-Flash-Next AWQ INT4 + FP8 PLE vLLM setup on the same machine, which reached 129.5 t/s decode and 5,513 t/s prefill at 8K, but GLM finished three one-shot coding tasks sooner (4m40s vs 5m30s, 10m05s vs 14m40s, 4m40s vs 24m10s).
- generation:
- 117.0 tokens/s
- prompt processing (prefill):
- 6066.0 tokens/s
- kv:
- fp8
User reports DeepSeek-V4.1-Flash at 117 tok/s decode on one stream at 128k context and 6,066 tok/s prefill at 105k tokens on eight 64 GB CMP 170HX mining cards.
Setup is vLLM with fp8 KV cache, PP=8, DSpark speculative decoding with 5 draft tokens, Engram tables in pinned host RAM, and 1M context.
Decode drops to 96 tok/s at 512k. Eight concurrent streams reach 532 tok/s aggregate (66 per stream) at 105k. KV pool holds 6.17M tokens. One card was capped to 180 W after PCIe bus drops.
- generation:
- 101.3 tokens/s
- prompt processing (prefill):
- 3191.0 tokens/s
- quant:
- W4A16 (W4A16)
- kv:
- BF16
codinglong-contexttool-usemultilingual
User reports Qwen3.8-Flash-Next at 101.3 t/s decode on Japanese prose and 170.0 t/s on code, single stream, on two NVIDIA CMP 170HX cards.
Setup is vLLM with W4A16 weights, BF16 KV cache, MTP k=4 speculative decoding, 262,144-token context, and expert parallel across the two cards. The PLE n-gram table was converted to FP8 locally to fit in 92 GiB of host RAM.
Aggregate throughput reaches 392 t/s at 4 concurrent requests. Prefill measures 3,191 t/s at 6,954 tokens and 3,261 t/s at 27,853 tokens. A 200,087-token prompt returned the planted code with 70.5 s time to first token and 69.0 t/s decode at depth. MTP is worth about 1.6x on prose and 2.6x on code. The user notes xhigh reasoning effort spent the whole budget without answering in 5 of 6 runs.
- generation:
- 95.0 tokens/s
- quant:
- FP8
- kv:
- FP8
codinglong-context
User reports DeepSeek V4 Flash Vision at up to ~95 tok/s on 4× CMP 170HX 64GB (256GB aggregate HBM2e).
Setup is vLLM with pipeline parallel 4, max_model_len 262144, FP8 KV cache, DSpark speculative decoding with 6 tokens, and max_num_seqs 2.
Normal coding workload runs ~50–70 tok/s, strong speculative-decoding periods ~80–90 tok/s, peak ~95 tok/s. Cards sit around 56–62GB VRAM each at 50–60°C. Next tests planned for Qwen3.8 Flash Next and GLM 5.3.
- generation:
- 116.0 tokens/s
- prompt processing (prefill):
- 6800-7950 tokens/s
- quant:
- W4A16
User reports Qwen3-Next-80B at 116 t/s single-stream decode on a single NVIDIA CMP 170HX with 64 GB HBM2e, at a 150 W cap.
Setup is vLLM 0.27.1 with W4A16 weights (40.9 GB), torch 2.13.0+cu130, CUDA 13.0, Ubuntu 26.04 LTS, on an AMD Ryzen Threadripper PRO 3945WX with 128 GB DDR4 ECC. Prefill over about 8.9k tokens measured 6800 to 7950 t/s; power draw 137 to 145 W.
Aggregate throughput at 8 concurrent requests was 352 t/s, saturating at 4 slots. Also measured on the same card for context: Ornith-1.5-35B FP8 at 122.5 t/s and Qwen3.8-27B W4A16 with DFlash2 at 127 t/s single-stream.
- generation:
- 98.1 tokens/s
- prompt processing (prefill):
- 5300.0 tokens/s
- quant:
- MXFP4+FP8
User reports DeepSeek-V4-Flash-0731 at 98.1 t/s single-stream decode on 4x CMP 170HX (64 GB HBM2e each), up from a 50.8 t/s baseline with DSpark speculative decoding.
Setup is vLLM with pipeline parallelism, native MXFP4+FP8 weights (~155.4 GiB), context verified to 1,047,736 tokens. At 100k context single-stream decode is 38.8 t/s with 14.6 s time to first token.
Aggregate throughput across 64 concurrent requests is 712.8 t/s with DSpark (472.0 t/s baseline), and 90.0 t/s at 100k context. Prefill across the 25k-77k context range is about 5,300 t/s.
- generation:
- 147.0 tokens/s
- quant:
- W4A16 (W4A16)
- kv:
- FP8
long-context
User reports Qwen3.8-27B at 147.0 t/s single-stream decode on one NVIDIA CMP 170HX 64GB, with 134.7 t/s at 4K, 100.1 t/s at 65K, ~90 t/s at 126K, and 64.9 t/s at 250K context.
Setup is vLLM 0.27.1 with a W4A16 target, a DFlash2 W4A16 drafter, FP8 target KV, BF16 draft KV, a custom SM80 split-KV verifier, full CUDA Graph, 35 verifier segments / 140 CTAs, k=3 draft tokens, MAX_SEQS=1, and 1350 MHz / 180W. These are decode-only numbers, not end-to-end throughput including prefill.
FP8 beat INT8 at long context (48.3 vs 115.4 ms/iter at 250K). Allowing 2 concurrent long requests made makespan and slowest-request throughput worse at both 126K and 250K.
- generation:
- 50.0 tokens/s
- quant:
- Q8
coding
User reports Qwen3.8 27B at around 50 t/s on a CMP 170HX, running at Q8 or BF16.
The user finds the model hallucinates classes and APIs in 90%+ of answers to domain-specific enterprise Java questions, while DeepSeek and Microsoft Copilot produce working code 99% of the time.
The user asks whether Qwen's agentic optimization is responsible and requests tips for a harness, system prompt, or skills to reduce hallucination.
- generation:
- 63.0 tokens/s
- prompt processing (prefill):
- 1700.0 tokens/s
- quant:
- Q6_K
- kv:
- F16
User reports Qwen3.6-35B-A3B at 63 t/s on 4x CMP 170HX 8GB cards flashed to 64GB each, 256GB total.
Setup uses MTP with little-MoE default.
Generation reaches 110 t/s with MTP optimistic.
- generation:
- 29.0 tokens/s
- prompt processing (prefill):
- 450.0 tokens/s
- quant:
- Q4_K_XL
User reports testing multiple models on 4x CMP 170HX 64GB cards, including DeepSeek V4-Flash 0731 with a Q4_K_XL quant, 13B active MoE, and 1M context, plain no-spec.
The cards are cut-down A100 mining cards with 64GB each, PCIe Gen2 x4, and no NVLink.
Other models tested include gpt-oss-120B, Qwen3.6-35B-A3B, GLM-4.5-Air, and MiniMax-M2.7.
- prompt processing (prefill):
- 3468.0 tokens/s
- quant:
- FP8
User reports an unlocked CMP 170HX, A100 silicon with a firmware unlock, reaching 193 TFLOPS tensor throughput, up from 6.3 TFLOPS.
In llama.cpp, pp512 rises from 599.6 to 3468 tok/s.
Serving Qwen3.8-27B-FP8 under vLLM, one 170HX beats a 2x3090 tensor-parallel pair on prefill at 197W versus 454W.