llamaperf

Hy4-Flash

on 2× NVIDIA RTX 3090 · llama.cpp · 131,072 ctx

Tone: positiveNVIDIA hardware
Nov 18, 2026
Throughput
20-24 t/s gen · 60-65 tokens/s prompt processing (prefill)Prompt processing is how fast the input is read, not how long you wait for the first word. How to measure time to first token.
Quant
Q3_K (GGUF)
KV cache
q8_0
System RAM
128 GB

Summary

User reports running Hy4-Flash Q3_K on 2x RTX 3090 with 128 GB DDR4 RAM, getting PP ~60-65 t/s and TG ~20-24 t/s. Setup is llama.cpp (llama-server router mode, a PR build) with Q3_K GGUF weights, q8_0 KV cache and 131072 context, using expert streaming via --moe-cache-mib 33768 and -cmoe, so experts run from system RAM. A logged run shows 4091 prompt tokens at 64.54 t/s and 100 generated tokens at 21.95 t/s. User says the model is slower but finishes answers with fewer tokens, and calls it a likely goto model.