Hy4-Flash
on NVIDIA RTX 3060 12GB · RustingStrata · 20,000 ctx
Use cases
coding
Summary
User reports Hy4-flash at 3.3 t/s generation and 11.6 t/s prompt reading on an RTX 3060 12GB with 32 GB of RAM, using a 20,000-token prompt and 64 tokens of output.
Setup is RustingStrata with an IQ2_S/IQ2_XXS GGUF of about 85 GB, streamed from an NVMe drive with experts split between CPU and GPU. The Vulkan backend on the same card gave 1.9 t/s generation and 10.6 t/s prompt reading.
After a short prompt the model writes 4.3 to 5.1 t/s on CUDA. The user notes one token needs about 2.6 GB of expert weights and calls it not suitable for a coding agent, though the full model runs on a 12 GB card. llama.cpp support was still a draft pull request, so no comparison was made.