llamaperf

Qwen3 0.6B

on NVIDIA RTX 3090

NVIDIA hardware
Oct 9, 2026
Throughput
531.0 t/s gen

Summary

User reports Qwen3-0.6B at 531 tok/s decode on an RTX 3090 using a custom CUDA megakernel, 3.9x faster than HuggingFace. Setup is a batch=1 bf16 cooperative megakernel with static shapes and short context; the project is a learning exercise, not a production engine. Decode falls to 422 tok/s at position 200 and 158 tok/s at longer contexts, where TensorRT-LLM's 355 tok/s is faster. The kernel is latency-bound by synchronization at 5% memory bandwidth utilization.