Summary
User reports Qwen3-0.6B at 531 tok/s decode on an RTX 3090 using a custom CUDA megakernel, 3.9x faster than HuggingFace.
Setup is a batch=1 bf16 cooperative megakernel with static shapes and short context; the project is a learning exercise, not a production engine.
Decode falls to 422 tok/s at position 200 and 158 tok/s at longer contexts, where TensorRT-LLM's 355 tok/s is faster. The kernel is latency-bound by synchronization at 5% memory bandwidth utilization.