Use cases
coding
Summary
User reports 140 t/s writing code with a custom inference engine built from scratch for Qwen3.5-27B on an RTX 3090 24GB, with CPU offload.
Setup uses the unsloth Q4_K_M GGUF weights with additional 8-bit quantization of the KV cache and hidden states.
Coding with a 4K-token file in context drops to 95 t/s, and prefill is about 1,600 tok/s. The user compares these figures against llama.cpp with MTP on the same hardware.