llamaperf

Qwen3.6 27B

on NVIDIA RTX 3090 · Luce DFlash (custom C++/CUDA stack on ggml) · 256,000 ctx

Tone: positiveNVIDIA hardware
Apr 28, 2026
Throughput
913.0 tokens/s prompt processing (prefill)Prompt processing measures input prefill speed; it does not tell you the time to first token.
Quant
Q4_K_M (GGUF)
KV cache
Q4
Flash Attention
on
VRAM reported
24 GB

Use cases

codingmath

Summary

User reports speculative decoding with DFlash on a single RTX 3090, targeting Qwen3.6-27B Q4_K_M GGUF at ~16 GB with a z-lab Qwen3.6-27B-DFlash bf16 draft at ~3.46 GB. Setup is DDTree tree-verify with block size 16, budget 22, and greedy verify. The KV cache is compressed to TQ3_0 at 3.5 bpv, roughly 9.7x versus F16, with a 4096-slot ring buffer enabling 256K context in 24 GB. Decode uses sliding-window flash attention with a 2048-token window, and prefill ubatch auto-bumps from 16 to 192 for prompts over 2048 tokens. The endpoint is OpenAI-compatible and CUDA only, with no Metal, ROCm, or multi-GPU support. Output is bit-identical to autoregressive in AR mode, and the draft matches the z-lab PyTorch reference at cos sim 0.999812.