TurboQuant: Extreme KV Cache Quantization to Under 3 Bits

Google Research published a blog and paper on TurboQuant, a new algorithm that quantizes the KV cache to under 3 bits with minimal accuracy loss, enabling larger context lengths on limited hardware.