Alibaba · 8 reports
Qwen3.5 VRAM requirements by size and quant →Unknown GPU
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports Qwen3.5 0.8B BF16 at roughly 3-3.2 t/s on a 7800X3D with 32 GB DDR5, running CPU-only single-threaded inference. Time to first token grows non-monotonically and tracks prompt token count.
RTX 3060 12GB · llama.cpp
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports running llama.cpp's ggml-rpc backend across a heterogeneous cluster pooling RAM and VRAM from an Acer laptop CPU, a Windows RTX 3060 on CUDA, and a Mac Mini on Metal. The primary API server runs on the weakest machine, and the setup uses llama.cpp's built-in benchmark script. The project is source-available under the Commons Clause.
Unknown GPU · qwen35-cpu
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports a custom engine and quant format for CPU inference at 122.9 t/s decode at batch size 1 and 642 t/s at batch size 16. The run is compared against llama.cpp and ik_llama.cpp. It measures PPL 15.80 and KL 0.0602.
2× AMD Strix Halo 128GB · llama.cpp · 32,768 ctx
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports a hybrid setup with Strix Halo 128GB and an R9700 32GB over x4 PCIe. Dense layers, KV cache, and the MTP drafter run on the R9700, while routed experts run on the Strix, using a custom llama.cpp fork. The user compares this to stock at 24 tok/s.
User benchmarks Qwen3.5-9B, NVIDIA-Nemotron-3.5-Lightning-30B-A3B and Gemma-4-12B-it on a single RTX 5090. Nemotron-3.5-Lightning-30B-A3B runs as a UD-Q4_K_XL GGUF and Gemma-4-12B-it as QAT w4a16. Qwen3.5-9B scores 0.717 on GPQA Diamond and 0.713 on LiveCodeBench.
2× D700 12GB · llama.cpp · 70,000 ctx
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports Qwen 2.5 Coder Q4 at 22 t/s. User compares Qwen 3.5 favorably to Claude Sonnet 4.6 for planning tasks.
User reports a new MLX quantization method, WinterMix, for Qwen3.5-122B-A10B, with two builds: WinterMix58 at 82 GiB and WinterMix48 at 68 GiB. Benchmarks show perplexity improvements over existing MLX quants. The user is enthusiastic about the results and the method's advantages for agentic workflows on Apple Silicon.
User reports a model based on Qwen3.5-4B, trained on 8xH100 for 3 days. It supports Safetensors, GGUF and MLX weights, requires as little as 4GB VRAM, and is available in GPTQ, W8A8, FP8, Q4 and Q6 quantizations. It has been tested with vLLM, SGLang and llama.cpp.