Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.5-4B think at ~150 t/s on an M4 Pro 48GB.
Setup is llama.cpp with Q4_K_M at 32k context, using 2.5 GB of RAM.
The model scores 6/7 on the CC-Agent harness and 14/14 on the V3.1 text suite, matching the 35B-A3B model at 1/8 the memory.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.5 35B-A3B at 8.34 t/s generation and 112.62 t/s prompt processing on an Intel Arc Pro B60 24GB.
Setup is llama.cpp build 8175 with the SYCL backend, Q4_K_XL GGUF weights, full GPU offload (-ngl 100).
User reports the card is ok for chat-length context on smaller models but slow for agentic tools like opencode/claude code, where initial prompt response can take 5-10 minutes. A comparison run of the same model on an RTX Pro 4500 Blackwell reached 133.47 t/s generation and 3807.62 t/s prompt processing.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5-35B-A3B at 47-51 tok/s generation on an RTX 5060 Ti 16GB over OCuLink in a Proxmox VM.
Setup is llama.cpp with UD-IQ3_XXS quant and q4_0 KV cache, fully on the GPU with 348 MiB headroom, at 160K context.
Prompt eval of 75K tokens took 64.8 s (1,168 tok/s). The 9B variant reached 40-50 tok/s with UD-Q4_K_XL and about 47-51 tok/s with IQ3_XXS.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
summarizationtool-use
User reports Qwen3.5 4B at ~85 tok/s decode on an M4 Max 48GB, running a Mac meeting notetaker stack.
Setup is llama.cpp with Q4_K_M weights; the full on-device stack totals 6.8 GB across five models.
A 26-minute meeting produced finished notes in 33 s warm. Speaker error dropped from 43.4% to 14.6% DER on AMI, and autocomplete p95 improved from 1.83 s to 0.49 s. MiniCPM5 2B was tried for notes but repeated itself and misassigned action items, so Qwen3.5 4B stayed the default.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.5 35B A3B running at roughly 17 t/s on a MacBook Pro M5 16GB.
Setup uses slotstream; no engine, quant or context length stated.
The figure is given as approximate.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 8 t/s with Qwen3.5 35B Q2 on an RTX 4060 Ti 8GB, with no CUDA device selected.
Setup is llama.cpp with a Q2 GGUF quant; the user notes VRAM usage was 3000MB/8k and that the run had no CUDA device selected.
The user is troubleshooting a cuBLAS crash and asks how to cleanly uninstall and reinstall llama.cpp with CUDA 12, and whether CUDA would improve generation speed.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
rag
User reports Qwen 3.5 3.5B at 34 t/s generation and 1000+ t/s prompt processing on two RTX 3060 12GB cards.
Setup uses a 4x oculink to mini PCIe 3x connection. User is building a new system with dual 16x lanes and NCCL.
User notes the cards are used for RAG and multiple purposes, and mentions the 12GB 3060 cards are becoming scarce and expensive.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.5 4B Q8_0 at 107 t/s generation and 4,445 t/s prompt processing on a Tesla V100 32GB.
Setup is llama.cpp (PrismML fork 3ae4f51, CUDA 12.9) with flash attention on, -ngl 99, tested at 0, 16K and 32K context; generation drops to 93 t/s at 32K.
An RTX 4070 control card reaches 91 t/s generation and 5,860 t/s prompt processing on the same model. The V100 draws 154 W while generating (0.69 tokens/joule) against the 4070's 173 W (0.53 tokens/joule).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.5-35B-A3B (MoE, 35B total, ~3.5B active) at 149.2 t/s decode and 3075 t/s prefill on an AMD Radeon AI PRO R9700 32GB.
Setup is llama.cpp (Vulkan, RADV driver) with UD-Q4_K_XL quant, flash attention on, ubatch 2048, and a one-line rm_kq=1 patch; single GPU.
The figures are the best RADV configuration (rm_kq=1 + ub2048); stock RADV gives 147.8 t/s decode and 2381 t/s prefill. AMDVLK with rm_kq=1 reaches 156.3 t/s decode but only 1823 t/s prefill. WikiText-2 perplexity is 6.9472 ± 0.046 across all configurations.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen3.5-9B at 62 t/s on an AMD Radeon RX 9070 XT (RDNA4/gfx1201) using llama-server with the Vulkan backend.
Setup is llama.cpp with GGUF Q6_K_XL weights at 65536 tokens of context, single request, no speculation. The same model under vLLM ROCm 7.2 with FP8 weights reached 48 t/s, which the user attributes to vLLM lacking native gfx1201 kernel support and falling back to FP32 dequantization.
The user reports llama-server Vulkan is 29% faster on this hardware and recommends it over vLLM until RDNA4 support lands upstream.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
multilingualtool-useagentic
User asks for recommendations on a small local model to replace Qwen3.5 4B as the reasoning layer of a local AI assistant.
User reports getting around 40-50 tokens/sec with Qwen3.5 4B on unspecified limited hardware, and values that responsiveness.
Priorities are general conversation, reasoning and instruction following, multilingual ability, context understanding, and reliable tool/function calling; the user is happy with the current model and wants a significant upgrade in the same memory and compute footprint.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User asks for help building an offline LLM rig and reports a test of Qwen3.5 9B at Q6_K on their main rig, getting 53 t/s prompt processing and 4.6 t/s generation with ik_llama.cpp.
Setup is ik_llama.cpp with Q6_K; the test rig's hardware is not specified. The planned build is dual Xeon 2683v4 with 256GB DDR4 ECC and a V100 16GB, targeting DeepSeek V4 Flash and MiMo V2.5 at UD-IQ4_XS.
User estimates 6-7 t/s generation and 50-60 t/s prompt processing for DeepSeek V4 Flash on CPU alone, hoping a GPU raises prompt processing to 200-300 t/s, and expects MiMo V2.5 to be 1-2 t/s slower. ik_llama.cpp doubled prompt processing and added 1 t/s generation over llama.cpp in their test.
User releases Mica v0.1 4B, a Qwen3.5-4B fine-tune with a rank-16 LoRA, as a decision model that returns calibrated probabilities for yes/no, choice, or score answers. It runs on an 8 GB GPU and was trained for under $30 of rented RTX 3090 time.
Setup is llama.cpp with GGUF weights and a direct logits readout. The Q5_K_M file is 3.5 GB with no measurable accuracy loss against BF16 on the calibration set.
On an RTX 3090 with one request at a time, median latency over 231 public JevBench items is 47 ms for Q4_K_M and 54 ms for BF16. On held-out English decisions Mica scores 67.0 versus 74.1 for Jev 1.13 and 55.0 for the Qwen3.5-4B base. On long inputs around 3.7k tokens Mica is slightly slower than JevK5.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User compares Qwen3.5-35B-A3B Q4_K_XL on an RTX 5090 against a Radeon AI PRO R9700, both 32 GB, using llama-bench with -ngl 99, flash attention and q8_0 KV cache.
On the RTX 5090 generation is 194.0 t/s and prompt processing is 7,026 t/s at 512 tokens, 6,953 at 1,024, 6,958 at 4,096 and 6,461 at 32,768.
The R9700 reaches 127.4 t/s generation and 2,713 / 2,643 / 2,546 / 1,877 t/s prompt processing at the same context lengths, a 1.52x generation and up to 3.44x prefill gap. The user notes these are single-stream llama-bench figures and calls the R9700 comfortable for interactive use at roughly 2.66x lower cost.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.5 4B at 70.4 t/s on an M2 Max using native GGUF support in transformers.
Setup is the Q4_K_M GGUF from unsloth, reusing ggml kernels on Apple Silicon. llama.cpp reached 71.8 t/s on the same model.
Two other models were also tested: Qwen3.8 27B UD-Q4_K_M at 15.9 t/s (vs 13.4 t/s with llama.cpp) and Qwen3.5 35B-A3B UD-IQ4_XS at 60.2 t/s (vs 61.3 t/s).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.5 0.8B BF16 at roughly 3-3.2 t/s on a 7800X3D with 32 GB DDR5, running CPU-only single-threaded inference.
Time to first token grows non-monotonically and tracks prompt token count.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports running llama.cpp's ggml-rpc backend across a heterogeneous cluster pooling RAM and VRAM from an Acer laptop CPU, a Windows RTX 3060 on CUDA, and a Mac Mini on Metal.
The primary API server runs on the weakest machine, and the setup uses llama.cpp's built-in benchmark script.
The project is source-available under the Commons Clause.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports a custom engine and quant format for CPU inference at 122.9 t/s decode at batch size 1 and 642 t/s at batch size 16.
The run is compared against llama.cpp and ik_llama.cpp.
It measures PPL 15.80 and KL 0.0602.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports a hybrid setup with Strix Halo 128GB and an R9700 32GB over x4 PCIe.
Dense layers, KV cache, and the MTP drafter run on the R9700, while routed experts run on the Strix, using a custom llama.cpp fork.
The user compares this to stock at 24 tok/s.
User benchmarks Qwen3.5-9B, NVIDIA-Nemotron-3.5-Lightning-30B-A3B and Gemma-4-12B-it on a single RTX 5090.
Nemotron-3.5-Lightning-30B-A3B runs as a UD-Q4_K_XL GGUF and Gemma-4-12B-it as QAT w4a16.
Qwen3.5-9B scores 0.717 on GPQA Diamond and 0.713 on LiveCodeBench.