Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.
Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.
Free to use. No account needed. Memory estimates and community measurements are labelled separately.
Search for a model to see its reported speeds across GPUs and Macs.
These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →
Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.
User reports Ling-3.0 Flash at 73.1 t/s on an M3 Ultra 256GB. Setup is oMLX with oQ5 affine mixed precision, group 64, on a 124B total / 5.1B active model. The figure is the median of three greedy 256-token runs (73.23, 73.10, 73.09) after a 256-token warm-up, with a 39-token prompt; peak memory is reported at 87.44 GB.
NVIDIA T1000 8GB · llama.cpp · 16,384 ctx
User benchmarks Ling 3.0 tiny against Gemma 4 26B-A4B for batched JSON speaker attribution on an NVIDIA T1000 8GB. Ling 3.0 tiny runs at 245 t/s prefill and 54 t/s decode, 1.8x faster end to end than Gemma. Setup is llama.cpp with the official Q4_K_M GGUF at 16384 context, all 25 layers on the GPU, no speculative decoding. Gemma uses UD-Q4_K_XL with experts in system RAM and an MTP draft head. Ling scores 56.9% exact speaker match versus Gemma's 94.6% across 485 lines, with narrator swaps and refusals. User is not adopting it.
NVIDIA DGX Spark · llama.cpp · 262,144 ctx
User compares generation throughput on a 128 GB DGX Spark at three starting-context depths for two deployment paths. Official INT4 on a vLLM fork reaches 38.3 t/s short, 7.9 t/s at ~45K and 4.6 t/s at ~90K. Q5_K_M on llama.cpp reaches 35.7 t/s short, 33.6 t/s at ~45K and 33.2 t/s at ~90K. The config is 262,144 tokens of max context with ~103 GB total memory used, and streaming separates TTFT from decode. These are the creator's measurements, not an independent rerun, and two variables change at once (runtime and quantization), so no pure engine effect is isolated. The later 131,072-context serving default should not be attached to this table.
User benchmarks Ling-3.0-flash on a DGX Spark at 40.9 t/s on a short coding task. Setup is CUDA graphs with MTP n=1. Prose throughput is 38.7 t/s at 512-token output and 37.3 t/s at 2048-token output. MTP n=2 and n=3 are slower for prose. Acceptance lengths are 1.87 at n=1, 2.39 at n=2 and 2.77 at n=3. Without MTP the baseline is 22.9 t/s with CUDA graphs and 20.8 t/s eager.
User reports Ling-3.0-flash at 40.2 tok/s single stream decode with the Q5_K_M quant on one DGX Spark, from benchmarks posted by sudoingX on X. Setup is the community GGUF quant ladder, with 5.1B active out of 124B parameters. Q4_K_M reaches 38.2 tok/s and Q6_K 32.0 tok/s. DeepSeek V4 Flash measured 16.5 tok/s on the same box.
User reports Ling-3.0-flash (124B) at 38.7 tok/s on the official INT4 quant on one DGX Spark, from benchmarks posted by sudoingX on X. The community GGUF reaches 35.2 tok/s, and the result is 2.4x what DeepSeek V4 Flash does on the same machine. The benchmarker first posted that the official quants do not run on a single Spark, then corrected that two days later.
On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking