Local model performance on your hardware

generation:

198.0 tokens/s

prompt processing (prefill):

5772.2 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

Source and attribution: How Quant site maintainer's first-person benchmark, https://howquant.dev/benchmarks/rtx-pro-5000-llama-cpp/ (article first published 2026-08-27, updated 2026-09-20; rechecked 2026-10-13). About/author context: https://howquant.dev/about/ . This report covers only the Qwen3.6-35B-A3B Q8_0 / 1×72 row, measured 2026-09-14. Model: Qwen3.6-35B-A3B Q8_0, GGUF; source Params label 35 B (3 B active); source file size 35.2 GiB. 35 B total and 3 B active are the source table labels. The selected 198.0 t/s result is the September 14 table row; the prose also mentions a 197.8 t/s matched-session control, which is not substituted or merged here. No exact benchmark GGUF filename is disclosed. Upstream/base-model reference: https://huggingface.co/Qwen/Qwen3.6-35B-A3B . This is not an identified tested Q8_0 GGUF download. Tested GGUF publisher/repository, revision and checksum are unknown; any current upstream revision is not the tested revision. Hardware: one active NVIDIA RTX PRO 5000 Blackwell 72 GB, GPU0; whole model placed on that card, --device CUDA0 -sm none. The host also contains a 48 GB GPU1, but it is not part of this single-card placement. 72 GB is nominal device capacity, not measured resident usage. Intel Core Ultra 5 250K Plus (18 hardware threads, iGPU disabled); 96 GB installed RAM (4×24 GB DDR5-6000); ASUS ProArt Z890-CREATOR WIFI, CPU-attached PCIe Gen5 x8 slot; Solidigm P44 Pro 2 TB NVMe. GPU power limit 250 W, below the stated 300 W factory default. Benchmark CPU-thread count is unknown. Runtime: Ubuntu 26.04 LTS, kernel 7.0.0-31-generic; NVIDIA driver 595.84; CUDA 13.2 as reported by nvidia-smi, nvcc V13.3.73. llama.cpp CUDA b10941, commit 4a8993735; native llama-bench. Source common settings: -ngl 999 -b 2048 -ub 512 -fa on -ctk f16 -ctv f16. Batch 2048, microbatch 512, flash attention on, K and V cache f16. Kernel boot settings intel_iommu=off iommu=off are reported for this system. The author says GGML_CUDA_P2P=1 was exported for most runs; its state in this exact single-GPU run is unknown. Workload and results: separate synthetic prompt-processing tests PP512 5772.2 t/s, PP2048 5547.5 t/s, PP8192 5365.6 t/s, and separate token-generation test TG128 198.0 t/s. Each cell is the mean of five repetitions (-r 5). PP512 and TG128 are the headline pair from the same placement, not a combined 512-input/128-output request. The 512/2048/8192 values are newly processed tokens, and 128 is generated tokens in the separate TG test. Configured maximum context, pre-existing KV/cache depth, parallel requests and exact warmup/timing boundaries are not disclosed. No per-cell standard deviations or raw timings are shown. This is author-reported native throughput, not measured end-to-end API latency. Reproduction shape only: llama-bench -m <tested-Q8_0-GGUF> -ngl 999 -b 2048 -ub 512 -fa on -ctk f16 -ctv f16 --device CUDA0 -sm none -r 5 -p 512,2048,8192 -n 128. This combines the page's documented common settings, single-GPU command example and repetition note; it is not a captured historical invocation. The author claims each table cell comes from stored run JSON with command lines and logs, but those raw artifacts were not independently inspected or linked in the captured page. Native-method provenance is the author's claim, supported by the visible source table and methodology, not a raw-log audit. Memory and comparability: 35.2 GiB is disk/model-file size, not peak resident VRAM or system RAM; both peak resident measurements are unknown. Whole-model single-GPU placement is source-reported, not independently measured residency. The matrix spans sessions/days; the author reports generation reproducibility around 1% and prompt-processing drift up to about 10% across sessions, so cross-session PP comparisons need caution. All four PP/TG cells above stay together in this one configuration report. Do not import the mixed-72+48 GB placements, another runtime, speculative/MTP results, or advertised model context into these measurements.

generation:

33.2 tokens/s

prompt processing (prefill):

2145.5 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

Source and attribution: How Quant site maintainer's first-person benchmark, https://howquant.dev/benchmarks/rtx-pro-5000-llama-cpp/ (article first published 2026-08-27, updated 2026-09-20; rechecked 2026-10-13). About/author context: https://howquant.dev/about/ . This report covers only the Gemma 4 31B it Q8_0 / 1×72 row, measured 2026-09-15. Model: Gemma 4 31B it Q8_0, GGUF; source Params label 30.7 B; source file size 30.4 GiB. 31B is the nominal model name; 30.7 B is the benchmark table count. No exact benchmark GGUF filename is disclosed. Upstream/base-model reference: https://huggingface.co/google/gemma-4-31B-it . This is not an identified tested Q8_0 GGUF download. Tested GGUF publisher/repository, revision and checksum are unknown; any current upstream revision is not the tested revision. Hardware: one active NVIDIA RTX PRO 5000 Blackwell 72 GB, GPU0; whole model placed on that card, --device CUDA0 -sm none. The host also contains a 48 GB GPU1, but it is not part of this single-card placement. 72 GB is nominal device capacity, not measured resident usage. Intel Core Ultra 5 250K Plus (18 hardware threads, iGPU disabled); 96 GB installed RAM (4×24 GB DDR5-6000); ASUS ProArt Z890-CREATOR WIFI, CPU-attached PCIe Gen5 x8 slot; Solidigm P44 Pro 2 TB NVMe. GPU power limit 250 W, below the stated 300 W factory default. Benchmark CPU-thread count is unknown. Runtime: Ubuntu 26.04 LTS, kernel 7.0.0-31-generic; NVIDIA driver 595.84; CUDA 13.2 as reported by nvidia-smi, nvcc V13.3.73. llama.cpp CUDA b10941, commit 4a8993735; native llama-bench. Source common settings: -ngl 999 -b 2048 -ub 512 -fa on -ctk f16 -ctv f16. Batch 2048, microbatch 512, flash attention on, K and V cache f16. Kernel boot settings intel_iommu=off iommu=off are reported for this system. The author says GGML_CUDA_P2P=1 was exported for most runs; its state in this exact single-GPU run is unknown. Workload and results: separate synthetic prompt-processing tests PP512 2145.5 t/s, PP2048 2093.5 t/s, PP8192 1987.9 t/s, and separate token-generation test TG128 33.2 t/s. Each cell is the mean of five repetitions (-r 5). PP512 and TG128 are the headline pair from the same placement, not a combined 512-input/128-output request. The 512/2048/8192 values are newly processed tokens, and 128 is generated tokens in the separate TG test. Configured maximum context, pre-existing KV/cache depth, parallel requests and exact warmup/timing boundaries are not disclosed. No per-cell standard deviations or raw timings are shown. This is author-reported native throughput, not measured end-to-end API latency. Reproduction shape only: llama-bench -m <tested-Q8_0-GGUF> -ngl 999 -b 2048 -ub 512 -fa on -ctk f16 -ctv f16 --device CUDA0 -sm none -r 5 -p 512,2048,8192 -n 128. This combines the page's documented common settings, single-GPU command example and repetition note; it is not a captured historical invocation. The author claims each table cell comes from stored run JSON with command lines and logs, but those raw artifacts were not independently inspected or linked in the captured page. Native-method provenance is the author's claim, supported by the visible source table and methodology, not a raw-log audit. Memory and comparability: 30.4 GiB is disk/model-file size, not peak resident VRAM or system RAM; both peak resident measurements are unknown. Whole-model single-GPU placement is source-reported, not independently measured residency. The matrix spans sessions/days; the author reports generation reproducibility around 1% and prompt-processing drift up to about 10% across sessions, so cross-session PP comparisons need caution. All four PP/TG cells above stay together in this one configuration report. Do not import the mixed-72+48 GB placements, another runtime, speculative/MTP results, or advertised model context into these measurements.

generation:

38.5 tokens/s

prompt processing (prefill):

2677.5 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

Source and attribution: How Quant site maintainer's first-person benchmark, https://howquant.dev/benchmarks/rtx-pro-5000-llama-cpp/ (article first published 2026-08-27, updated 2026-09-20; rechecked 2026-10-13). About/author context: https://howquant.dev/about/ . This report covers only the Muse Glimmer 30B Q8_0 / 1×72 row, measured 2026-09-15. Model: Muse Glimmer 30B Q8_0, GGUF; source Params label 27.9 B; source file size 27.6 GiB. 30B is the nominal model name; 27.9 B is the benchmark table count. Do not substitute the upstream full multimodal model size for that measured-file count. No exact benchmark GGUF filename is disclosed. Upstream/base-model reference: https://huggingface.co/meta-models/Muse-Glimmer-30B . This is not an identified tested Q8_0 GGUF download. Tested GGUF publisher/repository, revision and checksum are unknown; any current upstream revision is not the tested revision. Hardware: one active NVIDIA RTX PRO 5000 Blackwell 72 GB, GPU0; whole model placed on that card, --device CUDA0 -sm none. The host also contains a 48 GB GPU1, but it is not part of this single-card placement. 72 GB is nominal device capacity, not measured resident usage. Intel Core Ultra 5 250K Plus (18 hardware threads, iGPU disabled); 96 GB installed RAM (4×24 GB DDR5-6000); ASUS ProArt Z890-CREATOR WIFI, CPU-attached PCIe Gen5 x8 slot; Solidigm P44 Pro 2 TB NVMe. GPU power limit 250 W, below the stated 300 W factory default. Benchmark CPU-thread count is unknown. Runtime: Ubuntu 26.04 LTS, kernel 7.0.0-31-generic; NVIDIA driver 595.84; CUDA 13.2 as reported by nvidia-smi, nvcc V13.3.73. llama.cpp CUDA b10941, commit 4a8993735; native llama-bench. Source common settings: -ngl 999 -b 2048 -ub 512 -fa on -ctk f16 -ctv f16. Batch 2048, microbatch 512, flash attention on, K and V cache f16. Kernel boot settings intel_iommu=off iommu=off are reported for this system. The author says GGML_CUDA_P2P=1 was exported for most runs; its state in this exact single-GPU run is unknown. Workload and results: separate synthetic prompt-processing tests PP512 2677.5 t/s, PP2048 2644.6 t/s, PP8192 2571.7 t/s, and separate token-generation test TG128 38.5 t/s. Each cell is the mean of five repetitions (-r 5). PP512 and TG128 are the headline pair from the same placement, not a combined 512-input/128-output request. The 512/2048/8192 values are newly processed tokens, and 128 is generated tokens in the separate TG test. Configured maximum context, pre-existing KV/cache depth, parallel requests and exact warmup/timing boundaries are not disclosed. No per-cell standard deviations or raw timings are shown. This is author-reported native throughput, not measured end-to-end API latency. Reproduction shape only: llama-bench -m <tested-Q8_0-GGUF> -ngl 999 -b 2048 -ub 512 -fa on -ctk f16 -ctv f16 --device CUDA0 -sm none -r 5 -p 512,2048,8192 -n 128. This combines the page's documented common settings, single-GPU command example and repetition note; it is not a captured historical invocation. The author claims each table cell comes from stored run JSON with command lines and logs, but those raw artifacts were not independently inspected or linked in the captured page. Native-method provenance is the author's claim, supported by the visible source table and methodology, not a raw-log audit. Memory and comparability: 27.6 GiB is disk/model-file size, not peak resident VRAM or system RAM; both peak resident measurements are unknown. Whole-model single-GPU placement is source-reported, not independently measured residency. The matrix spans sessions/days; the author reports generation reproducibility around 1% and prompt-processing drift up to about 10% across sessions, so cross-session PP comparisons need caution. All four PP/TG cells above stay together in this one configuration report. Do not import the mixed-72+48 GB placements, another runtime, speculative/MTP results, or advertised model context into these measurements.

generation:

38.6 tokens/s

prompt processing (prefill):

2275.9 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

Source and attribution: How Quant site maintainer's first-person benchmark, https://howquant.dev/benchmarks/rtx-pro-5000-llama-cpp/ (article first published 2026-08-27, updated 2026-09-20; rechecked 2026-10-13). About/author context: https://howquant.dev/about/ . This report covers only the Qwen3.8 27B Q8_0 / 1×72 row, measured 2026-09-14. Model: Qwen3.8 27B Q8_0, GGUF; source Params label 27 B dense; source file size 27.1 GiB. The source command example names Qwen3.8-27B-Q8_0.gguf, but gives no tested publisher, revision or checksum. Upstream/base-model reference: https://huggingface.co/Qwen/Qwen3.8-27B . This is not an identified tested Q8_0 GGUF download. Tested GGUF publisher/repository, revision and checksum are unknown; any current upstream revision is not the tested revision. Hardware: one active NVIDIA RTX PRO 5000 Blackwell 72 GB, GPU0; whole model placed on that card, --device CUDA0 -sm none. The host also contains a 48 GB GPU1, but it is not part of this single-card placement. 72 GB is nominal device capacity, not measured resident usage. Intel Core Ultra 5 250K Plus (18 hardware threads, iGPU disabled); 96 GB installed RAM (4×24 GB DDR5-6000); ASUS ProArt Z890-CREATOR WIFI, CPU-attached PCIe Gen5 x8 slot; Solidigm P44 Pro 2 TB NVMe. GPU power limit 250 W, below the stated 300 W factory default. Benchmark CPU-thread count is unknown. Runtime: Ubuntu 26.04 LTS, kernel 7.0.0-31-generic; NVIDIA driver 595.84; CUDA 13.2 as reported by nvidia-smi, nvcc V13.3.73. llama.cpp CUDA b10941, commit 4a8993735; native llama-bench. Source common settings: -ngl 999 -b 2048 -ub 512 -fa on -ctk f16 -ctv f16. Batch 2048, microbatch 512, flash attention on, K and V cache f16. Kernel boot settings intel_iommu=off iommu=off are reported for this system. The author says GGML_CUDA_P2P=1 was exported for most runs; its state in this exact single-GPU run is unknown. Workload and results: separate synthetic prompt-processing tests PP512 2275.9 t/s, PP2048 2272.4 t/s, PP8192 2206.7 t/s, and separate token-generation test TG128 38.6 t/s. Each cell is the mean of five repetitions (-r 5). PP512 and TG128 are the headline pair from the same placement, not a combined 512-input/128-output request. The 512/2048/8192 values are newly processed tokens, and 128 is generated tokens in the separate TG test. Configured maximum context, pre-existing KV/cache depth, parallel requests and exact warmup/timing boundaries are not disclosed. No per-cell standard deviations or raw timings are shown. This is author-reported native throughput, not measured end-to-end API latency. Reproduction shape only: llama-bench -m <tested-Q8_0-GGUF> -ngl 999 -b 2048 -ub 512 -fa on -ctk f16 -ctv f16 --device CUDA0 -sm none -r 5 -p 512,2048,8192 -n 128. This combines the page's documented common settings, single-GPU command example and repetition note; it is not a captured historical invocation. The author claims each table cell comes from stored run JSON with command lines and logs, but those raw artifacts were not independently inspected or linked in the captured page. Native-method provenance is the author's claim, supported by the visible source table and methodology, not a raw-log audit. Memory and comparability: 27.1 GiB is disk/model-file size, not peak resident VRAM or system RAM; both peak resident measurements are unknown. Whole-model single-GPU placement is source-reported, not independently measured residency. The matrix spans sessions/days; the author reports generation reproducibility around 1% and prompt-processing drift up to about 10% across sessions, so cross-session PP comparisons need caution. All four PP/TG cells above stay together in this one configuration report. Do not import the mixed-72+48 GB placements, another runtime, speculative/MTP results, or advertised model context into these measurements.