TensorSharp benchmarks
An inference engine for running open-weight LLMs locally.
17 community reports
TensorSharp results by GPU
Hardware with reported TensorSharp results, with each report's model, quant and speed, newest first. Runs on several cards, with speculative decoding, with batched requests or with part of the model in system RAM say so, since each describes a different setup. Prompt processing is how fast the input is read, not how long you wait for the first word. How to measure time to first token.
TensorSharp on NVIDIA A40 48GB7 reports
- MiniMax M3.1 428B · UD-Q4_K_XL22.9 t/s, 2,567 t/s prompt(8 cards)
- MiniMax M3.1 · UD-Q4_K_XL22.9 t/s, 2,567 t/s prompt(8 cards)
- DeepSeek V4.1 Flash 552B · 16B active · UD-IQ2_XXS35.5 t/s, 961 t/s prompt(5 cards, part in system RAM)
- DeepSeek V4.1 Flash 552B · 16B active · UD-IQ2_XXS564 t/s prompt(5 cards, part in system RAM)
- DeepSeek V4.1 Flash 552B · 16B active · Q2_K41.1 t/s(8 cards)
TensorSharp on NVIDIA RTX 50902 reports
- Qwen3.8 27B · NVFP4234.4 t/s(speculative)
- Qwen3.8 27B · NVFP480.0 t/s
TensorSharp on NVIDIA A100 80GB1 report
- Qwen3.8 125B · 6B active · UD-Q2_K_XL56.8 t/s, 1,019 t/s prompt(2 cards)
TensorSharp on NVIDIA RTX 2000 Ada1 report
- Gemma 4 · Q8_051.7 t/s, 2,488 t/s prompt(2 cards)
TensorSharp on NVIDIA RTX 3080 Laptop 16GB1 report
- Qwen3.8 125B · 6B active11.1 t/s(part in system RAM)
This engine doesn't yet have an editorial profile on llamaperf. The community reports below show how it's been used in practice across different hardware.
Compare GPUs running TensorSharp
Every card is compared on one model, Qwen3.8 27B at 4-bit, the size and bit class that most cards here have plain runs of. A card with no run of that model shows none rather than a figure from a different model.
| GPU | VRAM | Reports | Median speed, Qwen3.8 27B 4-bit |
|---|---|---|---|
| NVIDIA A40 48GBnvidia | 48GB | 8 | no Qwen3.8 27B 4-bit run reported |
| M5 Pro 48GBapple | 48GB | 3 | no Qwen3.8 27B 4-bit run reported |
| NVIDIA RTX 5090nvidia | 32GB | 2 | Gen 80.0 t/s |
| NVIDIA A100 80GBnvidia | 80GB | 1 | no Qwen3.8 27B 4-bit run reported |
| NVIDIA RTX 2000 Adanvidia | 16GB | 1 | no Qwen3.8 27B 4-bit run reported |
| NVIDIA RTX 3080 Laptop 16GBnvidia | 16GB | 1 | no Qwen3.8 27B 4-bit run reported |
| NVIDIA RTX Pro 6000 Blackwellnvidia | 96GB | 1 | no Qwen3.8 27B 4-bit run reported |
TensorSharp against other engines
Pairs of plain runs on the same card, of the same model size at the same bit class: one device, one request, no speculative decoding, the whole model in memory. Context length and build still differ between the two sides, and each side shows its own.
| Model and GPU | TensorSharp | Other engine |
|---|---|---|
| Qwen3.8 27B 4-bitNVIDIA RTX 5090 | 80.0 t/sone generation report, 40K context | llama.cpp84.2 t/smedian of 3 generation reports, 40K to 120K context, 1 not stated |
| Qwen3.8 27B 4-bitNVIDIA RTX 5090 | 80.0 t/sone generation report, 40K context | vLLM73.2 t/sone generation report, context not stated |
Top models on TensorSharp
Frequently asked
Is TensorSharp faster than other engines?
llamaperf has 2 matched pairs of TensorSharp and other-engine runs on the same card, model size and bit class, but none is comparable enough to say which engine is faster: the sides have too few reports or were run at different context lengths. A pair counts toward a direction only when each side has at least 3 generation reports with a stated context, and the two sides' contexts overlap. The measured figures and their conditions are in the table on this page.