llamaperf

Qwen3.5 4B Mica v0.1

on NVIDIA RTX 3090 · llama.cpp

NVIDIA hardware
Sep 25, 2026
Quant
Q4_K_M (GGUF)

Use cases

agentictool-usecodingmultilingual

Summary

User releases Mica v0.1 4B, a Qwen3.5-4B fine-tune with a rank-16 LoRA, as a decision model that returns calibrated probabilities for yes/no, choice, or score answers. It runs on an 8 GB GPU and was trained for under $30 of rented RTX 3090 time. Setup is llama.cpp with GGUF weights and a direct logits readout. The Q5_K_M file is 3.5 GB with no measurable accuracy loss against BF16 on the calibration set. On an RTX 3090 with one request at a time, median latency over 231 public JevBench items is 47 ms for Q4_K_M and 54 ms for BF16. On held-out English decisions Mica scores 67.0 versus 74.1 for Jev 1.13 and 55.0 for the Qwen3.5-4B base. On long inputs around 3.7k tokens Mica is slightly slower than JevK5.