How to evaluate a local coding model on your repository

Tasks from work you know

Pull five or so tasks from your own backlog or history. A bug you've already fixed is the best kind, since you know what the right answer looks like, and a small, well-defined feature is nearly as good. Each one should be something you can judge in a few minutes and cover with tests you can run.

Before the first run, write down what done means for each task: the behaviour you expect, the files that should stay untouched, the tests that should pass and any command it mustn't run. Every candidate gets the same instructions and starts from the same commit.

Use a throwaway checkout

Do all of this in a disposable clone with no production credentials in it, and read every command the model wants to run before you let it. If a patch goes wrong, a git reset puts things back and your real work never notices. Starting each run from the same state also means you can repeat the test later.

A fixed context package

Write down which files, tool output and chat history the model gets. Hand one model the failing test and the right source file, make another find everything itself, and you've run two different tests.

For a first comparison, give every model the same fixed bundle of files. An agent that searches the repo on its own is a separate experiment, and when you get to one, note its tools and permissions too. The tools around a model move the results about as much as the model does.

Record the setup

Save the exact model build, quantization, runtime and context size. The coding reports show setups other people run on hardware like yours. They're community tags, so read them as ideas to try on your own repo. Choosing between two files of one model? The quantization test follows the same steps.

Run the tests, then read the diff

Passing tests are good evidence. You still read the diff, to check it handles the case in the ticket and leaves alone any interface or code it had no reason to change. Formal benchmarks start from the same place:

"SWE-bench evaluates models by applying their generated patches to real-world repositories and running the repository's tests to verify if the issue is resolved." (SWE-bench evaluation guide)

SWE-bench Verified, the best-known version, has 500 tasks (dataset on Hugging Face). Five tasks from your repo won't give you a score you can set beside that, and they don't need to. They answer the narrower question you care about when you're picking a model: does it help on your code?

Count the repair work

Note how much you had to fix. Merging after a quick read and rewriting half the patch are very different outcomes, even when both end with the tests passing.

Keep the model's explanation as a note beside the score. However sure of itself it sounds, it can't tell you the change works.

Time to a merged patch

Compare total task time, with retries and your own review included. A quick answer that needs three rounds of fixes can cost you more than a slower one that lands first time, and tokens per second is a small slice of that total.

Keep a running record

Write down which jobs each setup handles well enough that you'd rely on it, and where you still reach for something else. When you change the model or runtime, rerun the same tasks. Add new ones as your work shifts, and keep the old results labelled with their versions so you can see what moved.

The calculator narrows the field to models that fit your hardware, and the benchmark guide is worth a read before you trust a leaderboard number. Past that point it's your repo's call. A coding report helps the next reader most when it says what kind of change the model handled, so if you share your results, put the task types next to the setup.