How to Evaluate a Local LLM for a Real Task: A Repeatable Testing Framework

Local LLM evaluation is the discipline of determining whether a specific model, running on your specific hardware at a specific quantization level, can do your specific task well enough to ship. Every "specific" in that sentence is load-bearing, and together they outweigh any leaderboard: an MMLU score tells you how a model performs across a population of academic questions; it tells you almost nothing about whether the 4-bit build you can actually fit on your GPU will correctly extract vendor_name, line_items, and due_date from a scanned invoice with a smudged header.

The task doesn't care whether you're running Qwen3, Gemma 3, Llama 4, or a fine-tune of any of them. If you have a real task and a Friday deadline, this is the framework you run.

Why Population Benchmarks Are Professionally Useless for Deployment#

Benchmarks like MMLU and GPQA, and the public chat-arena leaderboards, are sampling instruments. They sample from broad capability distributions to rank models relative to each other, and they are genuinely useful for that: a researcher comparing architectures across thousands of tasks needs exactly this kind of aggregate. You are not that researcher.

Here is the concrete failure mode: a model can score near the top of a general reasoning benchmark while catastrophically failing a narrow extraction task, because its training rewarded verbose, hedged, discursive answers — exactly the wrong behavior for a pipeline that expects a bare JSON blob. Conversely, a smaller model with a lower benchmark rank may have seen more structured-output training signal and will outperform the "better" model on your task every single time.

There is a second gap that is specific to running models yourself. Leaderboard numbers are almost always produced from the full-precision model served on someone else's datacenter GPUs. You are going to run a quantized build — a 4-bit GGUF, an AWQ pack — on your own hardware, through your own inference server, with your own chat template. That is not the model the leaderboard measured. The rank ordering you copied off a table was established on an artifact you will never actually run.

The deeper problem is that benchmark scores are aggregate statistics, and your task is not a population. Your task is a point. A model that gets 73% on a mixed-domain benchmark might get 20% on your invoice corpus, or 95% — the benchmark number gives you no information to distinguish those outcomes.

The professional move is to stop consulting the leaderboard as an oracle and start running task-fidelity tests. The benchmark tells you which models are worth pulling down for evaluation. Your eval tells you which one to deploy.

Building a Golden Set: Minimum Viable Coverage#

A golden set is a versioned, labeled collection of inputs with known-correct outputs that you own and control. Building one well is most of the work.

The best source of examples is your own logs. Real inputs that have already flowed through your system — or through the manual process you're replacing — carry the exact distribution of mess you need to test against. Synthetic examples you invent tend to cluster on the happy path, which is precisely the region you least need to probe.

Minimum viable set size depends on output variance. For a deterministic extraction task (pull five fields from an invoice), around 40–60 examples is enough to get a meaningful signal — provided they cover the right distribution. For open-ended generation tasks, you need more, because scoring is harder and variance is higher. Do not start with fewer than 30 examples for any task; below that, a single fluke dominates your pass rate.

Coverage structure should follow three tiers:

• Happy path: Clean, representative inputs that any functional model should handle. These are your baseline. If a model fails here, the conversation is over.
• Edge cases: Inputs at the boundary of your task definition — an invoice with no due date, a line item spanning two lines, a currency in a non-default locale. These reveal where the model's implicit assumptions diverge from your data.
• Adversarial inputs: Inputs designed to trigger known failure modes — a prompt-injection string in the invoice footer, a document that is 90% boilerplate and 10% signal, an input that exceeds 80% of your context window. These are not exotic; they are what production data looks like after six months.

Aim for roughly 50% happy path, 30% edge case, 20% adversarial. Adjust after your first run based on where failures cluster.

Version control is non-negotiable. Store your golden set in a git repository. Tag each example with a UUID, a schema version, and a human-readable label describing what the example tests. When you add examples, do not modify existing ones — append only, and bump the set version. This discipline means a run from three months ago, against an older model, is still directly comparable to today's run. Without it, you will not know whether a change in pass rate came from the model or from the set drifting under you.

The Local LLM Failure-Mode Taxonomy#

Generic evaluation frameworks treat failure as a binary: right or wrong. Local LLM failures are more specific, and diagnosing them correctly determines your remediation path.

Prompt-format / chat-template mismatch: Every instruction-tuned model was fine-tuned against an exact chat template — specific role markers, special tokens, and usually a leading BOS token (<|im_start|>-style markers for some families, [INST] for others). A hosted API applies that template for you. When you run the raw weights yourself, you own it, and it is astonishingly easy to get wrong: a GGUF shipped with the wrong template baked into its metadata, an inference server that silently drops the BOS token, a hand-rolled prompt that omits a role marker. The model then sees a format it was never trained on and quietly degrades — worse instruction-following, more invented structure, more refusals. This failure is dangerous precisely because it imitates every other failure in this list. Rule it out first: confirm the exact template your model family expects, and inspect the fully-rendered string your server actually sends before you conclude anything about the model itself.

Hallucinated schema: The model returns valid-looking JSON but invents field names, nests data incorrectly, or returns a structure that parses but doesn't match your spec. This is distinct from a wrong value — the structure itself is wrong. Cause: insufficient schema grounding in the system prompt, or a model that has learned to prioritize fluent output over exact instruction compliance. Constrained decoding (a grammar or JSON-schema constraint enforced by your runtime) removes the structural failure entirely and lets you measure what you actually care about — whether the values are right.

Instruction bleed: Content from the system prompt leaks into the output, or instructions from a previous conversation turn contaminate the current one. This surfaces in multi-turn pipelines and is frequently caused by poorly delimited prompt templates. The model treats the boundary between instruction and content as porous.

Context-window degradation: Performance degrades measurably as inputs approach the model's context limit. Advertised context length is a ceiling, not a promise: a model that claims 128K tokens can start losing detail from the middle of a long input — the well-documented "lost in the middle" effect — long before you fill it. Local deployments make this worse, because to fit the model in memory you often run it at a reduced context size, and a KV cache under memory pressure degrades faster still. Test explicitly at 25%, 50%, 75%, and 95% of the context length you actually intend to run — not the number on the model card.

Quantization artifacts: This deserves first-class treatment, not a footnote. A model at 4-bit quantization (GGUF Q4_K_M, for instance) is a materially different model than its 8-bit or full-precision counterpart. Quantization compresses weight distributions in ways that disproportionately affect low-frequency tokens and rare patterns — precisely the patterns that appear in your edge cases. Run your golden set against every quantization level you plan to deploy, and remember that this is the variable the leaderboard never tested for you. Do not assume a 4-bit version of a well-performing 8-bit model degrades proportionally; the loss is nonlinear and task-dependent, and it lands hardest on exactly the rare patterns your edge cases are built from.

Refusal false-positives: The model refuses a legitimate task because its safety tuning misclassifies the input as harmful. Invoice extraction rarely triggers this, but document summarization, content moderation, and anything involving financial, medical, or legal language can. Log every refusal. If your false-positive refusal rate exceeds a few percent, you have a task-model alignment problem that prompt engineering alone may not solve.

Scoring Design: Rubrics Before You Run a Single Prompt#

Define your scoring rubric before you see any model output. Defining it after introduces motivated reasoning — you will unconsciously calibrate the rubric toward what the model actually did, not what your task requires.

Binary pass/fail is appropriate when the output is verifiable against ground truth by a deterministic function: does the JSON parse, does it contain the correct fields, does a regex match the extracted date format. Use binary scoring here. It is fast, automatable, and unambiguous.

Binary pass/fail breaks down for generative outputs — summaries, explanations, classifications with soft boundaries. A summary that captures 4 of 5 key points is not equivalent to one that captures 0 of 5, but binary scoring treats them identically. For these tasks, use a partial-credit rubric: define 3–5 specific criteria, score each independently on a 0/1 or 0/1/2 scale, and aggregate. The criteria must be defined in advance and specific enough that two different evaluators would score the same output identically at least 80% of the time.

"Good enough" is a legitimate threshold, but it must be explicit. Decide before you run: what pass rate on your golden set constitutes production-ready? For a high-stakes extraction pipeline feeding an accounting system, you might require 95% on happy path and 80% on edge cases. For an internal summarization tool with human review, 75% overall might be acceptable. The threshold is a business decision, not a technical one — make it consciously, write it down, and do not adjust it after you see results.

The Judge Layer: Automated Evaluation and Its Limits#

For large golden sets, human evaluation of every output is not practical. You need an automated judge layer.

Deterministic parsers are your first choice. If you can write a function that checks whether the output is correct — a JSON schema validator, a field-by-field comparator against ground truth, a regex — use it. It is fast, free of bias, and perfectly reproducible. Design your task outputs to be deterministically verifiable wherever possible; it is the single highest-leverage decision in the whole framework.

LLM-as-judge is appropriate when outputs are genuinely open-ended and a deterministic check would require recreating human judgment. Use a separate, stronger model as the evaluator — not the same model you are testing, and it does not have to run locally. A frontier hosted model is a perfectly good judge for a local model under test; the judge is not part of your deployment, so its cost and latency don't matter, only its independence from the thing it's grading. Give it a precise, rubric-aligned scoring prompt, not a vague "is this good?" question. The failure modes of LLM-as-judge are well-documented: positional bias (preferring the first of two options), verbosity bias (preferring longer outputs), and self-preference bias (rating outputs in its own style more highly). Mitigate these by randomizing output order, checking judge consistency against known-correct and known-wrong examples, and never using LLM-as-judge as the sole signal for a pass/fail decision on edge cases.

Human spot-check is non-negotiable in two situations: when you are calibrating your automated judge for the first time, and when your model is making decisions with real consequences. Sample at least 10% of automated-judge outputs for human review on every major eval run. If the human and automated scores diverge by more than a few percentage points, your judge is broken and you should fix it before trusting any results.

Repeatability Mechanics: Making Runs Comparable Over Time#

A single eval run is a data point. A series of comparable runs over time is a decision instrument.

Seed and temperature control: Set temperature to 0 for deterministic tasks. For generative tasks where temperature matters, fix it at your deployment value and set a random seed. Record both in your results ledger. A run at temperature 0.7 with no seed is not reproducible and should not be compared to anything. (Note that temperature 0 buys you near-determinism, not a guarantee — batching and floating-point ordering can still nudge a token — which is exactly why you log the rest of the environment.)

Hardware and runtime logging: Log the exact model file (quantization level and file hash), the inference framework and version (llama.cpp, Ollama, vLLM, LM Studio, MLX — they are not interchangeable and their defaults differ), the chat template, available VRAM or unified memory, and whether the run was fully on GPU, on CPU, or split across both. A model that fits entirely in memory produces a different latency profile — and occasionally different outputs, from floating-point ordering — than the same model partly offloaded to CPU. Local throughput is governed by memory bandwidth, not raw compute, so the quant level and the offload split are the two knobs that move your tokens-per-second; log them, or you cannot explain a regression later.

Results ledger structure: For each run, store the run ID, timestamp, model identifier, quantization level, set version, per-example scores, aggregate scores by tier (happy/edge/adversarial), and the scoring rubric version. Keep it in a flat file or a simple database you control — not in your head, not in a spreadsheet that will be overwritten. When a new model drops and you run the same set, the ledger lets you compare directly. The ledger is your institutional memory.

The Decision Protocol: What the Results Actually Mean#

The point of the framework is to produce a clear decision, not just a number. Here is how to read the output:

Ship it: Happy-path pass rate meets your threshold, edge-case pass rate meets your threshold, failure modes are diagnostic (not systemic), and you have human-spot-checked a sample of automated-judge calls with agreement. You are done.

Tune the prompt: Happy-path pass rate is high but edge-case rate is not. The model clearly understands the task but fails on specific patterns. Diagnose which edge cases fail and why — usually instruction bleed or missing schema constraints — and update the system prompt. Rerun the full set, not just the failing examples.

Quantize differently: Your results differ significantly between quantization levels, and the degradation is concentrated in edge cases or adversarial inputs. Run the next quantization level up. If your hardware cannot support it, you have a genuine constraint to take to your infrastructure decision.

This model is wrong for this task: Happy-path pass rate is below threshold, or refusal false-positives are systemic, or context-window degradation is severe enough that your real-world inputs will routinely fall into the degraded zone. Before you reach this verdict, spend ten minutes ruling out the cheap explanations — a wrong chat template, a dropped BOS token, a quant level you can trivially bump. A template bug and a bad model produce identical symptoms, and only one of them is worth abandoning the model over. If the cheap explanations are clean, stop spending time on prompt engineering: the model's training distribution does not match your task. Pull the next candidate from your shortlist and run the framework again.

The most common failure is spending two weeks on prompt engineering for a model that was wrong for the task on day one. The framework exists to make that call on day two.

---

Key Takeaways

• Benchmarks — academic or arena — rank the full-precision hosted model across a population; you run a quantized build on your own hardware against a single task. Run your own eval.
• Source your golden set from real logs, cover happy-path, edge-case, and adversarial inputs, and version it in git with stable IDs, append-only.
• Rule out the prompt-format / chat-template mismatch first — it imitates every other failure and is the most common local-specific bug.
• Quantization level is a first-class variable the leaderboard never tested; run every level you plan to deploy, because the loss is nonlinear and hits your edge cases hardest.
• Advertised context length is a ceiling, not a promise — test at the length you actually run, not the number on the model card.
• Define your scoring rubric and pass threshold before running a single prompt.
• Deterministic parser first, LLM-as-judge (which can be a hosted frontier model) second; keep a human in the loop for calibration and high-stakes calls.
• Log model file hash, framework and version, chat template, hardware and offload split, seed, and temperature so results stay comparable over time.
• The decision protocol has four exits — ship, tune the prompt, quantize differently, or replace the model — but rule out the cheap bugs before you take the last one.