Evaluating LLM Output When You Have No Ground Truth

Most teams ship AI features on vibes because the correct answer was never written down. You can do better without a labelled dataset.

Hafeez Baig•Mar 10, 2026•10 min read

Introduction

Most teams shipping an LLM feature cannot answer a simple question: how do you know it works?

Not "did it return a 200". How do you know the summary is accurate, that the answer is grounded, that last week's prompt change did not quietly make things worse for a category of inputs nobody thought to check.

The honest answer, in most cases, is that nobody knows. The feature ships because somebody tried it a few times and it seemed fine.

This is not laziness. The testing tools we all learned assume there is a correct answer written down somewhere, and for most language model features there is not. There is no labelled dataset, and building one looks expensive, so it never gets built.

This post is about what you do instead. The short version: a cheap, imperfect evaluation that runs on every change beats a perfect evaluation that never gets built.

1. Why Traditional Testing Breaks

Two properties of language model output break the testing model we are used to.

The first is non-determinism. Send the same input twice and you can get different text. Not wrong text, just different. assertEquals compares two strings for exact equality, and exact equality is not a property your output has.

The second is deeper. There is often no single right answer. Ask a model to summarise a page of release notes and there are dozens of good summaries. Some lead with the breaking change, some with the new feature. Both are correct. A test that pins one exact string as "the answer" is not testing correctness, it is testing that nothing changed.

Hence the trap: traditional assertions do not apply, so people conclude evaluation is impossible, so they do nothing. The mistake is treating "I cannot check exactly" as "I cannot check at all". You are not trying to prove the output is optimal. You are trying to catch it being wrong, and wrong is far easier to detect than optimal.

2. Why "It Looked Fine When I Tried It" Is Not Evaluation

Manual spot checking feels like testing. It is not. When you try a feature by hand, you test the inputs you thought of, which are the ones you had in mind when you wrote the prompt, which means they are exactly the ones it already handles. You are grading your own homework with the answer sheet open.

The failure mode is silent. You change a prompt to fix a formatting issue, try your three usual inputs, and ship. What you did not see is that the change also made the model more likely to answer questions where the retrieved documents contained nothing relevant, so now it invents an answer instead of saying it does not know. Nothing errors. Nothing alerts. Quality drops for a slice of traffic you never look at.

Model changes are worse, because you did not touch anything: the provider ships a new version, behaviour shifts, and your feature degrades without a line of code changing. Manual testing cannot catch that, because manual testing is not running.

Spot checking is not inaccurate so much as not repeatable and not automatic. It happens when someone remembers, on inputs someone chooses, against a standard nobody wrote down.

3. Start With a Golden Set

A golden set is a small, hand-written collection of representative inputs paired with a description of what a good answer must contain. Not the exact answer. What it must contain. That distinction is the whole trick: you are not writing the ideal summary, you are writing down the facts any acceptable summary would have to include.

Input: A support ticket where a customer reports a failed payment for order 4417 and asks for a refund.

A good answer must: mention order 4417, identify the issue as a payment failure, and state the refund request. It must not invent a refund amount.

This takes minutes per case, not hours, and you do not need thousands. Even 20 to 50 cases beats zero by an enormous margin, because zero catches nothing.

Choose cases that span your real input space rather than your happy path:

Typical cases. The inputs you already know work.
Edge cases. Empty input, very long input, input in a second language.
Adversarial cases. Input that invites the model to make something up.
Known past failures. Every bug you fixed becomes a permanent case. This is how the set earns its value over time.

I would start here before anything else on this list. It turns "we hope it works" into "here is what we check".

4. Property Based Checks That Need No Ground Truth

Some things about an answer can be verified without knowing what the answer should be. Property based checks verify that the output satisfies a rule, not that it matches a target. They are cheap, deterministic, and they run on any output at all, including live production traffic.

Check	Question it answers
Schema validity	Is it parseable JSON matching the expected shape?
Language	Is the reply in the language the user wrote in?
Length bounds	Is it within a sensible word or token range?
Citation existence	Does every cited source ID actually exist?
Forbidden claims	Does it avoid legal, medical, or pricing guarantees?

Here is a schema check. Nothing clever, and that is the point:

python

import json
from jsonschema import validate, ValidationError

ANSWER_SCHEMA = {
    "type": "object",
    "properties": {
        "answer":     {"type": "string", "minLength": 1},
        "confidence": {"type": "string", "enum": ["low", "medium", "high"]},
        "sources":    {"type": "array", "items": {"type": "string"}},
    },
    "required": ["answer", "confidence", "sources"],
    "additionalProperties": False,
}

def check_schema(raw_output, known_source_ids):
    failures = []
    try:
        parsed = json.loads(raw_output)
    except json.JSONDecodeError as e:
        return [f"not valid JSON: {e}"]

    try:
        validate(instance=parsed, schema=ANSWER_SCHEMA)
    except ValidationError as e:
        failures.append(f"schema mismatch: {e.message}")

    for source in parsed.get("sources", []):
        if source not in known_source_ids:
            failures.append(f"cited a source that does not exist: {source}")

    return failures

That last loop is worth pausing on. A model citing a document that does not exist is a hallucination you can catch with a set lookup. No judge, no human, no labels.

5. Assertion Based Grading

Once you have a golden set, you need a way to score against it. Assertion based grading checks for required facts rather than exact text. For each case you list what must be present and what must be absent, and check only those things. Everything else about the phrasing is free.

python

import re

CASES = [
    {
        "id": "refund-request-basic",
        "input": "Order 4417 payment failed. I'd like a refund please.",
        "must_include": ["4417", "refund"],
        "must_not_match": [r"\$\d+"],   # do not invent an amount
        "max_words": 60,
    },
]

def grade(case, output):
    failures = []

    for token in case.get("must_include", []):
        if token.lower() not in output.lower():
            failures.append(f"missing required fact: {token}")

    for pattern in case.get("must_not_match", []):
        if re.search(pattern, output):
            failures.append(f"forbidden pattern: {pattern}")

    if len(output.split()) > case.get("max_words", 10_000):
        failures.append("exceeded length bound")

    return failures

def run_golden_set(cases, generate):
    return [
        {"id": c["id"], "failures": grade(c, generate(c["input"]))}
        for c in cases
    ]

This is not sophisticated. It runs in seconds, costs almost nothing beyond the model calls, and it catches real regressions. That combination is what makes it worth building.

6. LLM as Judge, and Its Real Weaknesses

Some qualities resist assertions: tone, coherence, whether an explanation is actually helpful. For those you can use another model as a grader. This is LLM as judge: you give a second model the input, the output, and a rubric, and ask it to score.

It works better than people expect, and it has specific weaknesses you must design around.

Position bias. When comparing two answers, judges tend to favour whichever came first, regardless of quality.
Verbosity bias. Longer answers score higher, even when the extra length adds nothing.
Self preference. A judge tends to prefer text that looks like its own output.
Circularity. You are using a model to check a model. If both share a blind spot, the judge confidently approves the failure.

The mitigations are practical:

Write a rubric. Do not ask "is this good". Ask specific questions: does it answer the question asked, is every claim supported, is the tone right for a customer.
Use pairwise comparison. "Which of these two is better" is an easier and more reliable question than "rate this out of ten".
Randomise order. Cancels position bias, in one line of code.
Calibrate against humans. Have a person grade a sample the judge already graded. If they disagree often, fix the rubric before trusting the judge.

Treat the judge as a noisy signal: useful in aggregate, unreliable on any single case.

7. Grounding Checks for RAG

If you are building retrieval augmented generation, meaning you fetch documents and pass them to the model as context, you have an advantage: you know what the answer was supposed to be based on. That makes a specific check possible. Is every claim in the answer supported by the retrieved context?

Break the answer into claims and ask, for each, whether the retrieved documents support it. Any unsupported claim is either a hallucination or the model using its training data instead of your documents. Both are usually bugs. You can implement this with a judge, but the question is narrow and factual: "does this passage support this statement, yes or no". That is far more reliable than "is this answer good".

Two related checks are worth adding. Retrieval quality: did you fetch documents containing the answer at all? If not, you have a retrieval problem, not a prompt problem. And abstention: when the context contains nothing relevant, does the model say it does not know, or invent something? Put a golden case in specifically for this. It is one of the most damaging failures in production RAG.

8. Human Review as Sampling

Human review is your highest quality signal and your most expensive. Teams treat it as all or nothing: review every output, which is impossible, or review nothing, which is what actually happens.

Treat it as sampling instead. Review a fixed number of outputs per week; twenty is plenty for most features. Weight the sample toward what matters: cases the judge scored low, cases where the user retried, cases from a route you just changed. Use the same rubric your judge uses, so the two measure the same thing.

Every disagreement is valuable twice, telling you something about the feature and something about your evaluation. Feed the interesting ones back into the golden set, so it grows from real failures rather than imagination.

9. Regression Testing and Prompts as Code

Prompts are code. They change behaviour, they have bugs, and they need the same discipline as anything else in your repository: version control, review, and versioning so you know which prompt produced a given output. Above all, the golden set runs on every prompt change.

A prompt change with no evaluation attached is a deployment with no tests. The suite does not need to be exhaustive, it needs to be automatic, because a check that requires someone to remember is a check that does not exist. Run it when you change models, retrieval, or chunk sizes too. Anything on the path from input to output should trip the same wire.

10. Production Signals Are Evaluation

Your users evaluate your feature constantly. Most teams just do not record it.

Thumbs up and down. Low volume, high signal, cheap to add.
Edits. If a user rewrites the generated text before sending it, the diff tells you exactly what was wrong.
Retries. Asking again, rephrased, means the first answer failed.
Abandonment. The user reads the answer and leaves without acting. Weak individually, informative in aggregate.
Escalation. In a support context, handing off to a human is a direct verdict.

These are lagging indicators, so they do not replace pre-deployment evaluation. But they cover the inputs you never imagined, which is exactly what your golden set is missing. Mine the thumbs-down cases, write them up as golden cases, and the suite sharpens every week from real evidence.

Conclusion

None of these techniques are individually rigorous. A golden set of 30 cases does not cover your input space. A judge is biased. Sampled review misses most of what happens. Production signals arrive too late. Together, they are far better than what most teams have, which is nothing.

If you take one thing from this: build the small thing and wire it into your pipeline today. Thirty golden cases with assertion checks, running on every prompt change, will catch more real regressions than an elaborate framework that stays in a design document.

The perfect evaluation you never build catches zero bugs. The imperfect one that runs on every change catches them for years.

Start with twenty cases. Add one every time something breaks.