How to evaluate LLM applications before release

An LLM application passes or fails on the questions its users ask, not on a public leaderboard. This article sets out an evaluation method in eight parts, with the research behind each choice. Research figures carry a numbered source; directory counts are computed from published records at build time.

Where the evaluation set comes from

Build the evaluation set from real use: support tickets, search logs, pilot transcripts and the questions domain staff answer each week. NIST's Generative AI Profile states that "measurement gaps can arise from mismatches between laboratory and real-world settings" [5]. It adds that prompt sensitivity and the variety of usage contexts widen the gap between benchmarks and real use [5].

Freeze a versioned subset as a golden set. For each case, store the input, the context the application will see, an approved answer or a list of facts the answer must contain, and an intent tag. Include cases the application must refuse and cases where the source data holds no answer. Keep a second, rotating sample of recent production traffic so the frozen set is not the only view of quality.

golden set: A fixed set of inputs with approved outputs used for regression checks.

How rubric scoring works

A rubric turns "a good answer" into criteria that a second reviewer applies the same way. Write one criterion per line: factual correctness against the source, coverage of the required facts, adherence to format instructions, refusal behaviour and tone. Score each line as pass or fail, or on a short scale with a written anchor for every point. Overlap metrics against a reference answer are a weak substitute, since BLEU and ROUGE "have been shown to have relatively low correlation with human judgments" [4].

What an LLM judge can and cannot replace

An LLM-as-judge setup applies the rubric at a volume human reviewers cannot match. Zheng et al. found that strong model judges reached over 80% agreement with human preferences on MT-Bench and Chatbot Arena, the level humans reached with each other [1]. The same paper names position bias, verbosity bias, self-enhancement bias and limited reasoning ability as limits of the method [1].

LLM-as-judge: Using one language model to grade another model's answers against a rubric.

Each bias has a control that can be written into the evaluation plan.

• Position bias. A judge favours an answer because of where it appears in the prompt [1]. Swapping two answers changed which model looked better on 66 of 80 queries in one study [2]. Control: run each pairwise comparison in both orders, count a win only when the same answer wins both times, and record a tie otherwise [1].
• Verbosity bias. A judge favours longer answers "even if they are not as clear, high-quality, or accurate as shorter alternatives" [1]. Control: add a rubric line that penalises padding and check whether judge scores track answer length.
• Self-preference. A judge can score its own outputs above outputs that humans rate as equal [3]. Zheng et al. could not confirm the effect with their data [1]. Control: use a judge from a different model family than the model under test.

How human review calibrates the judge

Human reviewers set the standard that the judge is measured against. Have two reviewers score the same sample against the rubric without seeing each other's scores, then measure how often they agree. Rewrite any rubric line on which they disagree often. Then score the same sample with the judge and compare its agreement with the human majority against the agreement between humans.

A human label is a noisy reference too: in the MT-Bench study, agreement between humans on non-tie votes was 81% [1]. Wang et al. proposed sending the cases where a judge is unstable across answer orders to a human reviewer [2]. Repeat the calibration after any change to the judge model, the judge prompt or the rubric. Keep the human-scored sample next to the golden set so the comparison can be rerun.

How to handle output that changes between runs

The same input does not always produce the same output, even with settings meant to remove randomness. A study of 829 code generation problems found that "setting the temperature to 0 does not guarantee determinism" [8]. Atil et al. ran five models configured to be deterministic ten times on eight tasks and saw accuracy vary by up to 15% between runs [9]. A single run per case is a sample, not a measurement.

Run each case several times and score the mean per case. Miller advises choosing the number of repeats so that variance within a question becomes small next to the variance between questions, and reporting a standard error with every score [10]. When two versions are compared, the paper advises inference on per-question paired differences rather than on two overall averages [10]. Power analysis tells you whether the set is large enough to detect the size of change you care about [10].

Set two kinds of threshold. A score threshold applies to the mean and its confidence interval. A consistency threshold applies per case: a refusal case, for example, passes only if the application refuses in every run.

How to measure hallucination

NIST uses the term confabulation for a hallucination: "the production of confidently stated but erroneous or false content" [5]. A pass or fail mark on the whole answer hides partial errors, because generations often mix supported and unsupported information [6]. FActScore splits an answer into atomic facts and reports the share supported by a knowledge source [6]. For an application that answers from your own documents, those documents are the knowledge source.

hallucination: Model output that is fluent but not supported by facts or sources.

Where no reference source exists, repeated sampling gives a signal. SelfCheckGPT relies on the finding that for hallucinated facts, "stochastically sampled responses are likely to diverge and contradict one another" [7]. Report the rate per answer and per atomic claim, split by intent tag, because an acceptable average can hide one intent with a high rate. Applications that retrieve documents need retrieval measured on its own, as set out in RAG evaluation: retrieval and grounding.

How to catch regressions across prompt and model versions

Treat every change to the prompt, the model, the model version, the retrieval index or the tool set as a new release candidate. Small edits move results: Sclar et al. measured differences of up to 76 accuracy points from meaning-preserving changes to prompt formatting in a few-shot setting [11]. Hosted models also change under the same name, and Chen et al. measured one model's accuracy on identifying prime numbers at 84% in its March 2023 version and 51% in its June 2023 version [12]. Pin model versions where the provider allows it, and record the version string with every run.

Run the full golden set on every candidate and compare results case by case with the current release. Review every case that moved from pass to fail, even when the mean score went up. Add each production incident to the golden set once it is fixed, so the same defect is caught if it returns. The directory lists this work as prompt and model regression testing, and monitoring AI in production covers what happens after release.

What a release gate checks

A release gate is a written list of conditions, each of which blocks deployment when it fails. Write the list and its thresholds before the evaluation runs, so that no threshold is set after the results are known.

1. 01

   Version every input

   Record the prompt, model version, judge model and prompt, rubric version, index snapshot and golden set version for the run.

2. 02

   Run repeated trials

   Run every golden set case several times and compute the mean and standard error for each gate metric.

3. 03

   Compare paired scores

   Compare the candidate with the current release case by case and list every case that moved from pass to fail.

4. 04

   Check must-pass cases

   Require every refusal, safety and privacy case to pass in every run.

5. 05

   Check the hallucination rate

   Compare the share of unsupported claims per intent with its threshold.

6. 06

   Audit the judge

   Have a human review a sample of judge scores, including every case where the verdict changed when answer order was swapped.

7. 07

   Record the sign-off

   Record who approved the release, which gates passed and which known failures were accepted.

Adversarial and security checks belong in the same gate; the LLM application testing checklist maps each check to a directory service, including adversarial testing.

Which listed vendors state these services

34 published entries state LLM output evaluation for LLM applications on their AI page. The table counts entries for each related service.

A count shows that a vendor's own AI page names the service; it is a statement by the vendor, not a finding about how the work is delivered. The LLM applications page lists the entries, and how to choose an AI testing company sets out questions to put to each vendor about method.

Sources

[1] Zheng, L. et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." arXiv (NeurIPS 2023 Datasets and Benchmarks), 2023. https://arxiv.org/abs/2306.05685

[2] Wang, P. et al. "Large Language Models are not Fair Evaluators." arXiv, 2023. https://arxiv.org/abs/2305.17926

[3] Panickssery, A., Bowman, S. R., Feng, S. "LLM Evaluators Recognize and Favor Their Own Generations." arXiv, 2024. https://arxiv.org/abs/2404.13076

[4] Liu, Y. et al. "G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment." arXiv, 2023. https://arxiv.org/abs/2303.16634

[5] National Institute of Standards and Technology. "NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile." NIST, 2024. https://doi.org/10.6028/NIST.AI.600-1

[6] Min, S. et al. "FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation." arXiv (EMNLP 2023), 2023. https://arxiv.org/abs/2305.14251

[7] Manakul, P., Liusie, A., Gales, M. J. F. "SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models." arXiv (EMNLP 2023), 2023. https://arxiv.org/abs/2303.08896

[8] Ouyang, S. et al. "An Empirical Study of the Non-determinism of ChatGPT in Code Generation." arXiv, 2023. https://arxiv.org/abs/2308.02828

[9] Atil, B. et al. "Non-Determinism of 'Deterministic' LLM Settings." arXiv, 2024. https://arxiv.org/abs/2408.04667

[10] Miller, E. "Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations." arXiv, 2024. https://arxiv.org/abs/2411.00640

[11] Sclar, M. et al. "Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design." arXiv (ICLR 2024), 2023. https://arxiv.org/abs/2310.11324

[12] Chen, L., Zaharia, M., Zou, J. "How is ChatGPT's behavior changing over time?" arXiv, 2023. https://arxiv.org/abs/2307.09009