LLM eval harness
Grade LLM outputs against a dataset — exact match, regex, contains, JSON Schema, numeric tolerance, BLEU (sacreBLEU), ROUGE-L, embedding similarity or an LLM judge — with bootstrap confidence intervals, per-tag breakdowns and run comparisons by paired bootstrap and McNemar's test; paste outputs or call Claude, OpenAI or a local model with your own key.
Added
How to use it
A small, honest eval loop in the browser: a dataset of prompts and expected answers, one or more runs of outputs, graders that say pass or fail and why, and statistics that tell you whether the difference between two runs is real or noise. Paste outputs from anywhere, or let the page call a model with your own API key.
How to use it
Dataset. The built-in sample has 26 small tasks in seven tags (math, classify, extract, format,
summarize, translate, knowledge), each naming its own grader. For your own, paste or open a CSV
with a header (id,input,expected,tags,grader) or JSON Lines
({"id":"q1","input":"…","expected":"…","tags":["math"],"grader":"numeric"}). Several acceptable
answers: a JSON array, or ` || ` between them in a CSV cell. OpenAI evals' {"input":[messages],
"ideal":…} lines work too. Extra columns can be used in the prompt template as {{column}}.
Graders. A row’s grader column wins; otherwise the default on the Graders tab applies.
Arguments follow a colon.
| Grader | Passes when | Argument |
|---|---|---|
|
the normalised output equals a reference: case, whitespace, a trailing full stop; optionally punctuation and articles (SQuAD’s normalisation) |
|
|
the trimmed output matches |
the pattern, or |
|
the output contains the expected text — all or any of its |
the text |
|
the output is JSON (optionally found in a |
a schema |
|
as |
|
|
the last number in the output is within abs + rel·|expected| (1,157.63, $36, 3/4 and 45% are read as numbers) |
|
|
sentence BLEU ≥ threshold (default 30) |
threshold |
|
ROUGE-L F ≥ threshold (default 0.5) |
threshold |
|
the cosine similarity of all-MiniLM-L6-v2 embeddings ≥ threshold (default 0.8) |
threshold |
|
an LLM judge scores 1–5 against the reference; (score − 1)/4 ≥ threshold (default 0.7) |
threshold |
Try a grader on the Graders tab shows the work: BLEU’s 13a tokens, n-gram matches, precisions and brevity penalty; ROUGE-L’s longest common subsequence highlighted in the output.
Run. Either paste outputs — JSON Lines with id and output, a CSV id,output, or one per line
in dataset order — or Call a model: Anthropic (Claude), OpenAI, or any OpenAI-compatible server
(Ollama, vLLM, LM Studio) at a base URL, with a system prompt, a template, a token limit and the
number of parallel requests. Rate limits (429), Anthropic’s overloaded (529) and server errors are
retried with exponential backoff (honouring retry-after); Cancel stops and keeps what finished.
only rows that failed re-runs just those. A finished run shows its tokens and, for models in
the agent trace viewer's price table, the approximate list-price cost.
Results. Pass rate with a 95% bootstrap interval (and Wilson’s interval as a check), the mean score, corpus BLEU over the BLEU rows, a per-tag table and chart, and every row with its grade and the reason. Rows needing embeddings or the judge wait for Grade with embeddings / Run the judge. Export a run as JSON or CSV; export and import your dataset with all runs. Full screen (F) gives the per-tag chart — or, on the Compare tab, the A-against-B chart — the whole screen.
Compare. Two runs on the same rows: the pass-rate difference with a paired bootstrap interval, McNemar’s exact test on the rows where they disagree, the mean-score difference, per-tag changes, and the rows each run fixed or broke. The two sample runs pass 20 of 26 each — on different rows — so the interval includes 0 and McNemar’s p is 1: the same score is not the same model.
Your API key
The key is held in the page’s memory and sent only to the provider you chose, straight from your
browser. It is never put in a share link (links carry settings only, and the key is kept in a
separate store they do not read). Tick remember the key on this device to keep it in this
browser’s local storage — anyone with access to the browser profile, or a script on this site,
could then read it. Use a key with a spending limit. Anthropic’s API accepts calls from web pages
when they send the anthropic-dangerous-direct-browser-access: true header, which this page does;
OpenAI and local servers must allow this site’s origin (for Ollama, set OLLAMA_ORIGINS). Current
Claude models take no sampling parameters, so temperature is offered only for the others; effort
is passed as output_config.effort.
Limits
-
BLEU and ROUGE measure word overlap with the reference, not correctness: in the sample, model-b’s “The train is delayed by twenty minutes.” is a good translation and scores BLEU 27.05. Choose thresholds per task and read the failures.
-
ROUGE-L uses rouge-score’s defaults (no stemming, ASCII letters and digits only), so text in other scripts gets no tokens; BLEU uses sacreBLEU’s 13a tokeniser (no Chinese/Japanese/Korean tokenisers).
-
The JSON Schema support is a subset (no
$ref,format,dependenciesorpatternProperties). -
Embeddings (all-MiniLM-L6-v2, English, 256 word pieces) download ~23 MB once; the judge has the usual biases (length, style, its own model family).
-
Bootstrap intervals on a few dozen rows are wide, and percentile intervals undercover at very small n; treat them as a guide. Rows are assumed independent.
-
No streaming, tools, images or multi-turn conversations: one prompt in, one text out.
-
Runs live in this browser’s local storage (a few MB); export anything you want to keep.
Checked against
sacreBLEU 2.6.0 (sentence_bleu, corpus_bleu, Tokenizer13a: scores, precisions, brevity
penalty and lengths for eight sentence cases including several references, a too-short output and
no matches, and a five-sentence corpus — corpus BLEU 51.26, which is not the mean of the sentence
scores), Google’s rouge-score (RougeScorer(["rougeL"]): P, R and F for three pairs, e.g. F 0.6667
for the sample’s first summary), statsmodels 0.15 (mcnemar exact and continuity-corrected:
b = 1, c = 6 gives p = 0.125, χ² = 2.286), NumPy’s linear percentile, and the Wilson interval by
formula (7/10 → 0.397–0.892).
Sources: K. Papineni et al., “BLEU: a method for automatic evaluation of machine translation”, ACL 2002; M. Post, “A call for clarity in reporting BLEU scores”, WMT 2018 (sacreBLEU); C.-Y. Lin, “ROUGE: a package for automatic evaluation of summaries”, 2004; Q. McNemar, “Note on the sampling error of the difference between correlated proportions”, Psychometrika 1947; B. Efron & R. Tibshirani, An Introduction to the Bootstrap (1993); P. Rajpurkar et al., SQuAD (2016) for the answer normalisation; Anthropic’s Messages API documentation for the request shape and the browser-access header.
Related: Model evaluation workbench (classifier metrics), Ranking metrics, RAG chunking playground, Agent trace viewer, Tokenizer explorer, Embedding explorer and Statistics.
Linking to an example
/tools/eval-harness/?<parameters>#<tab>, where the tab is dataset, graders, run, results
or compare. Links carry settings — graders, thresholds, the provider and model, the prompt
template — never your dataset, runs or API key.
| Parameter | Meaning | Example |
|---|---|---|
|
a preset (below) |
|
|
the default grader |
|
|
provider ( |
|
|
the Try a grader box: grader, output and expected |
|
example= |
Tab | Shows |
|---|---|---|
|
results |
The sample dataset graded for model-a (open) |
|
compare |
model-a against model-b: same pass rate, different rows (open) |
|
graders |
BLEU worked through for one translation (open) |
|
graders |
ROUGE-L for a one-sentence summary (open) |
|
graders |
A JSON Schema check failing on a string age (open) |
|
graders |
Numeric tolerance: the last number in the text (open) |
For example,
?tg=rougeL&out=…&exp=…#graders
works through a ROUGE-L score by hand.
Feedback is a gift. Something to improve, a feature to add, or a bug? Tell me.
Runs entirely in your browser — network requests go only to the services named on this page.