Lesson 3.4:What a Benchmark Score Tells You (How Leaderboards Are Built)

Every new AI model arrives with a ranking. Its makers publish a table of test scores, set against the previous model and its rivals. Legal AI vendors do the same: their tool tops a legal leaderboard, or reaches a high accuracy on contract review. Behind each of these numbers is a benchmark: a fixed set of examples, the output a legal expert expects for each one, and a rule for grading.

The same model can come first on one legal leaderboard and far down another. Neither ranking is wrong. Each benchmark asks its own question, with its own examples and grading. A leaderboard ranks models on one set of examples, graded one way.

The examples behind a score

A benchmark starts as a dataset. Each example has two parts. The input is what the model receives. It can be a clause to classify or an email to sort, not only a question. The label is the output a legal expert expects for that input. It can be exclusivity: yes, or the redline the team would send.

Some benchmarks test one task: Legal Research Bench holds only US legal research questions, written by lawyers. Others gather many: LegalBench holds 162 small tasks, each with its own dataset. And some scores average several benchmarks into one number.

The people who write the labels are called annotators. For a legal benchmark, they are legal professionals, and their work is the slowest and most expensive part. Without labels, there is nothing to compare the model's output with, and so no score.

Labels are judgments too. In one experiment, two skilled attorneys each wrote the ideal redline for the same 20 NDAs. Of 500 edits, they agreed on 45% Crosby · How we do evals, Oct 2026 (opens in a new tab). This sets a ceiling. A model that matches one attorney on 45% of edits already agrees with that attorney as closely as the second one did. Good benchmarks measure how far their annotators agree, and report it. An expert label is also not always the best possible answer. In a 2026 study, a legal publisher asked its attorneys to compare its AI's work on 50 long tasks with answers written by experts. On nearly 40% of the tasks, they preferred the AI's answer Thomson Reuters · Evaluated like legal work, Aug 2026 (opens in a new tab).

The examples must also look like the real task, down to the type of document and the law that applies. And there must be enough of them. A few examples cover the common cases. A larger set also covers the exceptions and odd cases of real work. That is why more data usually gives a more reliable test.

A model learns from a very large amount of text, its training set. A benchmark is a test set: examples kept out of training. This matters because a model that saw the examples in training can recall their labels. Its score would then measure memory (lesson 3.2).

A test is only as good as its examples and the experts who labelled them. Once these are in place, each model answers the test inputs, and each output is compared with its label. How that comparison is made depends on the kind of task.

How outputs are graded

Grading turns hundreds of comparisons into one number. If a model gets 45 of 50 clauses right, its score is 90%. The rule behind that number, here the share of right answers, is called the metric. Which metric fits depends first on one question: does the task have a single right answer?

Some tasks have one right answer. Does this clause give exclusivity? Is this document a lease or a licence? Grading only takes comparing the output with the label. This grading is deterministic: the same output gets the same grade every time.

The share of right answers is only one possible metric. Take a model that checks 10 contracts for exclusivity clauses. It can miss a clause that is there, or flag one that isn't. The share counts a missed clause and a false alarm as the same mistake, but they don't cost the same. In due diligence, one missed clause can matter far more than a few false alarms, which only cost review time. A metric can count only the real clauses the model found, or only the flags that were right. The same outputs can then earn very different scores. The right metric should count mistakes the way your work does.

Most legal work has many ways to give the “right” answer. Two good advice notes can be worded and ordered differently, so there is no single ground truth to compare every output with. In this case, experts write a rubric: a checklist of the points a good output must contain, each one met or not met.

Then each point needs checking. Human experts are reliable but slow, so many benchmarks now use LLM-as-a-judge: another model receives the input, the rubric and the output, and decides for each point whether the output meets it. It is fast and cheap. It also needs checking, because it does not always give the same verdict. Asked again about the same pair of outputs, judges changed their verdict 13.6% of the time on average arXiv · Yagubyan, 2026 (opens in a new tab). Checked against experts, a judge can still do well. On Legal Research Bench, three attorneys agreed with each other on 83% of points, and three judge models agreed with the attorneys' majority on 86% to 87% Vals AI · Legal Research Bench, Sep 2026 (opens in a new tab).

Judges also have leanings. In the same study, one judge model picked the first of two outputs 72% of the time. In another, the strongest pull across judges was toward answers formatted with headings and bullet points arXiv · Soumik, Apr 2026 (opens in a new tab). A judge can also compare two outputs in place of grading one. “Which of these two redlines would you rather send to the client?” is easier to answer well than a mark out of ten Crosby · How we do evals, Oct 2026 (opens in a new tab).

The rubric raises one more choice: what counts as a pass. With partial credit, an answer earns the share of points it made. With the stricter rule, it counts only when every point is there. On Legal Research Bench, the same answers scored 68% to 86% with partial credit. When every point had to be there, they scored 13% to 43% Vals AI · Legal Research Bench, Sep 2026 (opens in a new tab).

So a score depends on more than the outputs. It also depends on the metric, on how each output is graded, and on what counts as a pass. Change one of these, and the same outputs earn a different number. Two scores can only be compared if they were graded the same way.

From scores to a leaderboard

Run every model on the same test set, grade them the same way and sort the scores: that is a leaderboard. Most legal leaderboards work like this.

Before you trust a leaderboard, ask three questions.

How hard is the test? A score only means something next to what other methods reach on the same test. The first reference point is the baseline: what a simple method scores. In a 2026 study of benchmarks built from European court registers, a keyword search with no AI at all scored 96% on one Dutch task arXiv · Multi-Legal-Bench, 2026 (opens in a new tab). A model scoring 97% there adds almost nothing.

The reverse is true too. Some tests are built to be hard. On Harvey's LAB, a task counts as done only when every criterion is met. As of 7 October 2026, the best model completes about a quarter of the tasks, while meeting around 90% of the individual criteria Vals AI · LAB leaderboard, Oct 2026 (opens in a new tab). On a test like this, 25% is the top of the field.

Does the test look like your work? The same model can rank very differently on two tests (the model that came first on Legal Research Bench is 51st on LegalBench). Neither ranking is wrong: LegalBench asks short, exam-style questions, while Legal Research Bench asks for long searches written up as advice. A ranking only holds for the work its test resembles: before you rely on one, check that its tasks and its law match what your team does.

Is one model really better than the next? Look at the margin. A model rarely gives the same output twice, so its score changes a little each time the test is run. A single run can be off by several points arXiv · Bjarnason et al., Feb 2026 (opens in a new tab). Careful leaderboards account for this: they run each example several times, or show a margin of error next to each score, as Legal Research Bench does Vals AI · Legal Research Bench, Sep 2026 (opens in a new tab). When two models' margins overlap, the leaderboard cannot tell them apart.

There is also a second type of leaderboard, with no labels at all. It relies on voting: lawyers see two outputs for the same task and pick the one they prefer. Thousands of these votes become a ranking, as in Harvey's BLB: Arena. It shows which outputs lawyers like best. That preference is a useful stand-in for the right answer, but a preferred output is not necessarily correct.

Where a score can mislead

A public test can lose value over time. Once published, its examples and labels can end up in the text that new models learn from. Researchers call this contamination: a model that saw an example and its label in training can give the label back from memory. In February 2026, OpenAI stopped reporting a popular coding benchmark because every frontier model it tested could reproduce the reference answers OpenAI · SWE-bench Verified, Feb 2026 (opens in a new tab). Contamination can happen by accident. When a lab tunes its model for the public tests that make headlines, the field uses a more familiar word: benchmaxxing. Accidental or not, the score rises, but what the model can actually do stays the same.

As models improve, scores bunch up near the top, and the test can no longer tell models apart. On LegalBench, as of 1 October 2026, the best 50 of 149 models sit within 5 points of each other Vals AI · LegalBench, Oct 2026 (opens in a new tab). Researchers call such a test saturated. This happens to many tests. A 2026 study of 60 benchmarks found that nearly half were saturated, more often the older ones arXiv · Akhtar et al., Feb 2026 (opens in a new tab). Tests built carefully by experts held up better.

A test can also lean one way from the start, depending on who built it. A benchmark built by a vendor can favour the vendor's tool. The vendor has every interest in showing that its tool performs best. That does not make the benchmark wrong, but it leaves room for bias. It chooses which tasks go into the test, and its own team writes the labels, so the test can end up shaped around its tool. The most useful scores come from tests built by someone who doesn't sell a tool for the tasks tested.

One way to avoid contamination is to keep the test private: a model can't learn from examples that were never published. The trade-off is trust. No one outside can check the examples or the labels, so the result is only as credible as the organisation behind it. A private test is most useful when it is run by an independent party with no tool to sell for the tasks it covers. One example of such a private benchmark is Lawve-bench-1. Its tasks are synthetic examples derived from real client matters, which gives a neutral score on the work legal teams actually do.

What this means for your work

In practice, you will meet these numbers in two places: a lab announcing a new model and a vendor presenting its tool. Either way, the score tells you how a model did on one test, graded one way. Before it guides a decision, check that the test looks like your work, who built it, and how it was graded.

Look at the cost too. Many leaderboards show the price per task, and models with the same score can differ widely: on the Legal Research Bench leaderboard, three models tied for first cost from $0.59 to $23.06 per question Vals AI · Legal Research leaderboard, Oct 2026 (opens in a new tab).

Even then, a public score only goes so far. No leaderboard holds your contracts or your team's way of working. The most reliable test is one built from your own matters, and lesson 3.5 shows how to build it from work your team has already done. The test that matters most is the one built from your own work.