AI Learning Lab · supplied records

Benchmarks and useful evidence

Year 8 · Lesson 5 · Content 2026.11.02.1

All supplied-record lessons · Open interactive lesson

Supplied-record alternative. Enter your responses in your school assignment. These supplied results support learning but do not show that you ran an algorithm.

Your goal

Explain what a score covers and what the school still needs to test.

To be able to judge whether a benchmark supports a decision about a stated local task.

Before you start

Would a test using only clear pictures establish performance on glare pictures?

Choices: No / Yes

Bridge feedback after trying

It can be a genuine clear-picture result with a limited scope.

Words for this lesson

Benchmark
A defined test and scoring procedure for comparison.
Test set
Examples used for evaluation rather than training.
Leaderboard
A ranking based on stated tests and scores.

The idea

A benchmark is useful when we know its task, examples and scoring procedure. A local service may face different conditions. A model can lead one test and trail another without either score being dishonest. Keep evaluation examples separate from development. Rewarding only a narrow score can also miss the real purpose, so ask what the score leaves out.

A completed example

Practice: a spelling tool scores 18/20 on typed words and 12/20 on faded scans using the same rule. The results are 90% and 60% on these supplied sets. It needs representative scan tests before a scan service relies on it.

Check your understanding

Two comparable scores need ...

  1. The same specified task and procedure
  2. Only similar names
  3. Only training accuracy
Practice feedback after trying

The same specified task and procedure. Describe data and scoring as well as the number.

Worse local results prove cheating?

  1. No
  2. Yes
  3. Only with glare
Practice feedback after trying

No. Different conditions can change performance without dishonesty.

Using test answers during development ...

  1. Weakens independence
  2. Guarantees future accuracy
  3. Fixes every bias
Practice feedback after trying

Weakens independence. A fresh separate evaluation is needed.

Task 1 · Make a prediction

Public winner must win locally

Options: Yes / No / Not sure

Write your prediction, or state that you skipped it.

Try it using supplied records

  1. Read what the public test includes. Predict whether its winner must also win the local school test.
  2. Reveal each model's local clear and glare counts.
  3. Use the fraction helper to compare the local totals and glare subgroup.
  4. Select which conditions are missing from the public evidence.

Supplied benchmark evidence

Complete data and source records

Supplied model results. Fictional teaching data.
ModelPublic correct out of 100 clear photosLocal clear correct out of 80Local glare correct out of 20
A98768
B957216
Read as stacked records
Model
A
Public correct out of 100 clear photos
98
Local clear correct out of 80
76
Local glare correct out of 20
8
Model
B
Public correct out of 100 clear photos
95
Local clear correct out of 80
72
Local glare correct out of 20
16
Separate procedure cards. Fictional teaching data.
CardDescription
LeakageAn unrelated developer practises repeatedly on published evaluation answers, then calls the same test independent. This is not evidence about A or B.
Narrow targetThe school rewards only the number of answers, so an assistant guesses instead of admitting uncertainty.
Read as stacked records
Card
Leakage
Description
An unrelated developer practises repeatedly on published evaluation answers, then calls the same test independent. This is not evidence about A or B.
Card
Narrow target
Description
The school rewards only the number of answers, so an assistant guesses instead of admitting uncertainty.

Expected results · supplied

These are supplied reference results. They are separate from your own explanation. No code needs to run to read them.

1. A local 76+8=84/100=84%, glare 8/20=40%. B local 72+16=88/100=88%, glare 16/20=80%. A leads public, B leads supplied local. The public sample is separate from the local denominator.

2. A limited B pilot with librarian review is a defensible decision; more evidence is also acceptable. Require a missing test, small-sample caveat or access condition, plus correction route in the written explanation.

3. Procedure card feedback: leakage weakens evaluation independence; keep fresh held-out cases. Narrow answer-count rewards can encourage unhelpful guessing; include correctness, useful abstention and review. Teacher More detail names this Goodhart's warning.

Help with this step

Hint 1

Use the local columns only for local accuracy.

Hint 2

Add clear and glare correct counts. Divide glare correct by 20 for the glare subgroup.

Hint 3

Practice: 30 clear plus 10 glare correct out of 50 gives 80% overall; report each group too.

Task 2 · Use the evidence

  1. Recommend a model for a limited local pilot or ask for more evidence. Use a local total and a glare result.
  2. Explain why the ranking changed without accusing either model of cheating.
  3. Read the separate leakage card and narrow-target card. State a specific concern and one improvement for each.

My explanation

Write in your school assignment. Keep these answers in your own words.

Model A

Model B

Your decision

Leakage card

Narrow target card

Pilot limits and review

Explain a fresh example

A wins the public test but B wins the school glare test. Give a local recommendation and name one reason the public score alone was insufficient.

My confidence

I need another explanation / I can explain with help / I can explain with my own example

My next step

Write what you will do next.

Optional depth

Design one change that reduces gaming and explain why it helps without promising perfect evaluation.

What this activity can show

Different scores can be valid for different scopes. A mismatch is not evidence of fraud. Goodhart's warning does not mean every benchmark is useless.

This is a student learning record. Submission is completed through the school assignment system.