Testing the benchmark claim

Year 9 · Lesson 3 · Content 2026.11.02.1

All supplied-record lessons · Open interactive lesson

Supplied-record alternative. Enter your responses in your school assignment. These supplied results support learning but do not show that you ran an algorithm.

Your goal

Audit what a test claim covers and plan the missing local evidence.

To be able to evaluate a benchmark claim using its procedure, population and local error evidence.

Before you start

A test includes typed requests only. Does it establish results for noisy speech?

Choices: No / Yes

Bridge feedback after trying

It gives evidence for its stated sample; another sample is needed.

Words for this lesson

Benchmark: A specified comparison test and procedure.
Distribution gap: A difference between test examples and use conditions.
Leakage: Evaluation information entering development and weakening independence.

The idea

A benchmark claim needs a task, population, version and scoring procedure. Local use may differ. That difference can limit the claim without proving dishonesty. Check whether evaluation answers entered development. Report small-sample results with their counts. A recommendation should say what is missing, who should collect it and what happens while uncertainty remains.

A completed example

Practice: a fixed procedure gets 19/20 clean typed requests and 13/20 noisy transcripts correct. These are 95% and 65% on the supplied samples. The difference suggests testing local conditions; it does not prove cheating or a precise cause.

Check your understanding

A genuine score can fail to predict local results because ...

The conditions differ
All benchmarks are fake
The name is shorter

Practice feedback after trying

The conditions differ. Report the scope rather than inventing dishonesty.

3/4 should be reported as ...

75% on a small four-case sample
Certain future reliability
No evidence at all

Practice feedback after trying

75% on a small four-case sample. Counts show the limited evidence.

Who used the test answers in development?

Cedar only in the supplied story
Atlas automatically
Beacon automatically

Practice feedback after trying

Cedar only in the supplied story. Keep the allegation attached to its stated evidence.

Task 1 · Make a prediction

Public winner must lead L20

Options: Yes / No / Not sure

Write your prediction, or state that you skipped it.

Try it using supplied records

Open the public-test passport: task, population, procedure and version.
Predict whether its winner must also lead the local test.
Reveal L20 results, then the noisy subgroup. Use numerator and denominator support.
Select the questions the benchmark has not answered, including local language, access and connection conditions.

Supplied benchmark evidence

Complete data and source records

Test passports. Fictional teaching data.
Test	Population	Procedure
B100	100 clean typed English requests on desktop	Same request IDs within each model comparison; route category compared with staff labels; model version recorded.
L20	16 typed and 4 noisy-room transcribed requests	Staff-checked labels; held out from development; same examples for both models.

Read as stacked records

Test: B100
Population: 100 clean typed English requests on desktop
Procedure: Same request IDs within each model comparison; route category compared with staff labels; model version recorded.

Test: L20
Population: 16 typed and 4 noisy-room transcribed requests
Procedure: Staff-checked labels; held out from development; same examples for both models.

Supplied results. Fictional teaching data.
Model	B100 correct out of 100	L20 typed correct out of 16	L20 noisy correct out of 4
Atlas	98	15	1
Beacon	96	14	3

Read as stacked records

Model: Atlas
B100 correct out of 100: 98
L20 typed correct out of 16: 15
L20 noisy correct out of 4: 1

Model: Beacon
B100 correct out of 100: 96
L20 typed correct out of 16: 14
L20 noisy correct out of 4: 3

Separate concern cards. Fictional teaching data.
Card	Evidence
1	The public report gives no noisy-room result.
2	An unrelated model Cedar trained on all B100 answers and then called B100 independent.
3	A team rewarded only for reply count guesses on uncertain requests.

Read as stacked records

Card: 1
Evidence: The public report gives no noisy-room result.

Card: 2
Evidence: An unrelated model Cedar trained on all B100 answers and then called B100 independent.

Card: 3
Evidence: A team rewarded only for reply count guesses on uncertain requests.

Expected results · supplied

These are supplied reference results. They are separate from your own explanation. No code needs to run to read them.

1. Atlas local=16/20=80%, noisy=1/4=25%. Beacon local=17/20=85%, noisy=3/4=75%. Do not mix B100 examples into L20. Keep raw counts alongside percentages, especially 1/4 and 3/4.

2. Concern keys: 1 scope gap, 2 leakage, 3 narrow target. Cedar story is not evidence against Atlas or Beacon. Missing tests include relevant Arabic requests, speech accessibility and slow connections; do not infer those groups inherently perform worse.

3. Suggested next step is a reviewed Beacon pilot or further comparison. Require an owner such as testing lead, representative fresh cases with checked labels, predeclared scoring and a final held-out evaluation. Do not auto-grade the recommendation by model name alone.

Help with this step

Hint 1

Keep public and local tests separate.

Hint 2

For noisy accuracy use four as the denominator and report the count.

Hint 3

Practice: 12 typed plus 3 noisy correct in 20 local cases gives 75% overall. The noisy group could also be 3/4=75%, but the populations differ.

Task 2 · Use the evidence

Write an audit with one public result, one local result and the four-case subgroup limit.
Classify the separate cards as scope gap, leakage or narrow target concern and give the supporting detail.
Propose a fresh local test, name its owner and preserve independence from development. Include a correction route.

My explanation

Write in your school assignment. Keep these answers in your own words.

Atlas

Local total correct (requests)
Local accuracy (%)
Noisy correct (requests)
Noisy total (requests)
Noisy accuracy (%)

Beacon

Local total correct (requests)
Local accuracy (%)
Noisy correct (requests)
Noisy total (requests)
Noisy accuracy (%)

Card 1

Concern type: Scope gap / Leakage / Narrow target concern
Evidence

Card 2

Concern type: Scope gap / Leakage / Narrow target concern
Evidence

Card 3

Concern type: Scope gap / Leakage / Narrow target concern
Evidence

Your audit

My qualified recommendation
Missing condition
Test owner
Independence protection
Correction route

Explain a fresh example

Give a qualified local recommendation using Atlas and Beacon results. Name a missing condition, an evidence owner and one way to protect test independence.

My confidence

I need another explanation / I can explain with help / I can explain with my own example

My next step

Write what you will do next.

Optional depth

Explain why repeatedly adjusting to the final held-out set eventually weakens its value, even if it was initially separate.

What this activity can show

The 20-case local test and four-case subgroup cannot establish future reliability. Missing coverage is an evidence gap, not a claim about a language or user group.

This is a student learning record. Submission is completed through the school assignment system.