- Stage
- Early
- Training correct out of 20
- 14
- Validation correct out of 20
- 13
Investigating two model failures
Year 7 · Lesson 5 · Content 2026.11.02.1
All supplied-record lessons · Open interactive lesson
Supplied-record alternative. Enter your responses in your school assignment. These supplied results support learning but do not show that you ran an algorithm.
Your goal
Use the evidence to choose the right check for an invented claim or a model that struggles with new examples.
To be able to use evidence to distinguish a hallucinated claim from a possible overfitting problem and suggest a suitable check.
Before you start
Which checks a factual number?
Choices: A reliable source record / A confident tone
Bridge feedback after trying
Compare the claim with a reliable source.
Which checks use beyond training?
Choices: Different examples / Repeating the same examples
Bridge feedback after trying
Use different examples kept out of training.
Words for this lesson
- Hallucination
- Generated information that is invented or unsupported.
- Overfitting
- Fitting training details so closely that new-example performance is weaker.
- Generalisation
- Using learned patterns successfully on new examples.
The idea
Different errors need different checks. For a generated factual claim, compare it with reliable records. For a possible overfitting problem, compare training performance with separate similar examples. A large gap suggests a problem but does not prove its cause. One wrong classification is not enough evidence to diagnose overfitting. It is responsible to ask for more evidence.
A completed example
Practice: a generated timetable says Friday but the approved timetable says Tuesday. Check and correct the factual claim. A different model gets 10/10 practice examples but 4/10 new examples right. Investigate its training, labels and new examples before naming a cause.
Check your understanding
Best stopping candidate among these stages?
- Middle, because validation is highest
- Late, because training is perfect
- Always Early
Practice feedback after trying
Middle, because validation is highest. Use validation to guide development, then a fresh final test.
One wrong classification proves ...
- Only that this prediction was wrong
- Definite overfitting
- Intentional lying
Practice feedback after trying
Only that this prediction was wrong. The cause needs more evidence.
Case B should first be checked against ...
- The actual source records
- The font
- More training epochs
Practice feedback after trying
The actual source records. Sources directly test its factual claims.
Task 1 · Make a prediction
Best training score must mean best new-example score
Options: Yes / No / Not sure
Write your prediction, or state that you skipped it.
Try it using supplied records
- Predict whether the stage with the best training score must be the best stage for new examples.
- Reveal Early, Middle and Late, one at a time. Compare the training and validation counts.
- Select a stage to investigate as a stopping point. Explain using the validation count.
- Open Case A and Case B. Choose whether to inspect source records or training and new-example results first.
Supplied development and case evidence
Complete data and source records
| Stage | Training correct out of 20 | Validation correct out of 20 |
|---|---|---|
| Early | 14 | 13 |
| Middle | 18 | 17 |
| Late | 20 | 12 |
Read as stacked records
- Stage
- Middle
- Training correct out of 20
- 18
- Validation correct out of 20
- 17
- Stage
- Late
- Training correct out of 20
- 20
- Validation correct out of 20
- 12
| Case | Evidence |
|---|---|
| A | Shape classifier: 12/12 training pictures correct, 3/12 separate comparable new pictures correct. Some training pictures have unusual background marks. Labels and code have not yet been checked. |
| B | A generated answer says 999 science books and a Desert Knowledge Medal. The library register says 248 science books. Its complete award record says no awards recorded. |
| C | A recycling classifier calls one can Paper. There is no training record, separate test set or label audit. |
| D extension | A fruit classifier worked with clear daytime pictures but now receives blurred night pictures. No comparable training and new-example scores are supplied. |
Read as stacked records
- Case
- A
- Evidence
- Shape classifier: 12/12 training pictures correct, 3/12 separate comparable new pictures correct. Some training pictures have unusual background marks. Labels and code have not yet been checked.
- Case
- B
- Evidence
- A generated answer says 999 science books and a Desert Knowledge Medal. The library register says 248 science books. Its complete award record says no awards recorded.
- Case
- C
- Evidence
- A recycling classifier calls one can Paper. There is no training record, separate test set or label audit.
- Case
- D extension
- Evidence
- A fruit classifier worked with clear daytime pictures but now receives blurred night pictures. No comparable training and new-example scores are supplied.
Expected results · supplied
These are supplied reference results. They are separate from your own explanation. No code needs to run to read them.
1. Development record is supplied evidence, not a live run. Middle has the highest validation count, 17/20. Late gives the largest training-validation gap, 8 items. Validation guides development; a further untouched test set would be needed for final evaluation.
2. A: possible overfitting; check labels, background reliance, code and representative fresh data. B: contradicted or fabricated claims; check and correct against records. C: insufficient evidence; obtain labels and comparable results. D: changed input conditions are plausible, not proven overfitting.
3. Use tabs Sources and Model results, never make diagnosis depend on interface colour. Require evidence selection before the diagnosis feedback appears. Multiple reasonable next checks are accepted under the rubric.
Help with this step
Hint 1
Write what you actually know before naming the failure.
Hint 2
A claim about a fact needs a source. A training gap needs comparable results on separate examples.
Hint 3
Practice: a single wrong weather label tells us that prediction was wrong. It does not reveal whether the model memorised training details.
Task 2 · Use the evidence
- For Cases A, B and C, state the strongest supported judgement.
- Attach one evidence detail and one suitable next check to each.
- Write two sentences explaining why A and B need different checks. Use suggests or not enough evidence when appropriate.
My explanation
Write in your school assignment. Keep these answers in your own words.
Stopping candidate
- Stage: Early / Middle / Late
- Reason
Case A
- Judgement
- Evidence
- Next check
Case B
- Judgement
- Evidence
- Next check
Case C
- Judgement
- Evidence
- Next check
Your explanation
- Why A and B need different checks
- Case D optional explanation
Explain a fresh example
A model is perfect on training items but poor on comparable new items. Another answer invents a school award. Give one suitable check for each and state a limit on your diagnosis.
My confidence
I need another explanation / I can explain with help / I can explain with my own example
My next step
Write what you will do next.
Optional depth
Analyse D. Explain why changed input conditions offer another explanation and how you would compare images fairly.
What this activity can show
A gap is not conclusive proof of overfitting. Hallucination and overfitting are not mutually exclusive properties of an entire system. No diagnosis implies machine intent.
This is a student learning record. Submission is completed through the school assignment system.