Drift gaps and monitoring

Year 9 · Lesson 4 · Content 2026.11.02.1

All supplied-record lessons · Open interactive lesson

Supplied-record alternative. Enter your responses in your school assignment. These supplied results support learning but do not show that you ran an algorithm.

Your goal

Use weekly evidence to decide when someone must investigate and what they should do.

To be able to use changing service evidence to propose a monitoring trigger, an owner and a proportionate response.

Before you start

Overall performance is 92%, but one device group is 70%. A rule says investigate any group below 80%. Should it trigger?

Choices: Yes / No

Bridge feedback after trying

The total does not override the group rule. 70% is below 80%.

Words for this lesson

Distribution shift: A change in the data or conditions a system faces.
Concept drift: A change in the relationship between input and correct answer.
Trigger: A stated condition that starts an investigation or action.

The idea

Check new verified outcomes during use. A total score can hide a smaller-group problem. Changed inputs, changed answer rules and missing coverage need different checks. A falling score does not prove its cause. A useful trigger names a condition, a person and an action. Warning signals such as timeouts help early, but accuracy needs outcomes checked against known answers.

A completed example

Practice: 93% overall with a device group at 75% reaches a below-80% subgroup trigger. This starts investigation. It does not prove that a new microphone caused the errors.

Check your understanding

Same text now has a different correct category after policy change:

Possible concept drift
Only a screen change
Proven lying

Practice feedback after trying

Possible concept drift. The input-to-answer relationship changed.

New noisy microphone with unchanged categories:

Input condition shift
Definitely concept drift
No relevant change

Practice feedback after trying

Input condition shift. The input changed; the target rule need not have.

A score drop tells us ...

Results changed and need investigation
The exact cause
Retraining must fix it

Practice feedback after trying

Results changed and need investigation. Several causes are possible.

Task 1 · Make a prediction

Week trigger

Options: No trigger / Overall only / Subgroup only / Both

Write your prediction, or state that you skipped it.

Try it using supplied records

Predict whether Week 1 triggers either rule. Reveal its checked results.
Repeat for Weeks 2 and 3. Use the displayed fractions or calculator to check each group.
Choose the first triggered week. State which rule or rules were reached.
Open both Week 3 update notes. Choose the evidence needed to investigate each without assuming either is the cause.

Calculated monitoring checks · fictional classroom rules

Complete data and source records

Manually checked weekly results. Fictional teaching data.
Week	Typed correct out of 80	Speech correct out of 20	Update note
1	78	19	Original setup
2	78	18	No recorded update
3	77	10	Microphone update and routing-code update both occurred

Read as stacked records

Week: 1
Typed correct out of 80: 78
Speech correct out of 20: 19
Update note: Original setup

Week: 2
Typed correct out of 80: 78
Speech correct out of 20: 18
Update note: No recorded update

Week: 3
Typed correct out of 80: 77
Speech correct out of 20: 10
Update note: Microphone update and routing-code update both occurred

Classroom trigger rules. Fictional teaching data.
Rule	Condition
Overall	Investigate if overall accuracy is below 90%.
Subgroup	Investigate if either subgroup accuracy is below 85%.

Read as stacked records

Rule: Overall
Condition: Investigate if overall accuracy is below 90%.

Rule: Subgroup
Condition: Investigate if either subgroup accuracy is below 85%.

Possible cause cards. Fictional teaching data.
Card	Possibility
A	New microphone records noisy audio; category rules unchanged.
B	Policy changes which identical request texts count as urgent.
C	Users add urgent to routine requests after seeing its queue effect.
D	An uncommon access device was barely represented in initial testing.

Read as stacked records

Card: A
Possibility: New microphone records noisy audio; category rules unchanged.

Card: B
Possibility: Policy changes which identical request texts count as urgent.

Card: C
Possibility: Users add urgent to routine requests after seeing its queue effect.

Card: D
Possibility: An uncommon access device was barely represented in initial testing.

Expected results · supplied

These are supplied reference results. They are separate from your own explanation. No code needs to run to read them.

1. W1 overall 97%, typed 97.5%, speech 95%, no trigger. W2 96%,97.5%,90%, no trigger. W3 87%,96.25%,50%, both overall and speech trigger. Rules use strict below; exactly 90 overall or 85 subgroup does not trigger that rule.

2. A input shift -> check matched audio quality; B possible concept drift -> compare policy versions and checked labels; C changed behaviour, possible gaming -> inspect usage and intent without assuming malice; D coverage gap -> test independent relevant-device examples.

3. Testing lead reviews weekly verified samples; library lead owns manual routing; IT lead checks both updates. There is no proven cause in the supplied record. Optional threshold comparison uses A .9 urgent, B .8 routine, C .7 urgent, D .5 routine, E .3 urgent, F .1 routine: at .4 FP2/FN1; at .75 FP1/FN2. It changes decisions, not the source scores or microphone.

Help with this step

Hint 1

Apply the total and group rules separately.

Hint 2

The word or means either rule can start investigation. Updates are clues to check, not proof.

Hint 3

Practice: overall 91% does not prevent a trigger if a subgroup is 60%. Name a human who can act and what happens during the check.

Task 2 · Use the evidence

Complete the weekly decision record with overall and group results.
Match the four cause cards to a possible explanation and one check.
Write a plan naming owner, weekly frequency, trigger, immediate manual route and next investigation.

My explanation

Write in your school assignment. Keep these answers in your own words.

Week 1

Overall accuracy (%)
Typed accuracy (%)
Speech accuracy (%)
Trigger choice: No trigger / Overall only / Subgroup only / Both

Week 2

Overall accuracy (%)
Typed accuracy (%)
Speech accuracy (%)
Trigger choice: No trigger / Overall only / Subgroup only / Both

Week 3

Overall accuracy (%)
Typed accuracy (%)
Speech accuracy (%)
Trigger choice: No trigger / Overall only / Subgroup only / Both

First trigger

First trigger week: 1 / 2 / 3

Cause A

Possible explanation
Evidence needed

Cause B

Possible explanation
Evidence needed

Cause C

Possible explanation
Evidence needed

Cause D

Possible explanation
Evidence needed

Plan

Owner
Frequency
Trigger
Immediate action
Next check

Optional threshold extension

0.4 FP (requests)
0.4 FN (requests)
0.75 FP (requests)
0.75 FN (requests)

Explain a fresh example

Week 3 reaches the supplied rules. Write an owner and action plan, then state what the update record cannot prove.

My confidence

I need another explanation / I can explain with help / I can explain with my own example

My next step

Write what you will do next.

Optional depth

If correct labels arrive later, what can be monitored immediately? Explain why timeouts, missing data and complaints are warning signals rather than accuracy scores.

What this activity can show

Threshold changes do not establish that drift was repaired. Underrepresentation may have existed from the start. All thresholds here are invented classroom criteria.

This is a student learning record. Submission is completed through the school assignment system.