Steering a service within its limits

Your goal

Choose a service improvement using test evidence and keep a route for failures.

To be able to recommend a way to improve a generative service using evidence about facts, behaviour and limits.

Before you start

A reply repeats the wrong category from last week. Would a shorter sentence necessarily fix the fact?

Choices: No / Yes

Words for this lesson

Capability

What a system can do under stated conditions.

Context window

The limited amount of input and recent text processed at once.

Fine tuning

Further training that adjusts learned model settings.

The idea

A prompt can clarify behaviour. Current approved context can supply facts. The system can still copy the wrong detail, so check its sources and test answers. Fine tuning changes learned settings through further training, but a proposal is not a tested result. Context is limited, so relevant evidence must actually be available. A simpler search or a person may fit the task better.

A completed example

Practice: the source lists Mint Atlas as science. A reply calls it history. Adding the source was useful, but the reply still failed the source check. More confident wording would not repair the category.

Check your understanding

9/10 correct proves ...

1. Performance on those ten cases
2. Universal reliability
3. No need for review

Does current context automatically change weights?

1. No
2. Yes
3. Only if there are two cards

Can we claim the fine-tuning proposal outperforms C?

1. No, it has no results
2. Yes, because it is advanced
3. Yes, because it costs more

Task 1 · Make a prediction

Trial C improvement proves weights changed

Options: Yes / No / Not sure

Write your prediction, or state that you skipped it.

Try it using supplied records

1. Compare the approved catalogue card with the authored reply and select the conflicting fact.
2. Use the window illustration: only two selected cards are available to the scripted response at once. Choose whether the relevant catalogue card is included.
3. Inspect trials A, B and C on the same ten questions. Predict whether C's improvement proves the model weights changed.
4. Open proposals D and E. Mark their performance as Untested.

Supplied trials · two-card context analogy

Complete data and source records

Expected results · supplied

These are supplied reference results. They are separate from your own explanation. No code needs to run to read them.

1. A question is correct only when the title exists and category and availability all match the approved catalogue. Correct rates A 50%, B 70%, C 90%. The unsupported-claim column overlaps with incorrect answers and is not added to the denominator.

2. Window illustration uses three cards: catalogue, opening hours, events. Two can be selected. If catalogue is excluded, show Relevant source unavailable; if included, show Source available, still check the reply. This is an analogy; real windows are commonly measured in tokens, not two cards. Use the approved catalogue entry as the Catalogue card and the two supplied additional source rows for Opening hours and Events.

3. A-C do not change learned settings in the scenario. D would if trained; E uses search. C combines interventions, so its gain cannot be attributed to context alone. Do not generate an invented outcome for D or E.

Help with this step

Task 2 · Use the evidence

1. Compare correct answers and unsupported title claims in A to C.
2. Recommend a limited next test or reviewed pilot, giving two results and one uncertainty.
3. State what can and cannot be claimed about D and E. Name the failure route and a relevant check before expanding use.

My explanation

Write in your school assignment. Keep these answers in your own words.

Source conflict

• Claim
• Correction

Trial A

• Correct out of 10 (questions)
• Percent (%)
• Unsupported claims (claims)
• Current source: Yes / No / Not yet established
• Review: Yes / No / Not yet established

Trial B

• Correct out of 10 (questions)
• Percent (%)
• Unsupported claims (claims)
• Current source: Yes / No / Not yet established
• Review: Yes / No / Not yet established

Trial C

• Correct out of 10 (questions)
• Percent (%)
• Unsupported claims (claims)
• Current source: Yes / No / Not yet established
• Review: Yes / No / Not yet established

Proposal D

• Evidence status: Tested / Untested
• Settings scope: No change required / Would change if trained / No trained model needed / Searches records
• Next evidence

Proposal E

• Evidence status: Tested / Untested
• Settings scope: No change required / Would change if trained / No trained model needed / Searches records
• Next evidence

Your recommendation

• My next pilot or comparison
• Failure route and uncertainty

Explain a fresh example

Recommend the next step after C achieved 9/10 with one unsupported claim. Include a failure route and explain what remains unknown.

My confidence

I need another explanation / I can explain with help / I can explain with my own example

My next step

Write what you will do next.

Optional depth

Design a comparison that separates context from source checking. Which conditions would you hold fixed?

What this activity can show

All outputs and trials are supplied fictional evidence. The two-card window is a labelled analogy. No live generation, retrieval service or fine tuning is implemented.

This is a student learning record. Submission is completed through the school assignment system.