# Unit 6 Introduction: Reflection Workflows and Support for Human Judgement

## Purpose and Delivery Context

This plan covers the first **three 45-minute lessons** of [Friday 6](friday.md). The remaining six lessons use the Arena: cases, full context, judge inspection and improvement, then generator improvement under a fixed judge. The [challenge](challenge.md) defines its own assignment and submissions.

In this delivery, Unit 6 followed Unit 3 because of lecturer availability. Keep the intended unit order. Supply small evidence packs; do not assume participants have completed Units 4–5.

Recall Unit 1's five phases and Unit 2's analysis workflow nested inside report production. The aim is to recognise, select and adapt these ways of working:

> Identify the difficulty, choose a workflow, specify its output and decide how to use that result.

A workflow has stages with inputs and outputs. It can span chats, people, models and files. Several stages can share a session; use a fresh context when its task, evidence, perspective or authority warrants separation. Saved files affect a later interaction only when supplied or read.

## The Two Directions of Judging

| | Lessons 1 and 2: AI reviews; humans control revision | Lesson 3: AI makes evidence available; humans judge |
|---|---|---|
| Leading question | Which criticism should we follow, and which edits should we carry out? | What do I need to see to spot a problem and decide what should improve? |
| AI contribution | Propose findings, compare against criteria, suggest changes, and implement accepted revisions | Locate passages, organise evidence, expose gaps, compare versions, and prepare an inspectable view |
| Human contribution | Define criteria, inspect findings, resolve disagreement, accept/reject/defer changes, and verify results | Define the decision and evidence needs, inspect originals, identify weaknesses, and formulate the judgement and next action |
| Main output | An improved artefact with a traceable revision decision | An evidence view that helps a human make and explain a judgement |
| Main risk to examine | Accepting plausible criticism or edits without checking their basis | Being steered by selective, incomplete, or misleading evidence presentation |

In Lessons 1–2, AI proposes a finding. In Lesson 3, it prepares evidence from which a human forms a judgement. Human responsibility remains in both arrangements.

In Lesson 3, initially omit AI scores, rankings, verdicts and proposed edits. Later comparison with AI recommendations is possible. Check the evidence view—a table or display of source passages and gaps—because selecting material already directs attention.

Reflection is the collaboration move. Interaction Literacy explains how criteria, context, roles and feedback affect it; Mechanism Literacy helps examine variation. System and Human-Context perspectives explain handoffs and decision authority.

## Learning Outcomes

After these lessons, participants should be able to:

1. Select a workflow in response to a concrete difficulty and explain when a simpler approach is sufficient.
2. Separate review, revision decisions, editing, and verification using explicit inputs and outputs.
3. Transfer a bounded task into a fresh session through artefacts and a handover prompt.
4. Pause a parent workflow, complete a useful sub-workflow, and return its result to the correct point.
5. Accept, reject, or defer AI feedback using criteria and evidence.
6. Specify and test an evidence view that helps another human identify an improvement.
7. Explain an observed benefit, limitation, and practical transfer rule without assuming that more sessions or roles automatically improve the result.

## Activity Formats and Classroom Organisation

Use these formats consistently:

- **Demonstration:** the teacher operates; participants observe, predict and annotate.
- **Guided device practice:** demonstrate one step, pause for participants to try it, then check that the expected output exists.
- **Group exercise:** groups of three or four choose an approach and inspect evidence without following each teacher action. Rotate driver, evidence checker and reporter.
- **Presentation and discussion:** exchange a concrete excerpt, decision and insight with a neighbouring group, then hear selected plenary examples.
- **Individual reflection:** predict, choose a workflow or record a transfer rule before hearing others' judgements.

Use one common demonstration case and a small choice of prepared cases for practice. Keep baseline, brief and evidence fixed within each comparison. Across domains, compare workflow choices; do not rank unlike artefacts on one quality scale.

Group reports answer: What was difficult? Which workflow helped? Which evidence changed our decision? What would we repeat or change? A diagram alone does not demonstrate improvement.

## Cases and Where to Obtain Them

Keep the main artefact short enough to inspect: approximately one page, three slides, a small table and chart, or a short workflow trace. Retain one case through Lessons 1 and 2. Lesson 3 may reuse it or move to an assessment or process-inspection case. Case selection should happen before class or from a prepared menu; browsing for material should not consume the lesson.

The following are preparation ideas, not claims that completed case packs already exist in this repository.

| Background or interest | Artefact to review | Where to obtain or prepare it | Evidence and useful tensions |
|---|---|---|---|
| Management, HR, organisational development | Recommendation for a six-month mentoring pilot | Prepare the fictional case described in [Friday 6](friday.md), with a brief, workload figures, stakeholder notes, and baseline memo | Evidence support versus persuasive presentation; ambition versus available staff time; omitted uncertainty |
| Education and assessment | Three short fictional answers to one question, or a draft feedback paragraph | Teacher-authored task and rubric; create clearly adequate, borderline, and weak responses. Participants may also contribute authorised teaching material | Criterion-to-passage mapping; partial evidence; an answer expressed differently from the reference; unsupported feedback |
| Communication, training, design | Three-slide explanation, tutorial, FAQ, or announcement | Reuse a participant's shareable Unit 1 output or prepare a draft from a short source and audience brief. [OpenLearn](https://www.open.edu/openlearn/) provides courses, articles, and other material from which a teacher can select a source | Accuracy versus accessibility; missing audience knowledge; attractive presentation that obscures an essential qualification |
| Research, analytics, public administration | Short interpretation of a table and chart | Reuse a checked Unit 2 output with its underlying results. For a new case, select data and metadata through [opendata.swiss](https://opendata.swiss/en), or an empirical project from [CORE Econ's Doing Economics](https://books.core-econ.org/doing-economics/index.html) | Numerical consistency; omitted denominator; mismatch between chart and claim; inference beyond available evidence |
| Entrepreneurship, consulting, project management | Business or project proposal | Teacher prepares three fictional one-page proposals against the same sponsor brief, budget, and selection criteria. Alternatively use a participant-owned, shareable proposal | Feasibility versus appeal; unequal evidence; missing assumptions; incompatible recommendations from reviewer perspectives |
| Operations, administration, customer service | Handover note, process instruction, or proposed customer response | Create a fictional process with a short policy and three events, or adapt an authorised public service description into a draft procedure | Missing prerequisite; ambiguous responsibility; incomplete handoff; evidence needed before recommending a change |
| Software and technical work | Small code change, README, test report, or execution trace | Use a teacher-owned toy project or a shareable Unit 2 script with a specification and prepared test output | Requirement coverage; observed failure versus guessed cause; documentation disagreement; a successful test that covers only part of the requirement |

For public-source cases, retain source details and use permissions appropriate to the selected material. Review a participant or teacher draft against the source; do not assume that an external source is itself defective. Keep classroom data authorised and non-sensitive. Fictional answers and proposals are sufficient for the assessment and batch examples.

For each prepared track, assemble:

- a fixed baseline and a short task/audience brief;
- three or four inspectable criteria and the evidence needed to apply them;
- an instructor key with two or three verifiable weaknesses, one genuine strength, and one arguable preference;
- one ambiguity or disagreement card for Lesson 2;
- a prepared review and evidence view for fallback use.

The instructor key is checked against the case evidence. AI may help draft the case, but its own critique is not the answer key. Preserve strengths so that participants can observe unnecessary revision as well as successful correction.

## Workflow Inventory for Later Visualisation

W0–W9 are reusable references, not a sequence or memorisation task. Introduce a workflow when it addresses an actual difficulty.

### Workflow Register

| ID | Workflow and trigger | Input → main steps → returned artefact | Human decision and exit |
|---|---|---|---|
| W0 | **Select and adapt a workflow.** A task or obstacle requires a next move | **Input:** Current goal and difficulty<br>1. Identify missing control or information.<br>2. Select/adapt a workflow, optionally with AI.<br>**Output:** Workflow card and next action | Human approves the scope and expected return; select a direct action if no sub-workflow is useful |
| W1 | **Reflective improvement.** An existing artefact needs review | **Input:** Fixed artefact, brief, criteria, evidence<br>1. Run W2 or W5.<br>2. Make human revision decisions.<br>3. Run W6.<br>**Output:** Revised artefact and verification record | Human decides which findings warrant action and whether to accept the revision, repeat a bounded step, or stop |
| W2 | **Review against criteria.** An artefact needs to be assessed | **Input:** Artefact version, criteria, source pack<br>1. Open a fresh reviewer session.<br>2. Inspect and link findings to evidence.<br>**Output:** Review report | Human checks the findings; reviewer does not edit the original; insufficient evidence is a valid result |
| W3 | **Clarify a criterion or evidence gap.** The current review cannot be justified | **Input:** Named ambiguity or missing evidence<br>1. Ask targeted questions.<br>2. Obtain human clarification or supplied evidence.<br>**Output:** Approved clarification note | Return to the paused step; if unresolved, defer the affected judgement rather than invent a basis |
| W4 | **Specify roles and prompts.** A missing perspective or unclear responsibility obstructs the work | **Input:** Task, difficulty, existing criteria<br>1. Ask AI to propose bounded roles, inputs, outputs, and prompts.<br>2. Inspect against the actual need.<br>**Output:** Approved role cards | Human rejects decorative or redundant roles and fixes each role's authority |
| W5 | **Combine reviews and reconcile changes.** Complementary or conflicting perspectives matter | **Input:** Same artefact version and evidence<br>1. Run W2 separately for each perspective.<br>2. Save separate Markdown reports or proposed edits.<br>3. Run a consolidation session.<br>**Output:** One proposed revision plan | Human accepts/rejects/defers each material change; unresolved conflicts remain visible; then return to W1 |
| W6 | **Edit accepted changes and verify.** A revision plan has been approved | **Input:** Baseline, accepted changes, sources<br>1. Run a bounded editing session to produce a revised version.<br>2. Verify separately against criteria and evidence.<br>**Output:** Revised version and change record | Human keeps or rejects changes based on the check; an increased judge score alone does not establish improvement |
| W7 | **Prepare evidence for human judgement.** A person needs to inspect and decide | **Input:** Decision question<br>1. Specify evidence requirements.<br>2. Have AI extract, align, and present evidence with locations and gaps.<br>3. Have the human inspect originals and judge.<br>**Output:** Decision record | Human identifies the weakness and action; initially no AI verdict, score, ranking, or proposed edit |
| W8 | **Capture workflow evidence.** A person needs to diagnose a process | **Input:** Decision question and process<br>1. Ask what observations would distinguish possible problems.<br>2. Design a minimal log.<br>3. Run or inspect a prepared trace.<br>**Output:** Evidence pack for W7 | Human checks coverage and missing observations; do not infer a cause the trace cannot distinguish |
| W9 | **Support human review across cases.** Several artefacts require comparable inspection | **Input:** Case IDs, originals, shared criteria<br>1. Repeat W7 preparation.<br>2. Inspect representative and ambiguous cases.<br>3. Adjust the view.<br>4. Prepare a small batch.<br>5. Record human decisions.<br>**Output:** Human decision records | Keep links to originals, missing evidence, and pending decisions; processing more cases does not validate the evidence extraction |

### Relationships to Show in Diagrams

- **Lessons 1–2 parent:** W0 selects W1. W1 contains W2, a human revision decision, and W6.
- **Clarification detour:** W2 pauses and invokes W3. The clarification note returns to the same criterion and artefact under review. If the criterion changes materially, reassess both compared versions against it.
- **Role-design detour:** W0 or W1 invokes W4 when an additional contribution needs definition. Its approved role cards configure W2 or W5.
- **Multiple-review branch:** W5 runs W2 for each chosen perspective on the same snapshot. The consolidation step produces one revision plan; it does not immediately overwrite the artefact.
- **Lesson 3 parent:** W0 selects W7. W8 can supply process evidence. W9 repeats the evidence-preparation part across cases while retaining human judgement.
- **Connection between both directions:** a human finding from W7 can become an accepted change for W6 or initiate W1. Display this as a deliberate decision, not an automatic rewrite.

In each diagram, distinguish human actions, AI actions, persisted artefacts, and decision points. Mark inputs, outputs, fresh-session boundaries, loops, and stop/defer paths. At every nested workflow show where it was invoked, what it returns, and where the parent resumes. Show one nesting level at a time for readability. Workflows may be nested wherever useful; the diagram is not a limit.

### Reusable Workflow Card

Participants complete this small record when selecting or adapting a workflow:

- **ID/name and trigger:** what difficulty makes this workflow useful now?
- **Parent and return point, if any:** which task is paused, and where will this result be used?
- **Inputs:** which artefact version, brief, criteria, and evidence are required?
- **Contributions:** who reviews, decides, edits, or verifies?
- **Returned artefact:** what must exist at the end?
- **Decision and stop condition:** what permits continuation, and what requires clarification, deferral, or stopping?
- **Expected benefit and cost:** what should become more controllable, and what extra effort is justified?

## Lesson 1 — Make Review an Inspectable Workflow

**Learning question:** Which parts of AI criticism deserve to become edits?

**Workflows:** W0, W1, W2, W6. Keep a short baseline unchanged. W1 may organise review and editing in one session; the guided exercise uses W2's fresh reviewer session to make the handoff visible. Neither arrangement establishes independent expertise.

| Minutes | Format | Teacher and participant activity | Observable result |
|---|---|---|---|
| 0–5 | Individual reflection and brief recall | Display the familiar context → clarification → approach → execution → review sequence. Participants locate today's work inside review, inspect their baseline themselves, and predict one defect AI will find and one it might miss | Initial human annotation and prediction |
| 5–10 | Demonstration | On the common case, ask for a harsh critique. Do not revise. Inspect one supported finding, one preference, or one unsupported objection if present. Use prepared contrasting output if needed and identify it as prepared | Annotated critique; distinction between forceful wording and justified findings |
| 10–25 | Guided device practice | In three short cycles, demonstrate and pause: open W2 in a fresh session with baseline, brief, evidence, and criteria; record a human accept/reject/defer decision on one or two findings; invoke W6 to edit an accepted finding and verify it against the original evidence | Saved review, revision decision, revised passage, verification note |
| 25–34 | Group exercise | Compare one participant case or a shared track. Check one finding against its source, identify a strength to preserve, and decide whether a proposed change is justified. Do not force a rejection if all findings are sound | One evidence-backed decision and one limit of the review |
| 34–42 | Presentation and discussion | Each group gives a 90-second explanation to a neighbouring group, then listens to its neighbour. Invite two one-minute plenary examples; use the remaining time to connect differences to criteria and evidence | Every group presents an excerpt, decision, and insight |
| 42–45 | Individual reflection | Complete a W1 card and finish: “I would use this loop when …; I would stop or defer when …” | First reusable workflow card |

### Facilitation and Evidence

“Harsh critique” is an experiment with an underspecified reviewing instruction, not the recommended default. Preserve its output so participants can compare the usefulness of findings. The later condition changes criteria, supplied context, and phase separation together; this short exercise does not isolate the effect of any one change.

Use a report structure such as: criterion, finding, artefact location, supporting evidence, uncertainty, and proposed change. Accept “no supported issue found” and “cannot assess from the supplied evidence”. A statement of model confidence is not a verification result.

For the human decision, record finding ID, accept/reject/defer, reason, and authorised edit. For verification, show the original passage, changed passage, criterion, evidence, and any remaining or new defect. A separate verifier session may assist, but a source check, calculation, test, or human inspection supplies the relevant corroboration.

The minimum successful outcome is one justified review decision and one checked revision, or an evidence-backed decision that no revision is warranted. Do not reward change for its own sake.

## Lesson 2 — Select, Nest, and Recombine Workflows

**Learning question:** What should we do when the review itself reveals a problem in how we are working?

**Workflows:** W0, W3, W4, W5, returning to W1 and W6. Reuse the case and saved state from Lesson 1. Specify a common version before producing additional reviews.

| Minutes | Format | Teacher and participant activity | Observable result |
|---|---|---|---|
| 0–4 | Individual reflection | Present an obstacle: an unclear criterion, missing evidence, or contradictory recommendations. Participants choose a next move before prompting | Workflow selection and predicted benefit |
| 4–9 | Demonstration | Invoke W4 for a concrete need, such as separating evidence support from implementation feasibility. Ask AI to propose roles and handover prompts; remove an unnecessary role and approve useful boundaries | Two role cards showing input, output, perspective, and authority |
| 9–20 | Guided device practice | Participants adapt the cards, open a reviewer session on their own devices, and save its report. Demonstrate a consolidation session receiving two reports and returning one proposed revision plan; pause for participants to perform the same consolidation on their devices. Supply a prepared second report when time or latency prevents another live run | Separate reports and a proposed consolidated plan, with conflicts retained |
| 20–32 | Group exercise | Each group receives one obstacle card below, selects a workflow, carries it out on a bounded passage, and records the result returned to the parent. The teacher coaches choices without prescribing a sequence | Completed sub-workflow, handoff, human decision, and marked return point |
| 32–40 | Presentation and discussion | Neighbouring groups exchange 90-second presentations. Two groups show contrasting choices in one-minute plenary reports. Discuss whether the extra step earned its cost | Comparison of when to clarify, add a perspective, reconcile, or stop |
| 40–45 | Guided device practice and individual reflection | Demonstrate resuming from the accepted returned artefact. Participants resume their own parent task with one bounded step and finish a workflow card and transfer rule | Evidence that the handoff changed the next action |

### Obstacle Cards for Short Group Work

Initially show participants only the obstacle. The suggested workflows below are teacher facilitation notes; reveal them after groups have selected and justified their own next move.

1. **Unclear criterion:** reviewers disagree about what “actionable” means. Use W3 to clarify the intended decision, approve one observable criterion, and return to the disputed finding.
2. **Conflicting perspectives:** one reviewer asks for brevity while another requires a qualification supported by the source. Use W5 to expose the conflict and propose a resolution. A human decides what must be preserved.
3. **Missing evidence:** a reviewer claims that the proposal is infeasible, but no workload information was supplied. Use W3 to identify what would be needed. Return either supplied evidence or an explicit deferred judgement.
4. **Unhelpful role proliferation:** a plan proposes five reviewers with overlapping tasks. Use W4 to reduce it to contributions that could change the decision, and explain the removed coordination cost.

In a large cohort, distribute cards across groups; no group needs to complete all four. Groups may conclude that another AI session is unnecessary if the human can resolve the issue directly.

### Roles, Markdown, and Handoffs

Keep an unchanged original and write proposed edits separately. A “read-only” prompt is an instruction, not a file permission; use permissions where supported.

Reviewers inspect the same snapshot without seeing one another's initial reports. The consolidator receives baseline, criteria, evidence and reports. It returns one revision plan with finding IDs, compatible edits, conflicts and deferred issues. Only accepted changes reach the editor. Use the same criteria for a before/after comparison; a new perspective can reveal another issue but does not prove improvement under the old criteria.

A role card states perspective, inputs, permitted actions, output and human decisions. A title does not confer expertise. AI can help draft roles, criteria or handoffs when a named difficulty warrants it. Stop designing once the workflow is ready to test.

## Lesson 3 — Make Evidence Visible for Human Judgement

**Learning question:** What do I need to see to identify a weakness and decide what should improve?

**Workflows:** W0 and W7; W8 for the telemetry example and W9 for the batch demonstration. The main manipulation is the evidence presentation available to a human, not the addition of a model verdict.

| Minutes | Format | Teacher and participant activity | Observable result |
|---|---|---|---|
| 0–5 | Individual reflection | Show a short fictional answer or proposal and a concrete decision. Participants note what they can judge, what they cannot judge, and what evidence they need | Initial judgement or explicit uncertainty and an evidence request |
| 5–12 | Demonstration | Use W7 to prepare an exam-correction view: criterion beside exact answer passages, reference material, and gaps, without an AI grade. Inspect an original passage and show how an omission could change a human judgement | Evidence view and a human finding; extraction distinguished from evaluation |
| 12–23 | Guided device practice | Participants choose assessment, proposal, communication, data, or process evidence. Ask AI what material is needed, approve the useful requirements, and produce a small evidence view. Check at least one extraction against its original | Evidence requirements and a checked view on each device |
| 23–33 | Group exercise | Exchange views with a neighbouring participant or group. A human reviewer identifies one supported concern or a reason to make no change. Another checks whether the view omitted relevant evidence. Improve the display or evidence request | Human judgement, source location, missing information, and proposed next action |
| 33–39 | Presentation and discussion | Groups present their view and insight in reciprocal 90-second exchanges; two one-minute plenary examples contrast different domains. Teacher identifies one transferable evidence requirement | Every group explains what the display enabled or obscured |
| 39–43 | Demonstration | Show a prepared business-proposal review interface or table. Open one case from the batch, trace a displayed passage to its original, and expose a missing-evidence case and a pending human decision | W9 overview and the role of storage, retrieval, and a review interface |
| 43–45 | Individual reflection | Write one rule: “To decide …, I need to see …; AI can prepare …, and I must still check …” | Personal rule for supporting human judgement |

### Three Application Pathways

| Pathway | Human decision | AI preparation | What the human checks |
|---|---|---|---|
| **Exam correction or feedback** | Which criterion is demonstrated, and what feedback is justified? | Align rubric clauses with original answer passages and relevant reference evidence; expose unmatched criteria and alternative passages | Whether passages support the criterion, whether omitted context matters, and whether an alternative valid answer is recognised |
| **Telemetry and workflow improvement** | Where should the process change, and what further observation is needed? | Organise a trace by phase; connect inputs, outputs, checks, handoffs, and human interventions; mark absent observations | Whether an alleged failure is observed, where evidence stops, and which additional measurement could distinguish explanations |
| **Business proposals or other cases at scale** | Which cases need clarification, revision, or further consideration? | Prepare comparable fields and criterion-linked source excerpts; retain contradictions, gaps, and links to complete originals | Whether evidence is comparable, whether the presentation privileges some cases, and whether each decision is justified |

For W8, telemetry means a record of observable workflow events. Ask AI: “What would you need to observe to make a useful suggestion about this workflow, and what would each observation let us distinguish?” A small log can record run/case ID, phase, input version, action, output location, check result, human intervention, and unresolved issue. Use a prepared trace during the lesson. Logging captures observable events; it is not access to a model's hidden thinking, and an absent log entry does not prove that an event did not occur.

For W9, begin with three cases, including a borderline or incomplete case, and check the evidence view before preparing more. A spreadsheet-style table, static page, or paper sketch can demonstrate the principle. If showing an existing application or one prepared with a coding agent, explain how the database retains cases and decisions and how the UI makes evidence inspectable. Building the application live is outside this lesson's time allocation.

The human judgement record should contain the decision, criterion, supporting and conflicting evidence, uncertainty, and next action. Avoid treating agreement between humans as automatic proof of correctness. Compare what they could substantiate from the available material.

The before/after experience is a classroom probe, not a causal estimate of interface quality: participants also become more familiar with the case. Ask which specific passage or relationship became visible and check that it is accurate.

## Prompt Scaffolds

Adapt these to the selected case. They illustrate workflow roles rather than fixed recipes.

**W2 — Reviewer**

> Review the supplied version against these criteria using the supplied brief and evidence. Do not edit the artefact. For each finding, identify the criterion, exact location, supporting evidence, uncertainty, and proposed change. Distinguish a supported defect from a preference or missing evidence. Include strengths that should survive revision. “No supported issue found” is an acceptable result.

**W4 — Role and prompt designer**

> We need to resolve [difficulty] in this review. Propose the smallest useful set of contributions. For each, specify perspective, inputs, permitted actions, output, and decisions reserved for the human. Draft its handover prompt. Explain which roles would be redundant. Wait for my decision before running the proposed review.

**W5 — Consolidator**

> Combine these separate reviews of the same version into one proposed Markdown revision plan. Preserve finding IDs and evidence links. Identify duplicates, compatible edits, conflicts, unsupported suggestions, and dependencies. Do not silently resolve consequential conflicts or change the original. Leave accept/reject/defer decisions for me.

**W7 — Evidence assistant**

> I need to decide [decision]. Help me specify what I must inspect. Using the supplied material, place the relevant original passages or observations beside each criterion and include their locations. Retain conflicting evidence and indicate gaps. Do not score, rank, judge, or rewrite the work. I will make the judgement after inspecting the evidence.

**Any handoff between sessions**

> Resume [workflow ID] at [step]. Use [listed artefacts and versions] as the working state. The accepted decisions are […]; unresolved issues are […]. Your task is […], and you may […]. Return [artefact and required fields]. Stop and ask if [condition]. Do not assume access to earlier chats.

## Working Artefacts and Evidence to Retain

Use Markdown or equivalent portable files. The following names describe proposed classroom outputs, not pre-existing repository files. Combine small records into one document where this reduces friction while preserving their roles.

| Proposed artefact | Purpose and owner | Used by |
|---|---|---|
| `brief.md`, `criteria.md`, source pack | Accepted task, audience, criteria, and evidence; human approved | W1–W7 as relevant |
| `baseline.md` | Unchanged starting artefact with version identity | Reviews and before/after checks |
| `review-a.md`, optionally `review-b.md` | AI findings and evidence locations from bounded perspectives | W5 and human revision decisions |
| `clarification.md`, `roles.md` | Approved output from a necessary design detour | Resumed W2 or W5 |
| `revision-plan.md` | Consolidated suggestions, conflicts, and human accept/reject/defer decisions | W6 |
| `revised.md`, `verification.md` | Revised artefact, changes, checks, remaining issues | Human acceptance and later challenge evidence |
| `evidence-view.md` or equivalent table/sketch | AI-prepared material for human inspection, with links to originals | W7 and W9 |
| `human-judgement.md` | Human finding, evidence, uncertainty, and next action | Further investigation or revision |
| `workflow-card.md` | Selected workflow, trigger, inputs, return point, stop rule, and lesson learned | Transfer to participants' own work |

Participants should leave with one checked review decision, one completed or explicitly deferred sub-workflow, one tested evidence view, and a practical rule. The introduction provides material for Challenge 6; it does not add a separate submission requirement or replace the challenge's experimental design.

## Teacher Preparation and Pacing

Prepare the common case and two or three alternatives with checked criteria, short templates, review reports and an evidence view. Have participants choose a track beforehand where possible. The case menu above offers options, not a requirement to prepare every domain.

Test fresh-session handoffs. Use prepared outputs if access or latency blocks a live run, label them and continue the human checking activity. Pasted text or paper views can support the same learning.

Give groups a presentation card: difficulty; workflow and return point; evidence excerpt; human decision; insight. Show the current workflow node during transitions, without introducing every branch at once.

Coordinate with the Arena lecturer. Lessons 4–9 preserve cases, context, outputs and judgements in the application. First improve the judge on stored outputs against checked expectations; then fix the selected judge and test a generator change. This classroom sequence adds no challenge obligation: the challenge's judge extension remains optional.

**Shorter delivery option:** If only two lessons are available, use one reviewer, demonstrate role design and consolidation with prepared reports, and combine one nesting exercise with W7. Move the batch-interface example and deeper judge testing into the later programme. This is an alternative schedule, not the documented nine-lesson delivery.

## Research Notes for the Teacher

The activity design above is a course-design proposal. Two research findings support testing the judge rather than treating fluent feedback as validation:

- [Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685) examined position, verbosity, and self-enhancement biases alongside useful agreement with human preferences in the models and tasks studied. Use this as motivation for stability checks, not a universal performance claim about current tools.
- [Huang et al., Large Language Models Cannot Self-Correct Reasoning Yet](https://arxiv.org/abs/2310.01798) reported limitations and sometimes degradation in intrinsic reasoning self-correction without external feedback under the tested conditions. This motivates checking revisions against sources, tests, and human inspection; it does not establish that every contemporary reflection workflow must fail.