Reflection Loops: Critique, Revision, and Better Reasoning

Position in Module#

• Unit: 6
• Topic: Reflection Loops: Critique, Revision, and Better Reasoning
• Literacy perspective: Interaction, secondarily Mechanism
• Collaboration level: Reflection
• Fundamentals focus: Generative AI Fundamentals

Delivery Structure and Sequence#

The day has nine 45-minute lessons: 405 minutes, excluding breaks and lunch. Lessons 1–3 develop reflection workflows and AI support for human judgement. Lessons 4–9 all use CustomerSupportArena26, alternating demonstration and participant practice.

The delivered Arena sequence is: test cases → full context → run and inspect the judge → improve the judge → recheck its findings → improve the generator → check the new outputs.

Judge improvement is core classroom work. The separate challenge requires generator improvement; its judge comparison is optional.

In this delivery, lecturer availability moved Unit 6 directly after Unit 3. Keep the original numbering and intended sequence. Provide the opening evidence packs without assuming completion of Units 4–5. In future delivery, use the same opening to consolidate their grounding and process-design work.

Literacy Perspective#

• Primary literacy: Interaction
• Secondary literacy, if any: Mechanism
• What this literacy explains: Criteria, reviewer instructions, context and feedback shape critique and revision. Mechanism Literacy helps interpret model variation and the limits of attributing changes to one instruction.
• Supporting connections: Data supplies cases and evidence; System makes stages and handoffs explicit; Human-Context locates judgement and responsibility.
• What participants should manipulate, observe, and interpret: Compare review arrangements, change a judge on fixed outputs, then change a generator under a fixed judge. Inspect supported findings, missed errors, unjustified criticism and preserved strengths.

Collaboration Level#

• Level relation: Reflection
• Collaboration move being learned: Design how evidence, critique, human decisions and edits connect.
• What participants should be able to do differently after this Friday: Select and nest a useful workflow, define its inputs and output, check AI criticism and verify a proposed improvement.

Generative AI Fundamentals#

Distinguish two contributions:

In Lesson 3, initially omit AI scores, rankings, verdicts and suggested edits. This makes evidence preparation visible. Check omissions as well as included passages: the presentation can steer attention.

An LLM judge evaluates an output against criteria. Use observable anchors and a factual basis. Fluent critique, valid formatting and higher scores do not establish correctness. Output quality and evaluation quality are separate questions; a better judge may lower a score by detecting a real error.

Reflection Workflows, Sessions, and Nesting#

Use introduction.md and its W0–W9 register. The IDs are reusable references, not a sequence to complete. Choose an arrangement that addresses the current difficulty:

• review, decide and edit in one session;
• ask a fresh reviewer to return findings without editing;
• run separate perspectives on the same snapshot;
• consolidate reports into a proposed revision plan, retaining conflicts;
• edit accepted changes and verify them;
• prepare evidence for a human decision.

A stage can contain another workflow. For example, pause a review to clarify a criterion, then return the approved clarification to that review. Record trigger, inputs, returned result and return point. A diagram showing one level does not limit nesting.

Fresh sessions separate context; they do not establish independent expertise. Files carry state only when read or supplied. Extra roles and sessions should justify their coordination cost.

Opening Lesson Materials#

The introduction provides timings, prompts, case choices and facilitation notes. Use these handouts:

• Script: timed activities, prompt scaffolds and classroom records;
• Workflow overview: W0–W9, inputs, outputs and human decisions;
• Figures: the two judging arrangements, reflection, perspectives, nesting and evidence preparation;
• Workflow card: a reusable record of a selected workflow.

A suitable common case is a fictional recommendation for a six-month staff-mentoring pilot. Prepare a sponsor brief, participation estimates, workload constraints, stakeholder notes and one uncertainty. Include strengths and verifiable weaknesses. This is a preparation suggestion, not an existing case pack.

Keep one case through Lessons 1–2. Lesson 3 may reuse it or use an assessment, proposal or process trace. Opening outputs are a checked review decision, a completed or deferred sub-workflow, a checked evidence view and a practical rule. They are classroom records, not extra challenge submissions.

CustomerSupportArena26: Cases, Context, and Pipeline#

Use the Arena repository and its README. Participants operate the application; software implementation is outside this Friday.

Data 1: Test Cases#

Import data/test_cases_b2b_software.json: 15 synthetic enquiries, with KA-001–KA-012 for development and KA-013–KA-015 reserved for the final test. Inspect IDs, groups, inputs, expected behaviour and typical errors.

Annotations help people form expectations. Check them against the case and context; they are not a required answer wording. Neither model receives them. Choose a small development subset and keep it fixed across comparisons. Possible preparation choices are:

These examples support repeat delivery; they do not claim that every listed case was run in class. Keep final cases out of development.

Data 2: Context#

Import data/context_b2b_software_full.json and select it for both Agent and Judge. It contains company context and W01–W15. The shorter context_b2b_software.json contains only W01–W10.

Check each case against the relevant rules before running. The full context covers contracts, deletion, demos, roadmap and working scope. W15 limits the generator to response drafts: it cannot inspect accounts, book a demo or act on a contract. A proposed action differs from a claim that it already happened.

Distinguish missing evidence, a false customer premise and a generator error. This lab needs the imported context; it does not require a retrieval system or prior Unit 4 implementation.

Pipeline and Human Decisions#

Trace one case before running the batch:

Agent and Judge have separate context selections. Check both. The judge receives neither case annotations nor the generator prompt. Inspect quotations and rule references as well as scores. The Arena's human-first view carries forward Lesson 3's principle; it does not implement every evidence-preparation workflow.

Friday Activity Table#

This pacing makes the delivered sequence repeatable; lesson boundaries are not a minute-by-minute historical record.

Experiment Focus#

• Experiment focus: reflection arrangements; judge improvement on fixed outputs; generator improvement under one fixed judge.
• Prediction: before each change, name the affected case or criterion, expected effect and a possible remaining or new problem.

Call the selected classroom judge J* and record its actual ID. J* may be J0. Evaluate both G0 and G1 with J*: comparing G0/J0 with G1/J1 confounds response and evaluator changes.

If an exploration changes context or rubric too, label it a combined redesign and re-evaluate the same outputs. Do not attribute the result to one prompt change. Opening activities also combine changes. One run per configuration cannot separate instruction effects from model variation; conclusions concern the observed cases. Ties, regressions and keeping the baseline are valid outcomes.

Observation Focus#

• Workflow: are inputs, outputs, human decisions and returns clear, and did the extra stage help?
• Evidence: are passages and rules accurate, relevant and sufficiently complete? What is missing?
• Judge: which findings are supported, missed or unjustified? Recheck the human expectation too.
• Generator: which errors changed, which strengths survived and what new problems appeared?
• Retain: versions, configurations, run IDs, predictions, outputs, human ratings, findings, selected evidence, comparison notes and export. Classroom records may be compact and shared.

Challenge Kickoff Deliverables#

Preparation happens within the Arena lessons, especially Lesson 9; there is no additional block. Participants leave with an example judge-first comparison, working access and a narrow own-domain task they can judge competently. Start the report, E01–E03 and reserved A01, checked context, prompts and three anchored criteria using the starter. Run personal G0 only if ready.

In Challenge 6, J0 means the fixed judge in the participant's own experiment. The required task compares G0/G1, tests the selected generator on A01 and submits one report plus the full export. The optional judge comparison reuses G0 outputs after that work. Classroom judge improvement and opening workflow records add no submission or passing requirements.

Recurring Agentic Coding Demonstration#

A coding review can illustrate reflection in the opening: preserve a change, gather findings, approve edits and verify them with tests. Role design may itself be nested. It is a transfer example, not an extra block after the six Arena lessons.

Teacher Setup#

Prepare the introduction, four handouts, short checked cases and fallback outputs. Test access and import the exact full-context and case files. Prepare a manageable subset, expectations, rubric, G0/J0 examples and possible judge/generator changes. Label prepared outputs; live results may differ.

Operational checks:

• Use OpenAI mode for substantive experiments; demo mode has fixed responses and synthetic ratings.
• Identical successful judge evaluations may be reused. A reopened result or unchanged judge copy is not a fresh independent judgement. Stability or order probes need a separate live design and are not required blocks.
• Enter human ratings before individual judge results and the aggregate comparison. Record reasons and prior exposure when revising a rating.
• Keep cached results, failed calls, incomplete runs and unresolved disagreement visible. Better/worse counts are judge scores, not independent verification.
• Keep final cases reserved. Explain Abschluss without calling a known, self-authored case blind.

Choose the opening domain, development subset, models, group format and amount of personal challenge work to fit the cohort. Keep the stated comparison controls and nine-lesson structure.