Why Models Behave Differently: Parameters, Variability, and Control

Position in Module#

• Unit: 3
• Topic: Why Models Behave Differently: Parameters, Variability, and Control
• Literacy perspective: Mechanism
• Collaboration level: Instruction, with mechanistic control
• Fundamentals focus: Generative AI Fundamentals

Literacy Perspective#

• Primary literacy: Mechanism
• Secondary literacy, if any: None
• What this literacy explains: Tokenisation, context and token selection help explain output differences and difficulties with exact character tasks.
• What participants should manipulate, observe, and interpret: Change one visible generation condition. Compare variation within each condition with differences between conditions, including compliance, factual stability and failures.

Collaboration Level#

• Level relation: Instruction, with mechanistic control
• Collaboration move being learned: Specify and test the conditions under which an instruction runs.
• What participants should be able to do differently after this Friday: Predict an effect, collect repeated observations and derive a practical control rule without claiming direct access to model internals.

Generative AI Fundamentals#

1. Tokens: Models process token sequences. Tokens may represent characters, word parts or other text units. Human-visible words are not necessarily single tokens.
2. Tokenisation: Demonstrate a simplified byte-pair procedure: merge frequent adjacent units to build a vocabulary. Label any hand-drawn split as illustrative; inspect the actual tokenizer before presenting a split as fact. See the Hugging Face BPE explanation.
3. Representation limits: Compare a familiar word, a typo such as Andlefingen, and an exact-letter task. Shared context may help recover intended meaning; it does not guarantee correct counting or character manipulation. Keep observed success separate from guesses about training examples.
4. Attention and context: Explain that attention combines information from positions in the available sequence. Surrounding text can change how an ambiguous expression is interpreted.
5. Token selection: A next-token distribution offers alternatives. Sampling settings influence selection, so repeated runs can differ. Lower variation does not establish correctness.

Demonstrate a failed or uncertain character task, then add explicit decomposition or a checking tool. The useful rule is to check exact operations explicitly. For exploratory work, test whether variation supplies useful alternatives.

Recurring Agentic Coding Demonstration#

If used, repeat one small coding task under two model or reasoning conditions. Keep repository state, brief, files, permissions and test command fixed. Retain each plan, diff, test result and configuration record. Compare consistency, constraints and edge cases. This is a behavioural comparison, not an explanation of hidden reasoning; the same method applies to non-coding tasks.

Shared Classroom Experiment#

Use a common task before participants design their own experiment. The default is:

From a fixed fictional briefing about an internal knowledge-base pilot, write a 160-word management note with exactly three benefits, three risks and one recommendation. Add no facts absent from the briefing.

Prepare a 400–600-word briefing with concrete facts and one uncertainty; a fixed request; criteria for structure, source fidelity, unsupported additions and usefulness; two configuration cards differing in one condition; and a run log. Add three short character or tokenisation tasks.

Each group runs at least three trials per condition. The default classroom comparison uses lower and higher sampling variability in the same model. If the interface does not expose that setting, use one documented model or reasoning difference, or prepared outputs with authentic configuration records. Do not change prompt, model and sampling together.

This short classroom exercise prepares the method. The challenge has its own model, reference data, automation and repetition requirements below.

Friday Activity Table#

Indicative contact time: 365 minutes, excluding breaks and lunch. Adjust pacing while preserving prediction, repetition, comparison and interpretation.

Use the shared pack, a controlled interface or prepared runs, and a spreadsheet or board for comparisons. The kickoff uses the local workspace and WAILAI materials from Unit 2.

Experiment Focus#

• Experiment focus: the effect of one generation condition on variation and task performance.
• Classroom constants: task, request, source material, output limit, tools, criteria and environment; keep the model fixed unless it is the selected classroom variable.
• Before each condition: predict a change in variation, compliance, factual stability or usefulness and explain why.

Observation Focus#

• Observation focus: repeated-output variation, between-condition differences, constraint compliance, unsupported claims and useful alternatives.
• Evidence participants should capture: request, configuration, numbered outputs, scores, comparison, character-task attempt and repair, unexpected results.
• Interpretation participants should be prepared to make: Which differences fit the prediction? Which might reflect ordinary variation? What combination of settings and verification fits the task?

Challenge Kickoff Deliverables#

Use the German Challenge 3 as the assignment authority. Participants leave with:

• a real task with expected outputs defined and checked before execution;
• a draft .agents/EXPERIMENT.md: references, GPT-3.5 Turbo version, fixed baseline prompt/settings, output format, repetitions, metrics and raw-result files;
• a plan for approximately 20–50 cases, ten independent calls per case per condition;
• a first traceable call or runner test, if ready, preserving the raw output;
• a WAILAI analysis plan using ANALYSIS.md, PYTHON.md and PLAN.md;
• an understanding that the diagnosed intervention and its prediction are recorded after A but before B.

In the challenge, B changes one factor. Cases, references, GPT-3.5 Turbo, repetitions, evaluator and analysis stay fixed. The three-trial classroom exercise and its model-substitution option do not replace these requirements. Review the reference check, optional model evaluator, raw-output records and submission folder in the brief. Failed improvement remains a valid result.

Teacher Setup#

Prepare the shared pack, verified settings, run log, character tasks and a completed comparison. Keep at least three authentic prepared runs per classroom condition for access failures. Test the challenge runner route and the local analysis environment before class.

Choose classroom tools, one manipulated condition, group size, scoring display and scaffolding. You may adapt the common task to another suitable domain while retaining shared inputs, criteria and repeated trials. Extra repetitions are useful where time permits. Keep configuration metadata and clearly identify prepared outputs. Do not convert classroom variations into new challenge obligations.