Security Control Experiment

Position in Module#

• Unit: 9
• Friday topic: Security and Trust Boundaries in Human-AI Collaboration
• Literacy perspective: System, Data, and Human-Context, with Interaction as the attack surface
• Collaboration level: Orchestration with security-aware control
• Fundamentals connection: Generative AI Security Fundamentals
• Challenge type: Security Control Experiment

This file is the detailed source of truth for the unit challenge. It specifies an evidence-based experiment that participants can transfer to their own professional, disciplinary, or organizational context.

Purpose and Demonstrated Mastery#

This challenge develops security-aware control in human-AI collaboration. Participants demonstrate that they can identify a trust-boundary failure, test it under controlled conditions, and redesign the workflow so that untrusted content, trusted instructions, source evidence, tool permissions, and human responsibility are handled more deliberately.

Clarify in the submitted work:

• the collaboration move participants practice: orchestration with security-aware control
• the literacy perspective through which they manipulate, observe, and interpret AI behaviour: System, Data, and Human-Context, with Interaction as the attack surface
• the generative or agentic AI fundamental that explains why the experiment matters: LLMs process instructions and evidence as language, so workflows need explicit boundaries between authority, source material, and action
• the caveat or limitation participants should keep visible: a workflow can appear safe under normal use but fail when untrusted content contains adversarial instructions, source provenance is weak, or permissions are too broad

Challenge Focus Logic#

• Difficulty: LLM systems may treat untrusted content as instruction, rely on poisoned or misleading sources, expose sensitive information, or trigger actions without adequate boundaries.
• Approach: Compare a weak baseline with a security-aware redesign that changes source handling, instruction hierarchy, retrieval trust, validation, permissions, approval checkpoints, or auditability.
• Artefacts: protocol document; prediction table; source/trust-boundary map; baseline and target run log; selected prompts, source excerpts, outputs, traces, screenshots, validation notes, or workflow diagrams; concise interpretation; one practical security-control rule for similar workflows.

Assignment#

Choose a realistic AI-supported task or workflow from your own context where security, trust, source quality, or responsibility matters.

Design a controlled comparison between:

• a baseline condition with a weak, implicit, or underspecified trust boundary
• a target condition with an explicit security-aware control

Before running the comparison, predict what should change and why. Then run or simulate the workflow, capture inspectable evidence, and interpret whether the target condition improved control.

The experiment must remain educational and contained. Do not attack live systems, bypass real provider safeguards, exfiltrate private data, or test against systems where you do not have permission.

Experimental Design#

Define before main execution:

• task/use case and intended output
• baseline configuration or workflow
• target configuration or workflow
• manipulated security-relevant condition
• what remains constant across comparison
• expected effects per manipulation/change
• evaluation criteria or conditions of satisfaction
• caveat to watch for: security controls may reduce one risk while leaving residual risks, usability costs, or false confidence

Possible manipulated conditions:

• clean source versus source containing adversarial or misleading instructions
• unlabelled source material versus labelled source authority and provenance
• direct answer generation versus answer generation plus source validation
• broad tool permissions versus least-privilege permissions
• no approval checkpoint versus explicit human approval before action
• unstructured prompt versus prompt with instruction hierarchy and source-handling rules
• single-step RAG answer versus separated retrieval, generation, citation check, and human review

Required Artefacts#

Required artefacts for this challenge:

• protocol document
• prediction table linking security-control changes to expected effects
• source/trust-boundary map identifying trusted instruction, user request, retrieved evidence, untrusted third-party content, tool permissions, and human approval points
• baseline and target run log
• selected evidence excerpts, screenshots, traces, retrieved passages, outputs, validation notes, or workflow diagrams
• concise interpretation plus one practical security-control rule for similar tasks

Unit-specific artefact focus:

protocol document; prediction table; source/trust-boundary map; baseline and target run log; selected prompts, source excerpts, outputs, traces, screenshots, validation notes, or workflow diagrams; concise interpretation; one practical security-control rule for similar workflows.

Evidence and Observation#

Participants should capture evidence that makes the comparison inspectable.

Define:

• what should be observed: whether the workflow preserves the difference between untrusted content, trusted instructions, source evidence, tool permission, and responsible action
• what counts as evidence for the selected use case
• which qualitative or quantitative criteria should be used
• how uncertainty, failures, caveats, usability costs, false positives, or unexpected results should be documented

Useful observation categories:

• Did the model follow or ignore malicious or misleading text in source material?
• Did the output distinguish evidence from instruction?
• Were citations or source references traceable?
• Were tool actions limited to appropriate permissions?
• Was human approval required at the right point?
• Did the redesign improve security without making the workflow unusable?
• What residual risk remained?

Interpretation#

Participants should explain:

• whether the target condition improved control compared with the baseline
• what evidence supports that conclusion
• how System, Data, Human-Context, and Interaction literacies explain the observed difference
• how the relevant security fundamental or caveat appeared in practice
• what residual risk remains after the redesign
• what practical rule they would carry into future work

The interpretation should not only say that the safer workflow was "better." It should explain which trust boundary or control point changed, why that mattered, and what limits remained.

Assessment Orientation#

Assess based on:

• meaningfulness of the baseline versus target comparison
• clarity of the trust-boundary or control-point diagnosis
• quality and coherence of protocol design
• credibility of variable/control isolation
• suitability of artefacts and evidence
• evidence-grounded interpretation and conceptual understanding
• explicit connection to relevant literacies, collaboration level, and security fundamental
• practical value of the final security-control rule
• responsible handling of the safety boundary

Teacher-Configurable Options#

• exact submission format
• work mode: individual or group
• whether the experiment uses pasted source excerpts, a toy RAG setup, a workflow diagram, or a real but permitted internal use case
• minimum number of runs or source variants
• whether tool use must be implemented or may be tightly specified
• optional extensions such as stakeholder feedback, lightweight threat modelling, or policy mapping