Security Control Experiment

Position in Module#

• Unit: 9
• Friday topic: Security and Trust Boundaries in Human-AI Collaboration
• Literacy perspective: System, Data, and Human-Context, with Interaction as the attack surface
• Collaboration level: Orchestration with security-aware control
• Fundamentals connection: Everyday Generative and Agentic AI Security
• Challenge type: Security Control Experiment

This file is the detailed source of truth for the unit challenge. It specifies an evidence-based, contained experiment that participants can transfer to their own professional, disciplinary, or organizational context.

Purpose and Demonstrated Mastery#

This challenge develops security-aware control in human-AI collaboration. Participants demonstrate that they can map a realistic task or workflow, identify a security-relevant trust-boundary or control-point weakness, predict how it may affect behaviour, test it safely, and redesign the collaboration so that content, evidence, instructions, sensitive information, permissions, actions, and human responsibility are handled more deliberately.

The aim is not to demonstrate professional offensive-security skill or to prove that a workflow is secure. The aim is to produce evidence that one deliberate control changes an observable outcome and to explain what risk still remains.

Clarify in the submitted work:

• the collaboration move being practised: orchestration with security-aware control
• the literacy perspectives used to manipulate, observe, and interpret behaviour: System, Data, Human-Context, and Interaction
• the relevant security pattern, such as indirect prompt injection, poisoned or misleading source content, sensitive-information exposure, excessive agency, tool misuse, missing validation, weak approval, or inappropriate human reliance
• the conceptual reason the experiment matters: LLM-supported workflows do not reliably enforce the distinction between instructions, content, evidence, and authority, so capability and responsibility boundaries must be designed around them
• the limitation that remains visible: layered controls can reduce likelihood or impact but do not justify a claim of complete security

Learning Goals Assessed#

The challenge should provide evidence that participants can:

1. map the security-relevant boundaries and responsibilities of an AI-supported task
2. correctly distinguish the failure or attack pattern relevant to their experiment
3. form a prediction about how untrusted content, source quality, data access, permissions, validation, approval, or human reliance should affect the outcome
4. compare a weak baseline with a controlled redesign while keeping the main task sufficiently stable
5. capture inspectable evidence of model, workflow, tool, or human behaviour
6. evaluate the effectiveness, cost, and limitations of the selected control
7. formulate a practical rule and escalation or approval boundary for similar future work

Scope and Safety Boundary#

The experiment must be educational, contained, and authorized.

Participants must not:

• attack live services, colleagues, customers, or systems they do not own or have explicit permission to test
• attempt to extract real private data, credentials, system prompts, or protected model information
• bypass real provider safeguards, execute destructive actions, or publish operational exploit instructions
• upload confidential or regulated information merely to make the experiment realistic
• interpret one successful control test as proof of security

Use harmless prepared or participant-authored test content, synthetic or non-sensitive data, and reversible actions. An implemented agent or retrieval system is optional; a well-specified simulation with inspectable evidence is acceptable.

Training-data poisoning, model backdoors, model inversion, membership inference, and specialist embedding attacks may inform the risk analysis at awareness level, but they are not expected as hands-on experiments.

Challenge Focus Logic#

• Difficulty: AI-supported work may confuse untrusted content with instruction, rely on manipulated or weakly governed sources, expose sensitive information, act with excessive capability, or invite inappropriate human reliance.
• Approach: Compare a weak baseline with a security-aware redesign that changes source handling, data access, instruction boundaries, retrieval trust, output validation, tool permissions, approval checkpoints, auditability, or human decision logic.
• Artefacts: protocol document; prediction table; source/trust-boundary map; baseline and target run log; selected evidence such as prompts, source excerpts, outputs, citations, traces, screenshots, validation notes, permission maps, approval records, or workflow diagrams; concise interpretation; one practical security-control rule.

Assignment#

Choose a realistic AI-supported office task or workflow from your own context where information integrity, confidentiality, authority, action, trust, or responsibility matters.

Choose one primary pathway:

Pathway A: Office Content and Indirect Prompt Injection#

Use a document, email, web excerpt, ticket, meeting note, form submission, or similar source. Compare ordinary source content with a prepared variant containing a harmless conflicting instruction or misleading cue. Test whether the workflow preserves the distinction between source evidence and authorized instruction.

Pathway B: Grounded or Retrieval-Supported Work#

Use a task that relies on a source set, knowledge base, pasted evidence, or simulated RAG process. Compare a weak source, provenance, access, or validation condition with a more controlled design. Test effects on integrity, confidentiality, citation traceability, or appropriate reliance.

Pathway C: Agentic Workflow Assurance#

Use an implemented or tightly specified workflow involving tools, memory, external communication, or state-changing actions. Compare broad or implicit capability with a design using least agency, scoped permissions, deterministic validation, exact-action approval, audit traces, or clearer human responsibility.

Across all pathways, design a controlled comparison between:

• a baseline condition with a weak, implicit, or underspecified trust boundary or control point
• a target condition with an explicit security-aware control

Before running the comparison, predict what should change and why. Then run or simulate the workflow, capture inspectable evidence, and interpret whether the target condition improved control, what it cost, and what risk remained.

Experimental Design#

Define before main execution:

• task or use case and intended output or action
• people, information, systems, or decisions that could be affected
• relevant risk pattern and why it fits the case
• baseline configuration or workflow
• target configuration or workflow
• manipulated security-relevant condition
• what remains constant across comparison
• expected effects per change
• evaluation criteria or conditions of satisfaction
• evidence-capture plan
• authorization and safety boundary
• caveat to watch for: a control may reduce one risk while leaving residual risk, usability costs, false positives, approval fatigue, or false confidence

Possible manipulated conditions:

• clean source versus source containing a harmless adversarial, conflicting, or misleading instruction
• unlabelled source material versus explicit provenance and source-authority labels
• broad context sharing versus task-minimal data and context
• retrieval followed by generation versus authorization before retrieval plus source validation
• direct answer generation versus generation followed by deterministic or human validation
• broad tool permissions versus task-scoped, least-privilege permissions
• unnecessary autonomy versus least agency for the task
• no approval checkpoint versus exact-action human confirmation before an external, privileged, or irreversible action
• unstructured prompt versus explicit source-handling and authority rules, while recognizing that prompt rules alone are not guarantees
• single-step grounded answer versus separated retrieval, generation, citation checking, and human review
• persistent memory write versus reviewed, logged, or non-persistent handling of the same information
• hidden or summarized action details versus a review interface that exposes the actual proposed action

Run enough repetitions or variants to avoid basing the interpretation on one accidental output. The teacher may set a minimum based on the available system and class time.

Required Artefacts#

Required artefacts for this challenge:

• protocol document
• prediction table linking the security-control change to expected effects
• source/trust-boundary map identifying:
  • user intent and trusted instructions
  • retrieved evidence and untrusted third-party content
  • sensitive information and access boundaries
  • model output and validation
  • tool capabilities, permissions, and possible actions where relevant
  • human approval, escalation, and responsibility points
• baseline and target run log
• selected evidence excerpts, screenshots, citations, traces, retrieved passages, outputs, validation notes, permission maps, approval records, or workflow diagrams
• concise interpretation
• one practical security-control rule for similar tasks, including when human escalation or approval is required

The artefacts should make both the observed improvement and the residual risk inspectable.

Evidence and Observation#

Participants should capture evidence that makes the comparison reviewable.

Define:

• what should be observed: whether the workflow preserves the difference between untrusted content, trusted instructions, evidence, sensitive information, permitted capability, and responsible action
• what counts as evidence for the selected use case
• which qualitative or quantitative criteria should be used
• how repeated or inconsistent behaviour will be recorded
• how uncertainty, failures, caveats, usability costs, false positives, approval burden, or unexpected results will be documented

Useful observation categories:

• Did the model follow, repeat, ignore, or flag conflicting or misleading content?
• Did the output distinguish source evidence from instruction or recommendation?
• Were source provenance, citations, and retrieved passages traceable?
• Was information retrieved or exposed only to an authorized task and user?
• Did prompts, logs, traces, context, or outputs contain unnecessary sensitive information?
• Were tool calls and external actions limited to the intended task and permission scope?
• Could memory or another persistent artefact carry the manipulated content into later work?
• Was human approval required at the right point, and did the reviewer see the actual proposed action?
• Did confident or persuasive output influence human judgment despite weak evidence?
• Did the redesign improve control without making the workflow unusable?
• What residual risk remained?

Interpretation#

Participants should explain:

• whether the target condition improved control compared with the baseline
• what evidence supports that conclusion and how consistent it was
• which security pattern best explains the observed failure or risk
• how System, Data, Human-Context, and Interaction literacies explain the observed difference
• why the selected control affected likelihood, impact, traceability, or responsibility
• what usability cost, workflow friction, false positive, or approval burden the control introduced
• what residual risk remains after the redesign
• what practical rule, responsibility boundary, or escalation condition they would carry into future work

The interpretation should not only say that the safer workflow was "better." It should explain which trust boundary or control point changed, why that mattered, what the evidence cannot establish, and why the system should not be described as completely secure.

Assessment Orientation#

Assess based on:

• relevance of the selected office or agentic use case
• correct and useful identification of the security pattern being tested
• clarity of the trust-boundary, data-access, capability, and responsibility map
• meaningfulness of the baseline versus target comparison
• quality and coherence of protocol design
• credibility of variable and control isolation
• responsible, contained handling of all test content and systems
• suitability and completeness of artefacts and evidence
• evidence-grounded interpretation across relevant literacies
• recognition of control limitations, usability trade-offs, and residual risk
• practical value of the final security-control and escalation rule

Passing should not depend on whether the prepared failure occurs in every run. A well-designed experiment that documents non-occurrence, variability, or an ineffective control can demonstrate mastery if the evidence and interpretation are credible.

Teacher-Configurable Options#

• exact submission format
• work mode: individual, pair, or group
• which of the three pathways are offered or required
• whether the experiment uses pasted source excerpts, a prepared toy retrieval setup, a workflow diagram, or a real but permitted internal use case
• whether agentic tool use must be implemented, may be demonstrated, or may be tightly specified
• minimum number of runs or source variants
• required evidence types
• whether stakeholder feedback or peer review is included
• optional extensions such as lightweight threat modelling, organizational policy mapping, or a comparison with relevant OWASP risk categories