Security and Trust Boundaries in Human-AI Collaboration

Position in Module#

• Unit: 9
• Topic: Security and Trust Boundaries in Human-AI Collaboration
• Literacy perspective: System, Data, and Human-Context, with Interaction as the attack surface
• Collaboration level: Orchestration with security-aware control
• Fundamentals focus: Everyday Generative and Agentic AI Security

Learning Goals#

By the end of the Friday, participants should be able to:

1. map the security-relevant boundaries of an AI-supported task, including user intent, trusted instructions, untrusted content, sensitive data, retrieved evidence, tools, possible actions, and human responsibility
2. distinguish ordinary misinformation, direct prompt injection, indirect prompt injection, jailbreaking, poisoned sources or memory, sensitive-information disclosure, and excessive agency
3. predict and test how untrusted content in an email, document, web page, ticket, retrieved passage, or tool response can redirect AI behaviour
4. evaluate safeguards as layers rather than guarantees, including source handling, data minimisation, input controls, output validation, least privilege, least agency, approval checkpoints, and audit traces
5. assess the integrity and confidentiality of a grounded or retrieval-supported workflow, including provenance, authorization, traceability, source poisoning, and the privacy implications of embeddings and logs
6. assess an agentic workflow for goal hijacking, tool misuse, excessive permissions, unsafe external communication, memory or context poisoning, and missing human checkpoints
7. recognize how fluent or persuasive output can produce automation bias, inappropriate reliance, and unclear accountability
8. prepare a contained, evidence-based security-control experiment in a realistic professional context

Literacy Perspective#

• Primary literacy: System Literacy
• Secondary literacies: Data Literacy, Human-Context Literacy, Interaction Literacy
• What these literacies explain: System Literacy explains how security depends on workflow architecture, tool permissions, checkpoints, handoffs, and boundaries between trusted and untrusted inputs. Data Literacy explains how poisoned, misleading, weakly governed, or overexposed sources affect integrity and confidentiality. Human-Context Literacy explains why trust, accountability, disclosure, appropriate reliance, and escalation matter. Interaction Literacy explains how crafted or conflicting language can redirect model behaviour.
• What participants should manipulate, observe, and interpret: Participants manipulate source trust, instruction placement, retrieval content, validation steps, workflow checkpoints, permission boundaries, or approval requirements. They observe whether the system treats text as evidence or authority, whether sensitive information or unsafe action becomes possible, and whether added controls improve traceability and responsibility without creating false confidence.

Collaboration Level#

• Level relation: Orchestration with security-aware control
• Collaboration move being learned: Participants learn to design human-AI workflows that distinguish trusted instructions, user requests, retrieved evidence, untrusted third-party content, model output, tool authority, and responsible human action.
• What participants should be able to do differently after this Friday: They should be able to inspect an AI-supported workflow for security-relevant trust boundaries and redesign it with clearer source handling, validation, data minimisation, scoped permissions, explicit approval points, and accountable human oversight.

Scope and Safety Boundary#

This is not a professional penetration-testing or exploit-development unit. It develops informed, security-aware use and workflow design for everyday generative and agentic AI.

Core and experiential:

• direct and indirect prompt injection, with emphasis on indirect injection through office content
• the distinction between evidence, instruction, authority, and action
• malicious or misleading retrieved content and weak source provenance
• layered safeguards and their limits
• sensitive-information handling in prompts, context, retrieval, outputs, traces, and logs
• agent goal hijacking, tool misuse, excessive agency, permissions, validation, and approval
• human overreliance, accountability, escalation, and residual risk

Awareness level:

• jailbreaking as the subset of prompt injection aimed at overriding model safety safeguards
• training-data and model poisoning, fine-tuned backdoors, and model supply-chain uncertainty
• model inversion, membership inference, and the possibility that embeddings or model behaviour reveal protected source information
• specialist vector and embedding attacks beyond ordinary source, access-control, and provenance decisions

Out of scope:

• attacking live services or systems without explicit authorization
• teaching operational safeguard-bypass recipes, system-prompt extraction, data exfiltration, or exploit chains
• hands-on training-data manipulation, model extraction, model stealing, or exploitation of production vulnerabilities
• treating successful completion of a classroom test as proof that a workflow is secure

Everyday Generative and Agentic AI Security Fundamentals#

The Friday uses the OWASP GenAI LLM Top 10 2026 and the OWASP Top 10 for Agentic Applications as reference maps, not as lists participants must memorize. The most relevant risk families for this course are prompt injection, sensitive-information disclosure, excessive agency, data and model poisoning, hidden-context exposure, vector and embedding weaknesses, improper output handling, agent goal hijacking, tool misuse, memory or context poisoning, and human-agent trust exploitation.

The conceptual foundation should make the following distinctions visible:

• LLMs process instructions and evidence through the same language-processing channel. This creates a boundary problem when text that should be treated as data is interpreted as an instruction.
• Direct prompt injection is supplied through the immediate user interaction. Indirect prompt injection arrives through external content such as an email, document, web page, ticket, retrieved passage, database record, image, tool response, or persistent memory.
• Jailbreaking is an important term to recognize, but for this unit it is treated as an attempt to override model-level safety safeguards rather than as a separate practical skill.
• Ordinary misinformation is not necessarily an attack. It becomes security-relevant when confident but false output influences decisions, enters downstream processes, or triggers actions.
• Runtime instructions carried by retrieved content are an indirect prompt-injection problem. Durable corruption of training, fine-tuning, embeddings, persistent memory, or model artefacts is a poisoning problem. Participants should be able to distinguish these even when both produce misleading behaviour.
• Grounding does not automatically create trust. A retrieval-supported system can become less reliable or less confidential when provenance is weak, authorization is applied too late, a corpus is poisoned, or a vector store exposes information across users or contexts.
• Embeddings should not be treated as anonymized source material. At awareness level, participants should understand that embeddings, logs, traces, context, and generated output may disclose information about underlying documents or users.
• Prompt injection becomes more consequential when a model can access private data, call tools, communicate externally, persist memory, or change system state. Security therefore depends strongly on capability and workflow design.
• System prompts, filters, safety training, and output filters can reduce risk but are not guarantees. High-impact controls should be layered and enforced outside the model where possible.
• Useful system-level controls include data minimisation, source provenance, authorization before retrieval, separation of external content, deterministic output validation, least-privilege tools, least agency, exact-action approval, audit trails, rate or action limits, and clear responsibility boundaries.
• Human approval is only useful when the reviewer can see and understand the actual proposed action. Fluent explanations, urgency, authority cues, and approval fatigue can undermine oversight.

The central learning point is:

Secure human-AI collaboration depends on preserving the difference between untrusted content, trusted instructions, source evidence, permitted capability, and responsible action.

Recurring Agentic Coding or Office-Workflow Demonstration#

Demonstrate a small office or agentic coding workflow in which the AI reads an untrusted email, issue, README section, web page, PDF, shared document, or retrieved passage containing a harmless conflicting instruction. Compare a weak workflow, in which the model can treat the content as authority or act with broad capability, with a controlled workflow that separates evidence from authority and limits what happens downstream.

The demonstration should make visible:

• the intended task and responsible human decision-maker
• trusted project or organizational instructions
• user intent
• retrieved or externally supplied evidence and its provenance
• any sensitive information available to the workflow
• model output and any deterministic validation applied to it
• available tool permissions and possible external or state-changing actions
• exact points where human confirmation is required
• logs, traces, or artefacts that make behaviour inspectable

The recurring coding case may be used to preserve continuity with earlier Fridays, but a non-programming office case should also be available so that participants can transfer the security logic into their own contexts. The point is not to teach exploit writing. It is to show how trust boundaries, data access, permissions, validation, and human reliance shape controllability.

Friday Activity Table#

Experiment Focus#

• Experiment focus: trust-boundary and control-point design in a realistic office or agentic workflow
• Manipulated variable, condition, or design decision: Participants vary at least one security-relevant condition, such as clean versus manipulated content, explicit versus implicit source authority, broad versus restricted data or tool access, absent versus present validation, absent versus exact-action approval, or unsegmented versus separated retrieval and action steps.
• What remains constant across comparison: Keep the user task, intended output, source domain, available non-manipulated information, and evaluation criteria stable enough that observed differences can be interpreted credibly.
• Safety constraint: Use harmless prepared or participant-authored test content and systems the participant is authorized to use. No live safeguard bypass, private-data extraction, or unauthorized action is required.

Observation Focus#

• Observation focus: whether the workflow preserves the distinction between untrusted content, trusted instructions, evidence, sensitive information, permitted capability, and responsible action
• Evidence participants should capture: prompts, source texts, provenance labels, retrieved excerpts, outputs, citations, tool-call traces, workflow diagrams, permission maps, approval records, validation notes, screenshots, comparison tables, or stakeholder observations
• Interpretation participants should be prepared to make: Explain which failure pattern appeared, how the manipulated trust boundary or control point changed the observed outcome, what usability or workflow cost the control introduced, and what residual risk remains.

Challenge Kickoff Deliverables#

Participants must leave Friday with:

• a selected context-relevant task or workflow and one of three pathways:
  • office document or email use exposed to indirect prompt injection or misleading content
  • grounded or retrieval-supported work exposed to weak provenance, poisoned sources, or inappropriate access
  • agentic work exposed to goal hijacking, excessive permissions, unsafe action, memory risk, or missing approval
• a source/trust-boundary sketch identifying intent, instructions, content, evidence, sensitive information, capabilities, actions, and human responsibility
• a baseline definition for a weak or underspecified condition
• a target condition or redesigned control
• a first protocol draft covering the intended output, prediction, manipulated condition, constants, evaluation criteria, safety boundary, and evidence plan
• first captured evidence, such as a classified source, baseline run, prepared manipulated-content example, workflow sketch, capability map, or approval/checkpoint map

The kickoff should already reflect the challenge-focus logic:

• Difficulty: AI-supported work may confuse data with instruction, rely on manipulated or weakly governed sources, expose sensitive information, act with excessive capability, or invite inappropriate human reliance.
• Approach: Compare a weak baseline with a security-aware redesign that changes source handling, data access, instruction boundaries, retrieval trust, validation, permissions, approval, or auditability.
• Artefacts: protocol document; prediction table; source/trust-boundary map; baseline and target run log; selected prompts, source excerpts, outputs, traces, screenshots, validation notes, permission maps, approval records, or workflow diagrams; concise interpretation; one practical security-control rule.

Teacher Setup#

Mandatory:

• a short, office-relevant risk map based on current OWASP LLM and agentic guidance without requiring taxonomy memorization
• a set of harmless cases that distinguish misinformation, injection, jailbreaking, poisoning, disclosure, excessive agency, and overreliance
• prepared clean and manipulated source texts, or a low-friction simulated retrieval/workflow setup
• one non-programming office example and, where useful, the recurring agentic coding example
• trust-boundary, capability-map, protocol, and evidence-capture templates
• a control catalogue covering source handling, data minimisation, authorization, validation, permissions, approval, logging, and human responsibility
• a clear safety boundary: all tests must be educational, contained, and authorized

Configurable:

• exact tools or platforms
• whether retrieval is simulated with pasted excerpts or implemented with a prepared system
• whether an agentic workflow is executed, demonstrated, or assessed as a tabletop design
• whether participants inspect tool-call traces, workflow diagrams, generated outputs, or a combination
• work mode: individual, pair, or group
• minimum run count, scaffolding depth, and peer-feedback format
• which professional domains are represented in the source pack