Leadership · lesson 3 of 4 · 16 min
Secure the AI lifecycle
Protect AI systems from unsafe inputs, data leakage, excessive access, and failure across their full lifecycle.
- Enumerate the AI-specific attack surface
- Apply least privilege to agents
- Extend logging and IR to AI systems
The surface is wider than the model
Six places break. Training and fine-tuning data (poisoning, leakage of anything you fine-tuned on). The prompt and context path (injection through any content the system reads). The model and its supply chain (weights, unsigned artifacts, malicious pickles from public hubs). Retrieval and vector stores (documents a user should not be able to reach through the assistant). Tools and integrations (the model's ability to act). Output handling (downstream systems trusting model text as code or as a command).
An AI system granted more functionality, permissions, or autonomy than its task requires, so that a single manipulated input can cause consequential real-world action.
An internal assistant was given the same document permissions as the service account that indexed the file share. Staff could ask it questions about HR investigations they had no access to. Nothing was hacked. The retrieval layer simply inherited permissions nobody re-scoped, and the model answered faithfully.
Least privilege applies to agents too
The most consequential control in an agentic deployment is the permission set. An assistant that can read the mailbox, browse the web, and send mail is a data exfiltration path that requires no exploit, only content. Scope credentials to the task, keep read and write separate, require explicit human confirmation for any consequential action, and log every tool call with its arguments. Assume the model will be talked into using every permission you grant.
Log what a responder would need
AI incidents are hard to investigate because the evidence is usually not retained. At minimum keep: the prompt and system prompt version, the model and version, retrieved document ids, every tool call and its arguments, the output, and the identity of the requester. Retain it long enough to matter and treat it as sensitive, because prompts contain whatever people pasted into them.
Rehearse the shutdown
Every high-tier system needs an owner who can disable it, a tested mechanism to do so, a defined fallback for the work it was doing, and a written trigger for pulling it. Run the drill. The first time a team discovers that disabling the assistant also takes down the ticket queue should not be during the incident.
Audit one assistant's permissions
Pick a deployed AI assistant and write down every system it can read from and every action it can take.
Include permissions inherited from a service account. Those are the ones that surprise people.
0 of 5 practice steps checked. Saved on this device.
Assistant could read the full SharePoint index via a service account, send email as the user, and create tickets. Task required reading two document libraries and drafting replies. Revoked: send-as. Replaced with draft-only plus a human send. Retrieval re-scoped to the two libraries with user-level permission checks at query time.
Check your understanding
Answer both to mark the lesson complete. You can try again. This is practice, not a graded exam.
1. Which control most directly limits the damage from a successful prompt injection?
2. An internal assistant surfaces documents a user cannot otherwise open. The root cause is usually:
Three things to remember
- Six surfaces: training data, prompt path, model supply chain, retrieval, tools, output handling.
- Least privilege and human confirmation bound what an injection can achieve.
- Log prompts, versions, retrieved ids, and tool calls, and rehearse the shutdown.

The full oversight framework behind this path, across identity, data, AI, vendors, and risk.
Examples are composites written for teaching unless a source is cited. Figures come from the sources listed where given.