hego.red - Practical AI/LLM Red Teaming Notes

Practical notes on AI/LLM red teaming

Start here

Welcome. These are practical, hands-on notes on red teaming LLMs, written from a pentester's point of view. Use the tabs above to move around: Foundation explains what you are really testing, Attacks and Methodology are the how, Bootcamp is a guided course and PortSwigger has worked labs, Scope and Attack Flow help you scope a target, and Terminology is the quick dictionary. New here? Just keep reading this tab from top to bottom. Press / any time to search everything.

Foundation - What Are We Actually Testing?

Read this first. In one minute it shows you what an "AI feature" really is, the words model / chatbot / agent, and how the AI makes an answer. Once you see the picture, the other tabs make sense.

1. You test the app, not the brain

An "AI feature" is just a normal app with an AI model plugged in. You test the app the client built. The model's brain is usually the vendor's (Claude, OpenAI) and out of scope.

You / the chat box
where you type your message
↓
The AppYOU TEST THIS
the client built this part:
  • adds hidden rules (the system prompt)
  • may read documents or a database (RAG)
  • may call tools (email, database, run code)
  • shows the answer back to the user
↓
The Model / "the brain"usually the vendor's
it just turns text into more text
Almost every bug lives in the green box (the app), not the brain. Getting the brain to say something rude is the vendor's problem, not a real finding.

2. Model vs Chatbot vs Agent

These three words confuse everyone. It is just a ladder - each step adds one thing.

Model (LLM)
the brain. Reads text, guesses the next word. That is all.
→
Chatbot
model + hidden rules + a chat window. It talks to you.
→
RAG app
chatbot + it can read documents and your data.
→
Agent
model + tools. It can DO things and take steps, not just talk.

The further right, the more it can do - and the more you can attack.

3. How it makes an answer

The model does not "think". It reads text and guesses the next word, again and again. The thing worth watching is not the guessing, it is who wrote the text it is guessing from. Five kinds of text end up in front of it, and only one of them was written by the people who built the app.

System promptWritten by the developer. The only text here that the app owns, and the prize for anyone who can make the model repeat it back. ↳ prompt leaking
Your messageWritten by whoever is typing. On a public chatbot, that is anyone in the world. ↳ direct injection
A documentWritten by whoever wrote the file, the page or the record. Anyone who can leave one where the app will read it is now writing to the model. ↳ indirect injection
A tool resultWritten by whoever controls what the tool touched: a website, an API, an inbox, a database row. ↳ agent hijacking
The answerWritten by the model out of everything above. Any source it names is something it generated, not something the runtime tracked. ↳ insecure output

One picture, read left to right. Five kinds of text arrive on the left, get joined into a single string, and are handed to the model. Watch the amber document: it sits inside the green user turn, and by the time the model has it, it is gone. Every box carries the attack it turns into, and the three dashed marks are the only places a control can stand.

next turn: what you were shown comes back as history System promptthe developer wrote it ↳ prompt leaking Conversation historyearlier turns, already grey ↳ memory poisoning Your messagewhoever is typing ↳ direct injection Retrieved documentwhoever wrote the file ↳ indirect injection Tool resultwhoever owns the site ↳ agent hijacking prompt assembly adds role tags input filter CONTEXT WINDOW system assistant tool user pasted document the model system assistant tool user the roles survived. the authors did not. 1 roles are also tokens 2 reads all text before it 3 predicts next, repeats a role is a prior, not a rule output filter Rendered to youmarkdown, links, images ↳ insecure output The answerone grey blob citations are generated A tool callif it asked for one ↳ excessive agency tool approval A website, an API,an inbox, a database row ↳ data exfiltration the agent loop: whatever it fetched comes back as text 1234567

Every arrow carries text. The role tags survive all the way into the model, but they label turns and not authors, and all three controls are outside the box.

  1. Five sources. Only the first was written by the people who built the app. The other four come from whoever is typing, whoever wrote a file, or whoever owns a site the app touched. A picture counts too, because the model reads the text inside it.
  2. A template joins them, with a role tag on each turn. The tags are real and they survive into the token sequence. They mark turns. They do not mark who wrote the text inside a turn.
  3. The context window is a length limit, not a store. When a conversation outgrows it something has to go. Most apps pin the system prompt and evict from the middle or summarise; the naive ones drop from the top and lose it.
  4. The model reads it as one sequence. Each token reads everything before it and nothing after, which is why text placed early steers everything that follows. It predicts one token, appends it, and runs again.
  5. The answer comes out without provenance. If it cites a source, that citation was written by the same pass that read the poisoned text. It is a claim, not a chain of custody.
  6. It is rendered, or it becomes a tool call. Rendering runs markdown, links and images in your browser. A tool call reaches out to something the app does not own.
  7. Both come back around. What you were shown returns as history, what the tool fetched returns as new text, and the next request starts with more material nobody vouched for.

4. The one flaw everything comes from

Look at the model box in the diagram again. The role tags went in and came out the other side, but the amber document did not: it was folded into the green user turn, and inside the model there is nothing left that says a stranger wrote it.

The roles are real. A modern API takes system, user, assistant and tool as separate fields, and they survive into the token sequence as delimiter tokens. What does not survive is anything finer than a turn.

A retrieved document has no role of its own. It is pasted inside a user turn, or inside a tool turn, so by the time the model reads it, it carries exactly the standing of the turn it was folded into. The same goes for anything a tool brought back: the wrapper and the attacker's website are one blob under one label.

And a role is a prior, not a permission. Models are trained to weight a system instruction above a user one, and most of the time that training holds. It is a tendency learned from examples, not a rule the runtime enforces, which is why a confident enough instruction sitting in a tool result can still win. Instruction and data remain the same material.

So the controls that actually exist sit before this box or after it: input filters, retrieval allowlists, output scanners, tool approval, sandboxes. The ones that try to work inside it, like instruction-hierarchy training, are priors too, and priors bend. Which is why the useful question is never "is the model safe". It is which of these arrows can an attacker write on, and what is on the other end of the box when they do.

This is the whole game: the role tags mark turns, not authors, so text that arrived as data can act like an instruction. That is prompt injection, and almost every other attack builds on it.

5. So where are the bugs?

Each layer you saw above has its own bug. This is the OWASP LLM Top 10, in plain words:

LayerThe bug
Your inputPrompt injection - your text acts like a command.
The modelJailbreak (break its safety); if fine-tuned, leak its training data.
Documents / RAGPoison the documents so the model obeys them (indirect injection).
ToolsMake it use a tool it shouldn't, or attack the tool's input (SQLi, SSRF, run code).
The answer being shownInsecure output handling - the app runs the answer as HTML/SQL, so XSS and friends.

6. The risk depends on the app

The same answer can be fine in one app and a disaster in another. So before you test, ask: what is this app for, and what would count as "bad" here?

AppWhat "bad" looks like
Story / game generatorwants wild, creative output - almost anything goes.
Internal HR or support botmust stick to the facts - making up a policy is the bug.
Email writer for the companyshould be honest but on-brand - rude or dishonest text is the bug.

You usually want the model's intelligence (good language and reasoning), not its knowledge - it should answer from your data and say "I don't know" otherwise.

Two myths to drop. (1) "AI risk is just sci-fi robots taking over." No - the real risks are here now: your bot can leak data, give harmful answers, or get the company sued today. (2) "A bigger, smarter model is safer." Benchmark scores do not tell you how safe it is in YOUR app. Test your app, not the leaderboard.

Sources: OWASP Top 10 for LLM Apps, PortSwigger Web LLM attacks, MITRE ATLAS. Next: the Terminology tab for the words, then Methodology for the plan.