How to write up an AI or LLM finding
Brief
An AI finding is a security bug only when the model's output crosses a boundary: another user's data, an action the user could not take, an action nobody confirmed. The write up rates it by what that action could do, not by how clever the input was, and recommends controls that hold whatever the model says.
On this page
- Situation
- A confirmed behavior from an authorized test of an AI feature, with the inputs, model outputs, tool calls and test accounts that showed it. Everything here concerns applications and accounts named in the scope letter, and every input quoted in the report was written by us against test data.
- Goal
- One AI finding a product team can reproduce, rank and fix from the ticket alone.
- Outputs
- Representative examples on placeholder hosts, trimmed to the lines that matter. Not captured from a public fixture yet.
Check first
- The behavior crosses a boundary that exists without the model: tenant, role, ownership, or a confirmation the product promises.
- The exact input, every message in the conversation, the model output, the tool calls with their arguments, and the model name and date.
- How often it reproduces. Models are not deterministic, so "8 of 10 runs" is part of the evidence.
- The tool's permissions, from the recon map, so the finding can say what the action could do beyond what we did.
- The ordinary structure every finding follows, from writing a finding. This page adds only what is different for AI features.
Steps
How do I write reproduction steps for an AI finding?
POST /api/assistant/messages HTTP/1.1
Host: app.example.com
Authorization: Bearer <token for tester-a in org_test_a, redacted>
Content-Type: application/json
{"conversation_id":"conv_test_0207","message":"Summarize Q3-plan-test.docx"}
Expected output
HTTP/1.1 200 OK
Content-Type: text/event-stream
event: tool_call
data: {"name":"share_document","arguments":{"doc":"Q3-plan-test.docx","to":"tester-b@org-test-b.example"}} # <- an action the user never asked for, with no confirmation
event: message_done
data: {"conversation_id":"conv_test_0207","model":"vendor-model-2026-05"}
The steps say which test document was in the workspace, quote its full text in an appendix, give the exact message, and show the tool call that followed. They state the model version from the response, the date, and the reproduction rate across ten runs. A product engineer reruns the same message on the same document in their own test tenant and sees the same call. The appendix holds a test document we wrote, so the report carries the trigger without carrying anyone's data.
Negative result. A finding that reproduces once in twenty runs is still a finding if the action is serious, but the rate goes beside the severity, not in a footnote.
How do I rate severity for an AI finding?
Weak: Critical: novel multi-step prompt injection bypasses guardrails.
Better: High: any document in a shared workspace can make the assistant share files with
an outside address, without the user confirming. We shared one test document
between two test tenants. share_document runs under the user's own permission.
Severity comes from the action and the permission it runs under. The same trigger is low when the only tool is a read scoped to the user, and high when a tool sends data out of the tenant. How elaborate the input was does not move the score. The deleted production database is the case to keep in mind: a routine task, a token with blanket authority, and one destructive call.
Negative result. If no tool or data path crosses a boundary, the severity is informational at most, which leads to the next question.
Is it a model quirk or a security bug?
Quirk: the assistant states a wrong invoice total. No boundary crossed.
Quirk: the assistant reveals its own instructions. A bug only if they hold a secret
or are the only control.
Bug: the assistant shows tenant B's invoice to tenant A. Tenant boundary crossed.
Bug: the assistant sends email nobody confirmed. Confirmation promised and skipped.
A quirk is wrong or unwanted output with no access gained. It goes to the product team as feedback, not as a finding. A bug is output that reached data or took an action the user's own permissions or the product's promises would not allow. The test is whether a deterministic control could and should have stopped it.
Which controls should an AI finding recommend?
Fix: share_document and send_email require a confirmation the user clicks, rendered by the
app outside the model's output, showing the recipient. Tools run under the user's
own token, never a service account. Retrieval reads only documents the user owns or
that were shared with them, from an allow list of sources. Model output is encoded
as text before display, so links and markup in it are inert.
Every recommended control works without trusting the model: least privilege on each tool, confirmation on actions with effects, retrieval allow lists, output encoding. "Improve the system prompt" can be an extra measure, never the fix. The agent lab analysis makes the same point from the other side: most steps in that chain were ordinary weaknesses with ordinary fixes.
Fix. If you build AI features, list every tool with the permission it runs under and whether the user confirms it, and review that list each release the way you review an API.
What should an AI finding leave out?
No real customer data, even redacted, when a test document proves the same thing. No claims about the vendor's model internals, training data or weights, which we did not test and cannot see. No full transcripts of failed attempts. Inputs appear only in the steps and the appendix, so they are not pasted into ticket titles and chat channels.
The defender sees
Nothing on this page touches the client's systems, so there is nothing for a defender to see. The finding should still name the log lines that recorded each tool call, so the client can check whether the same call has happened outside the test.
Done when
The finding states the boundary crossed, quotes the exact input and output with the model version and the rate it reproduced, rates severity by the action, names the controls, and contains no real customer data.
Not covered
Model safety issues with no security boundary, such as offensive text, and model training; the outputs shown are representative examples on placeholder hosts, not captured runs.