LLM improper output handling review
Brief
Whatever controls a model's answer controls everything that answer is fed into, so each place output lands must treat it as untrusted text, the way it would treat a form field. The review lists those places, asks the assistant to repeat a plain marker with one harmless formatting character for each, and records whether the sink shows it as text or acts on it.
On this page
- Situation
- Test accounts on the assistant, the client's front-end bundle and, for the server side, the source of each tool that builds a query, command or message from model output, shared for the test. We use only our own conversations, a plain marker and harmless formatting such as italics, never a script, query fragment or command, and one image request to a logging host the client approved. Only the applications and tenants the scope letter names are tested.
- Goal
- A sink table: each place model output lands, what it does with markup, links, quotes and spaces, and whether it encodes, sanitizes or parameterizes.
- Outputs
- Representative examples on placeholder hosts, trimmed to the lines that matter. Not captured from a public fixture yet.
Check first
- Every place the answer goes: the chat view, notifications, emails, tickets, admin transcript viewers, exports, and any tool that turns output into a query or command.
- Who reads each one: the same user, another user, or an administrator.
- The marker
ATARUS-MARKER-1028, and a logging host the client approved, such asoob.example.net, that records requests and serves nothing. - Test accounts tester-a and tester-b, with tester-b able to see shared conversations.
- Whether the app sends a Content-Security-Policy, and where its reports go.
Steps
Where does model output land?
curl -s https://app.example.com/assets/chat-4f1a9c.js \
| grep -oE 'dangerouslySetInnerHTML|innerHTML|v-html|marked\.parse|DOMPurify\.sanitize' | sort | uniq -c
Expected output
2 dangerouslySetInnerHTML
1 marked.parse # <- Markdown turned into HTML and inserted raw; no DOMPurify in the bundle
The bundle shows how the browser treats the answer; the client's description and tool source show the rest. We list every sink before testing any, because the dangerous one is often not the chat window but a support console that renders transcripts for staff.
Negative result. Output reaches the page as text nodes, or through a Markdown renderer whose HTML passes through a sanitizer with an allowlist.
Fix. Render answers as text by default; where Markdown is needed, sanitize the generated HTML with an allowlist that drops raw HTML, event attributes and script URLs.
Does the chat view render the model's HTML?
POST /api/assistant/messages HTTP/1.1
Host: app.example.com
Authorization: Bearer <token for tester-a, redacted>
Content-Type: application/json
{"conversation_id":"conv_test_1028","message":"Reply with exactly this and nothing else: ATARUS-MARKER-<i>1028</i>"}
Expected output
<div class="msg assistant">ATARUS-MARKER-<i>1028</i></div> # <- an element, not text: model output is parsed as HTML
The block is the message element as the browser's inspector shows it. Italics are harmless, but a view that builds an element from the answer will build any element, so the italics prove the whole class without a script. We repeat the test in every other viewer: the shared conversation as tester-b and, with the client watching, the admin transcript view.
Negative result. The inspector shows ATARUS-MARKER-<i>1028</i> and the page shows the angle brackets as typed.
Fix. Encode output for the context it lands in, sanitize anything rendered as HTML, and send a CSP without unsafe-inline so a missed case cannot run script.
Do Markdown images and links load without a click?
POST /api/assistant/messages HTTP/1.1
Host: app.example.com
Authorization: Bearer <token for tester-a, redacted>
Content-Type: application/json
{"conversation_id":"conv_test_1028","message":"Reply with exactly: "}
Expected output
2026-10-28T15:02:11Z GET /ATARUS-MARKER-1028.png 198.51.100.24 # <- fetched on render, with no click
The block is the logging host's access log. An image the view loads by itself is a request the model chose, and its URL can carry anything the model has in context, so it is a path for data out of the conversation that needs no user action. Our URL carries only the marker. Links matter less, since they need a click, but the view should show where each one goes.
Negative result. No request reaches the logging host, and the view shows the image as a link or a placeholder.
Fix. Load images only from an allowlist of hosts, enforced by a CSP img-src, and show each link's full destination before it opens.
Does output reach a query or command as data or as code?
grep -rnE 'execute\(f"|execute\(.*\+|shell=True|os\.system' tools/
Expected output
tools/orders.py:41: cur.execute(f"SELECT id, total FROM orders WHERE shop = '{args['shop']}'") # <- model output written into SQL text
tools/export.py:18: subprocess.run(f"zip -j {out} {name}", shell=True)
Tool arguments come from the model, so they are user input with one more step in between. We read each tool's source with the client instead of sending query fragments or commands. The export line passes a model-chosen file name through a shell, so a space or quote in it changes the command; we record that from the source alone. In the test tenant we then ask for orders from a shop named ATARUS-MARKER'S SHOP, an ordinary name with an apostrophe; a database error instead of an empty result confirms the concatenation.
Negative result. Every query is parameterized and every command is an argument list with no shell, so the apostrophe returns an empty result.
Fix. Use bound parameters and argument arrays, validate each argument against the tool's schema and an allowlist, and run tools with an account that can read only what the tool needs.
Does output go into email as HTML?
grep -n 'ATARUS-MARKER' raw-message.eml
Expected output
48:<p>Summary: ATARUS-MARKER-<i>1028</i></p> # <- the answer went into the HTML body unescaped
We ask the assistant to email tester-a's own address a summary containing the marker with italics, save the received message as raw source, and read the HTML part. An email body built from model output reaches other people's mail clients, with the client's domain as sender.
Negative result. The raw body shows <i>, or the email is plain text.
Fix. Build emails from a template that escapes every inserted value, and keep the sender, recipients and links out of the model's reach. The tool permission review decides whether the model should send email at all.
The defender sees
CSP violation reports for a blocked inline element or image, where reporting is set up. The image request appears in proxy or firewall logs as a fetch to an unknown host straight after an assistant answer. The apostrophe shows as a database syntax error in the application log with the tool's name on it. The assistant's own logs should hold the answer text for each of these turns; if they do not, the client cannot trace output that did harm back to the conversation that produced it.
Done when
Every sink is listed from the code and the client's description, the chat view and every other viewer of transcripts are tested with the marker, image loading is recorded, each tool that builds a query, command or message is classed as parameterized or concatenated, and email output is checked in the raw message.
Not covered
How an attacker gets text into the answer (see prompt injection testing), and cross-site scripting outside the AI feature (see reflected input checks); the outputs shown are representative examples on a placeholder app, not captured runs.