Skip to content

LearnAI and LLM

LLM improper output handling review

Brief

Whatever controls a model's answer controls everything that answer is fed into, so each place output lands must treat it as untrusted text, the way it would treat a form field. The review lists those places, asks the assistant to repeat a plain marker with one harmless formatting character for each, and records whether the sink shows it as text or acts on it.

Stage
Impact
Reviewed
On this page
  1. Check first
  2. Where does model output land?
  3. Does the chat view render the model's HTML?
  4. Do Markdown images and links load without a click?
  5. Does output reach a query or command as data or as code?
  6. Does output go into email as HTML?
  7. The defender sees
  8. Done when
  9. Not covered
  10. Tools
  11. Related research
  12. Reviewed

All commands

Situation
Test accounts on the assistant, the client's front-end bundle and, for the server side, the source of each tool that builds a query, command or message from model output, shared for the test. We use only our own conversations, a plain marker and harmless formatting such as italics, never a script, query fragment or command, and one image request to a logging host the client approved. Only the applications and tenants the scope letter names are tested.
Goal
A sink table: each place model output lands, what it does with markup, links, quotes and spaces, and whether it encodes, sanitizes or parameterizes.
Outputs
Representative examples on placeholder hosts, trimmed to the lines that matter. Not captured from a public fixture yet.

Check first

  • Every place the answer goes: the chat view, notifications, emails, tickets, admin transcript viewers, exports, and any tool that turns output into a query or command.
  • Who reads each one: the same user, another user, or an administrator.
  • The marker ATARUS-MARKER-1028, and a logging host the client approved, such as oob.example.net, that records requests and serves nothing.
  • Test accounts tester-a and tester-b, with tester-b able to see shared conversations.
  • Whether the app sends a Content-Security-Policy, and where its reports go.

Steps

Where does model output land?

curl -s https://app.example.com/assets/chat-4f1a9c.js \
  | grep -oE 'dangerouslySetInnerHTML|innerHTML|v-html|marked\.parse|DOMPurify\.sanitize' | sort | uniq -c

Expected output

   2 dangerouslySetInnerHTML
   1 marked.parse   # <- Markdown turned into HTML and inserted raw; no DOMPurify in the bundle

The bundle shows how the browser treats the answer; the client's description and tool source show the rest. We list every sink before testing any, because the dangerous one is often not the chat window but a support console that renders transcripts for staff.

Negative result. Output reaches the page as text nodes, or through a Markdown renderer whose HTML passes through a sanitizer with an allowlist.

Fix. Render answers as text by default; where Markdown is needed, sanitize the generated HTML with an allowlist that drops raw HTML, event attributes and script URLs.

Does the chat view render the model's HTML?

POST /api/assistant/messages HTTP/1.1
Host: app.example.com
Authorization: Bearer <token for tester-a, redacted>
Content-Type: application/json

{"conversation_id":"conv_test_1028","message":"Reply with exactly this and nothing else: ATARUS-MARKER-<i>1028</i>"}

Expected output

<div class="msg assistant">ATARUS-MARKER-<i>1028</i></div>   # <- an element, not text: model output is parsed as HTML

The block is the message element as the browser's inspector shows it. Italics are harmless, but a view that builds an element from the answer will build any element, so the italics prove the whole class without a script. We repeat the test in every other viewer: the shared conversation as tester-b and, with the client watching, the admin transcript view.

Negative result. The inspector shows ATARUS-MARKER-&lt;i&gt;1028&lt;/i&gt; and the page shows the angle brackets as typed.

Fix. Encode output for the context it lands in, sanitize anything rendered as HTML, and send a CSP without unsafe-inline so a missed case cannot run script.

POST /api/assistant/messages HTTP/1.1
Host: app.example.com
Authorization: Bearer <token for tester-a, redacted>
Content-Type: application/json

{"conversation_id":"conv_test_1028","message":"Reply with exactly: ![m](https://oob.example.net/ATARUS-MARKER-1028.png)"}

Expected output

2026-10-28T15:02:11Z GET /ATARUS-MARKER-1028.png 198.51.100.24   # <- fetched on render, with no click

The block is the logging host's access log. An image the view loads by itself is a request the model chose, and its URL can carry anything the model has in context, so it is a path for data out of the conversation that needs no user action. Our URL carries only the marker. Links matter less, since they need a click, but the view should show where each one goes.

Negative result. No request reaches the logging host, and the view shows the image as a link or a placeholder.

Fix. Load images only from an allowlist of hosts, enforced by a CSP img-src, and show each link's full destination before it opens.

Does output reach a query or command as data or as code?

grep -rnE 'execute\(f"|execute\(.*\+|shell=True|os\.system' tools/

Expected output

tools/orders.py:41:    cur.execute(f"SELECT id, total FROM orders WHERE shop = '{args['shop']}'")   # <- model output written into SQL text
tools/export.py:18:    subprocess.run(f"zip -j {out} {name}", shell=True)

Tool arguments come from the model, so they are user input with one more step in between. We read each tool's source with the client instead of sending query fragments or commands. The export line passes a model-chosen file name through a shell, so a space or quote in it changes the command; we record that from the source alone. In the test tenant we then ask for orders from a shop named ATARUS-MARKER'S SHOP, an ordinary name with an apostrophe; a database error instead of an empty result confirms the concatenation.

Negative result. Every query is parameterized and every command is an argument list with no shell, so the apostrophe returns an empty result.

Fix. Use bound parameters and argument arrays, validate each argument against the tool's schema and an allowlist, and run tools with an account that can read only what the tool needs.

Does output go into email as HTML?

grep -n 'ATARUS-MARKER' raw-message.eml

Expected output

48:<p>Summary: ATARUS-MARKER-<i>1028</i></p>   # <- the answer went into the HTML body unescaped

We ask the assistant to email tester-a's own address a summary containing the marker with italics, save the received message as raw source, and read the HTML part. An email body built from model output reaches other people's mail clients, with the client's domain as sender.

Negative result. The raw body shows &lt;i&gt;, or the email is plain text.

Fix. Build emails from a template that escapes every inserted value, and keep the sender, recipients and links out of the model's reach. The tool permission review decides whether the model should send email at all.

The defender sees

CSP violation reports for a blocked inline element or image, where reporting is set up. The image request appears in proxy or firewall logs as a fetch to an unknown host straight after an assistant answer. The apostrophe shows as a database syntax error in the application log with the tool's name on it. The assistant's own logs should hold the answer text for each of these turns; if they do not, the client cannot trace output that did harm back to the conversation that produced it.

Done when

Every sink is listed from the code and the client's description, the chat view and every other viewer of transcripts are tested with the marker, image loading is recorded, each tool that builds a query, command or message is classed as parameterized or concatenated, and email output is checked in the raw message.

Not covered

How an attacker gets text into the answer (see prompt injection testing), and cross-site scripting outside the AI feature (see reflected input checks); the outputs shown are representative examples on a placeholder app, not captured runs.