How to report a phishing simulation
Brief
A social engineering report is about controls, not people: each step of the pretext either met a control that held or found one that was missing, and the report says which, with numbers that never point at a person. The awareness team and the security team get different parts of it, and both get a retest that measures the same things again.
On this page
- Check first
- How do I report results without naming anyone?
- Which control failed at each step, and what fixes it?
- What does the detection timeline look like?
- What does the awareness team get, and what does the security team get?
- How do I plan the retest?
- The defender sees
- Done when
- Not covered
- Tools
- Related research
- Reviewed
- Situation
- A finished, approved simulation, either a phishing simulation or a help desk test, with its per-recipient record still held under the agreed retention and read access to the client's mail and click logs. Everything here works on that record and those logs for the campaign the scope letter names, and the per-recipient record is deleted once the report is accepted.
- Goal
- One report with aggregate results by team, a control chain with the failed control and its fix at each step, a detection timeline, separate sections for the awareness and security teams, and a retest plan.
- Outputs
- Representative examples on placeholder hosts, trimmed to the lines that matter. Not captured from a public fixture yet.
Check first
- The approval letter's statement that results are for control improvement only, which the report repeats on its first page.
- The per-recipient export, held only until the report is accepted, with names replaced by an id and a team.
- The SOC's ticket for the first report and the time each action was taken.
- Whether the simulation was delivered through an allow rule or a simulation override, which decides what the mail filter result means.
Steps
How do I report results without naming anyone?
awk -F, 'NR > 1 { s[$2]++; c[$2] += ($3 == "yes"); r[$2] += ($4 == "yes") }
END { for (t in s) { k = (s[t] < 10) ? "Other" : t; S[k] += s[t]; C[k] += c[t]; R[k] += r[t] }
for (k in S) printf "%s\t%d\t%d\t%d\n", k, S[k], C[k], R[k] }' results-pseudonymised.csv
Expected output
Finance 62 9 21
Engineering 212 19 71
Other 27 6 2 # <- three teams under ten recipients folded together
The export holds an id, a team, clicked and reported; names never leave the platform. Teams under ten are folded into Other so no row can be traced to a person. The report gives sent, clicked and reported per team and the report rate first, since reporting is what defends the organization.
Negative result. Every team already has ten or more recipients, so nothing needs folding; the rule still goes in the method section.
Fix. Set the minimum group size in the approval letter, so nobody asks for a per-person list after the fact.
Which control failed at each step, and what fixes it?
EmailEvents
| where Timestamp between (datetime(2026-10-14) .. datetime(2026-10-15)) and SenderMailFromDomain == "sim-training.example.net"
| summarize Messages = count() by DeliveryAction, OrgLevelPolicy
Expected output
DeliveryAction OrgLevelPolicy Messages
Delivered Phishing simulation 301 # <- delivered by the override, so the filter was not tested
Step Control Held? Fix
Delivery mail filter not tested separate filter test with no override
Click Safe Links, browser filter no block newly registered domains
Report report button, mailbox partly button in every client, SOC reads it
Response pull from all inboxes no automated removal on first SOC triage
Help desk reset callback to number on file no callback required in the ticket form
Each step of the pretext meets a control, and the report says which and whether it held. A delivery made through an override is reported as untested, not as a filter failure.
Negative result. Every control held or was outside scope, and the table says which.
Fix. One owner and one date per row, agreed with the client in the readout.
What does the detection timeline look like?
let sim = "sim-training.example.net";
UrlClickEvents
| where Timestamp between (datetime(2026-10-14) .. datetime(2026-10-15)) and Url has sim
| summarize FirstClick = min(Timestamp), LastClick = max(Timestamp), Clicks = count()
Expected output
FirstClick LastClick Clicks
2026-10-14 09:03:41 2026-10-14 16:52:10 34 # <- clicks went on for almost eight hours
09:00 first delivery 09:11 first report reaches the SOC mailbox
09:03 first click 10:40 SOC triages the report
16:52 last click none copies never pulled from inboxes
The timeline puts clicks beside the SOC's actions. A report at 09:11 that nobody acted on until 10:40 is the gap this campaign found, and it is a process gap.
Negative result. The SOC pulled remaining copies within minutes of the first report, and clicks stopped.
Fix. Triage user reports within fifteen minutes and remove matching messages from all inboxes on confirmation.
What does the awareness team get, and what does the security team get?
Awareness team: report rate and click rate by team, the cues in the pretext people missed,
and which teams reported fastest, to learn from.
Security team: the control table, the timeline, filter and override settings, the help desk
verification gap, and every log query used, so they can rerun it.
Neither: names, per-person results, or quotes from help desk calls.
The awareness team needs patterns to teach from; the security team needs controls to change. Neither needs a list of people, and the report says so.
Negative result. A request for named results goes back to the approval letter, which rules it out.
Fix. Write the distribution list into the approval, so the report goes only to the people who will act on it.
How do I plan the retest?
join -t "$(printf '\t')" round1-by-team.tsv round2-by-team.tsv | awk -F'\t' '{ printf "%s\t%.0f%%\t%.0f%%\n", $1, 100*$4/$2, 100*$7/$5 }'
Expected output
Finance 34% 52% # <- report rate up after the report button reached every mail client
Engineering 33% 41%
The retest uses a different pretext of the same difficulty, the same audience rules and the same measures, after the fixes are in, so the change is in the controls rather than in people remembering the last email. Both files must be sorted by team for join.
Negative result. No change in report rate or response time means the fixes did not reach the step they targeted.
Fix. Retest each fixed control on its own as well, such as a help desk callback check, rather than waiting for the next full campaign.
The defender sees
Nothing on this page touches users. The queries run in the client's own advanced hunting under our review account, and the report hands the security team those same queries so they can watch for the real campaign this one imitated.
Done when
No table row describes fewer than ten people, every step of the pretext names the control that applied and whether it held, the timeline runs from first delivery to the SOC's last action, each audience has its section, and the retest names the measures and the date.
Not covered
Running the simulation, disciplinary or HR use of results, which the report rules out in its first paragraph, and voice or SMS campaigns beyond the help desk; the outputs shown are representative examples on placeholder domains, not captured runs.