Sample incident report — INC-2026-0173
Every Critical or High incident on a managed workflow ships a report in this format within 24 hours of resolution. Customer name, workflow specifics, and vendor names have been anonymized. Numbers are plausible for the workflow class.
Metadata
| Incident ID | INC-2026-0173 |
| Customer | [REDACTED] · ~35-person professional services firm |
| Workflow | Weekly client status digest generator |
| Severity | High |
| Silent-failure mode | SFM-1 (executes "successfully" but produces empty output) |
| Detection at | 2026-06-14 09:47 PT (18 minutes after failure onset) |
| Response at | 2026-06-14 09:52 PT (5 minutes after detection) |
| Contained at | 2026-06-14 10:41 PT (49 minutes after detection) |
| Resolved at | 2026-06-14 15:12 PT (5h 25min after detection) |
Symptom
Weekly client digest emails delivered on schedule, addressed correctly, subject lines correctly personalized — but the "highlights of the week" section rendered as an empty bullet list. Three digests went to three clients before the output-shape validator caught the empty section and paged on-call.
Blast radius
Three client-facing emails delivered with degraded content. Fifteen additional digests queued for the same day were held before send. Zero client-visible actions (no missed replies, no confused replies received). Two of the three clients acknowledged the digest in follow-up threads without noting the empty section; one flagged it, was replied to within an hour with the corrected version + brief explanation, and continued the engagement without escalation.
Root cause
A third-party CRM API deprecated the field name the workflow used to pull "activities logged this week" and started returning that field as null. The workflow's data-fetch step succeeded (200 OK, valid JSON structure) but the downstream template-fill step rendered an empty list rather than raising on the null. The workflow's own output-shape validation ran only on the delivery envelope, not on the section-body content, so the empty section shipped.
Remediation
- Immediate (09:52–10:41 PT): queued digests were held; the three sent digests were flagged for follow-up; a manually assembled corrected digest was sent to the client who flagged.
- Same-day (12:00–15:12 PT): the workflow's data-fetch step was updated to use the new API field name; a section-content validator was added upstream of the template-fill step that raises if any required section is empty; the workflow was tested against three historical data windows and re-enabled.
- Follow-up (next 48 hours): section-content validators were retrofitted onto the four other digest-generation workflows for the same customer that used the same API pattern. None had failed yet — pre-emptive.
Prevention
The pre-existing output-shape validator only checked the delivery envelope (email address present, subject non-empty, HTML body parses). It did not check that each promised section had content. That gap is now closed for this workflow and its four peers.
Structural fix: the workflow-authoring template in Phronimos's internal library now requires a section-content validator any time a workflow's output has named sections. This closes the same-shape gap for future workflows before it opens.
SLA impact
Detection at 18 minutes exceeded the High-severity target of 2 hours: no breach. Response at 5 minutes was inside the 4-business-hour target: no breach. No SLA credit applied.
What the client saw
A 6-line notification email within 30 minutes of detection ("Digest for [client] shipped with an empty highlights section — we've held the rest of today's digests and are pulling the corrected version now — will confirm within 2 hours"). A 30-minute follow-up call to walk through this same report the next morning. This incident report itself, delivered as PDF within 24 hours of resolution.