D-20 · OBSERVABILITY · METRICS
Deliverability dashboard: metrics that matter
Opens mislead; bounces, complaints, deferrals, and engagement cohorts predict. This scenario defines the minimal dashboard worth checking daily.
Symptom
A team watches opens and clicks exclusively. Placement degrades for a month before anyone notices, because privacy-prefetch inflates opens while spam-foldering hides in the denominator nobody examines. By the time revenue-per-send visibly drops, the domain reputation has been mediocre for weeks and recovery takes quarters. The dashboard answered “did they engage?” while nobody asked “did we deliver?”
Vanity segmentation hides it further: blended rates across transactional and bulk mask a failing bulk stream behind healthy transactional engagement.
Cause
Deliverability signals live upstream of engagement: acceptance vs deferral vs bounce at the gateway, complaint and trap feedback from providers, authentication alignment rates, and per-stream, per-cohort engagement that reveals list rot. Opens conflate prefetch, images-off clients, and dark-mode proxies into noise; clicks skew toward the already-engaged. Without per-stream splits (transactional vs lifecycle vs bulk), per-provider splits (Gmail vs Yahoo vs Microsoft), and cohort splits (recent vs dormant), aggregates lie by construction.
Alert absence is the process cause: nobody pages on deferral spikes or complaint creep, so signals accumulate unread until users report missing mail.
Fix
Build the minimal board: per stream, per provider — delivery/acceptance rate, hard and soft bounce rates, complaint rate, deferral rate and reasons, authentication pass rates, unsubscribe rate, and engagement per cohort (30/90/365-day). Add provider telemetry: Postmaster domain reputation and spam rate, Microsoft SNDS-style signals where available, and seed smoke results. Set alert thresholds: transactional hard bounces above 2%, bulk complaints above 0.1% (freeze at 0.3%), deferrals above 5%, authentication passes below 99%.
Review daily during warmups and launches, weekly otherwise, with stream owners named per row so every red cell has a human.
Prevention
Version the dashboard definition alongside the classification table: metric queries in source control, threshold changes reviewed like code. Correlate deploys (template changes, vendor migrations, list imports) with signal overlays so causes surface in days not months. Quarterly, audit which metrics actually preceded incidents and prune the rest — dashboards rot into wallpaper without curation. Treat deliverability review as reliability review: same cadence, same seriousness, same blameless post-mortems.
Worked example
A team ships a template redesign and watches opens hold steady at 38% — apparently safe. The deliverability board tells another story: Gmail deferrals triple, bulk complaint rates creep from 0.05% to 0.18%, and authentication passes dip to 97% because the new template pulls images from an unauthenticated CDN domain. Opens stay flat for two weeks on prefetch noise while reputation decays underneath.
Because deferral alerts page the on-call engineer on day two, the team reverts image hosting to the aligned domain, complaints plateau, and Postmaster reputation never drops below Medium. The post-mortem adds image-domain allowlists to CI. Without the board, the same incident would have surfaced a month later as missing trial emails — a far more expensive discovery channel.