D-15 · API · IDEMPOTENCY
API timeouts and idempotency keys
Timeouts are ambiguous: the server may have queued your message while you saw failure. Idempotency keys turn retries into safe replays.
Symptom
During a provider slowdown, welcome emails duplicate: users receive two identical onboarding messages minutes apart, and receipts double-fire. Logs show client timeouts followed by automatic retries — each retry a fresh send because no idempotency key bound attempts to one logical message. Finance notices when duplicate receipts confuse customers; lifecycle metrics inflate when welcome variants double-count.
The mirror symptom is missing mail: fearing duplicates, the team disables retries entirely, and genuinely failed sends never recover.
Cause
Distributed sends have three outcomes — accepted, rejected, unknown — and timeouts land in “unknown.” Without a server-recognized deduplication key, the client cannot distinguish “never received, safe to resend” from “queued, resend duplicates.” Short client timeouts relative to provider P99 latency manufacture ambiguity: the request succeeds server-side after the client gives up. Failover across keys or regions without shared key state repeats the pattern at larger scale.
Vendor support varies: some APIs offer native idempotency headers with defined key lifetimes, others rely on caller-supplied message IDs or custom deduplication windows. Assuming uniformity guarantees duplicates somewhere.
Fix
Generate a deterministic idempotency key per logical send (for example, hash of recipient + template version + event ID) and transmit it via the vendor’s native mechanism; where none exists, enforce single-flight per key in your queue layer and record provider message IDs before acknowledging work. Set client timeouts above provider P99 with bounded retries (exponential backoff, jitter, dead-lettering), and make every worker crash-safe: attempt → record → acknowledge, never acknowledge before recording.
Reconcile after incidents by key: list attempted keys, query provider status per key, and replay only keys with no confirmed acceptance.
Prevention
Test timeout behavior in staging with injected latency, assert single delivery per key under retry storms, and monitor duplicate-rate proxies (same-key multi-accepts, complaint-tagged duplicates). Document each vendor’s key lifetime and scope, rotate key namespaces per environment to prevent staging keys colliding with production, and include idempotency verification in every vendor evaluation — it outranks minor price differences.
Worked example
A checkout service sends receipts with keys derived from order IDs, for example receipt-88412-v1. During a regional slowdown, the first attempt times out after 8 seconds although the provider queued the message. The worker retries with the identical key; the provider recognizes the replay and returns the original message ID instead of creating a second email. The customer gets exactly one receipt, and reconciliation shows one accepted key rather than two sends.
Contrast the unkeyed path: without the key, the retry becomes a new submission, the customer receives two receipts with different message IDs, and support must explain the duplicate charge email. Multiply that across a thousand checkouts and the incident becomes a trust event. Keys convert ambiguous timeouts into a verifiable single outcome, which is why every send path — receipts, resets, onboarding — should generate its key from stable business identifiers before the first attempt.