A payment webhook hits at 02:14 during deploy. App returns 503. Provider retries three times. By morning the customer has three active subscriptions and one invoice. Nobody implemented idempotence — only a naive INSERT.
Webhooks are not "nice HTTP callbacks". They are at-least-once deliveries with retries, unordered events, and sometimes minutes of delay.
Delivery model
Assume at-least-once: duplicates are normal. Sometimes non-FIFO ordering across event types. Handlers must be idempotent and tolerate logical reorder (cancelled before paid is rare but possible).
Signature and timestamp
Verify HMAC-SHA256 with rotatable secret. Reject if timestamp > 5 min (replay). Constant-time signature compare.
Log signature failures separately from app 500s — do not confuse attack and bug.
Idempotence in practice
| Approach | Pro | Con |
|---|---|---|
| event_id in DB | Simple | TTL cleanup |
| Idempotency-Key | API standard | Variable provider support |
| Business state (paid_at) | Robust | Race conditions |
Combine provider key + SQL unique constraint.
HTTP responses and async
Processing < 2 s: sync 200. Else 202 + queue. Worker uses event_id. Never heavy work before ack without idempotence.
Dead-letter queue after N business failures.
Observability
Metrics: handler latency, 2xx/4xx/5xx rates, queue lag, ignored duplicates. Trace event_id → job → invoice. Runbook: manual replay without double effect.
Production endpoint checklist
- [ ] HMAC signature verified constant-time
- [ ] Timestamp anti-replay (<5 min)
- [ ] event_id table unique constraint
- [ ] 2xx/202 response < provider timeout
- [ ] Async queue for processing >2 s
- [ ] Dead-letter after N business failures
- [ ] Structured logs event_id + latency
- [ ] Idempotence tested (3 identical POSTs)
- [ ] Documented provider IP firewall
- [ ] Manual replay runbook without double effect
- [ ] Planned webhook secret rotation
- [ ] GDPR: limited payload retention
Check before exposing public URL — "almost good" webhook costs more than two-day release delay.
Replay scenarios and incidents
Imagine a deploy restarting PHP-FPM for thirty seconds: the payment provider sends five spaced retries. Without an idempotence table, you credit the same payment five times. Support discovers the bug before app logs because the accounting dashboard aggregates duplicates faster than technical monitoring.
A legitimate replay differs from a malicious one: the first comes from the provider after timeout, the second from an attacker replaying an unsigned webhook URL (if you skipped HMAC). Signature binds body to secret; without it, replaying a captured POST suffices.
For tests, use provider replay tools (Stripe CLI, GitHub redelivery) in staging with disposable DB. Verify the third identical retry changes nothing. Automate in CI if the endpoint is critical.
On shared hosting, check PHP max_execution_time and nginx proxy timeout — a 504 gateway triggers provider retries even if your handler would have finished two seconds later. Align timeouts: nginx > PHP > provider webhook timeout, with margin.
Providers differ on signature headers (Stripe-Signature, X-Hub-Signature-256, etc.) — encapsulate verification per adapter, not one-time doc copy-paste.
Version webhook secret like passwords: rotation with dual active secret 24h if provider supports.
Load test: simulate 10× webhook burst — size queue depth and workers before Black Friday.
Keep a dated runbook, before/after metrics, post-incident review — cumulative discipline beats Friday night panic.
Keep a dated runbook, before/after metrics, post-incident review — cumulative discipline beats Friday night panic.
Keep a dated runbook, before/after metrics, post-incident review — cumulative discipline beats Friday night panic.
Keep a dated runbook, before/after metrics, post-incident review — cumulative discipline beats Friday night panic.
Keep a dated runbook, before/after metrics, post-incident review — cumulative discipline beats Friday night panic.
Keep a dated runbook, before/after metrics, post-incident review — cumulative discipline beats Friday night panic.
Shared timeouts
Align nginx > PHP > provider timeout — 504 triggers retries.
Decide and move forward without blind spots
- Verify HMAC signature constant-time with anti-replay timestamp under five minutes.
- Deduplicate by event_id — unique DB constraint, idempotence tested with three identical POSTs.
- Respond 2xx/202 quickly — async queue if processing exceeds two seconds; dead-letter after N business failures.
- Document provider IPs — firewall, webhook secret rotation, manual replay runbook without double effect.
- Limit payload retention — GDPR compliance on stored bodies.
To pick reliable API hosting (latency, SLA), use our comparison tool, directory, and observability guides.
Frequently asked questions
Why sign webhooks?
HMAC proves the body came from the provider and was not tampered with. Without it, anyone can POST to your endpoint.
Which HTTP codes for retries?
2xx acknowledges. 4xx non-retryable for permanent errors (bad signature). 5xx or timeout triggers retries — hence mandatory idempotence.
How to deduplicate?
Store event_id (or stable hash) with unique constraint; ignore duplicates already processed even if body differs slightly.
Emitter timeout?
Often 5–30 s. Respond fast (202 + async job) if processing is long, but then guarantee worker idempotence.
Before opening the endpoint in prod, simulate three identical POSTs — if the database changes three times, you are not ready.