The Problem
Most production systems that integrate third-party webhooks — Stripe, GitHub, payment processors — are brittle. They lose events during Redis outages, don't verify signatures properly, and retry storms bring down downstream endpoints when they should back off gracefully.
The standard advice is "just use a queue." But a queue backed by Redis alone means your events vanish if Redis restarts. For payment webhooks, "lost event" means "lost money."
What I Built
A source-agnostic webhook delivery engine that applies write-ahead persistence at the application layer. Every inbound event is committed to PostgreSQL before being enqueued to Redis, so events survive queue failures. Outbound delivery is HMAC-signed for authenticity, retries use exponential backoff, and permanently failing endpoints get routed to a dead letter queue. BullMQ workers handle concurrency without blocking the main API thread.
The Decisions
PostgreSQL-first persistence
Chose to write to disk before enqueuing to Redis. Tradeoff: slight latency increase on ingest (~2-5ms). Gain: zero data loss, even during total queue failure. For webhook delivery, durability beats speed.
HMAC-SHA256 on outbound
Every delivery is signed so receiving endpoints can verify authenticity. Unsigned webhooks are a security hole — this isn't optional, it's table stakes.
Exponential backoff + dead letter queue
Instead of hammering failing endpoints in tight retry loops, Conduit backs off geometrically and eventually parks permanently failing deliveries. Prevents retry storms from cascading.
BullMQ worker isolation
Queue workers run independently from the Express API thread. Clean separation of ingest and delivery — the API stays responsive even under heavy queue load.
Verified Against Production
End-to-end validated against real Stripe webhooks in sandbox — signature verification, event storage, queue processing, and outbound HMAC-signed delivery all confirmed.
"The pattern transfers directly to wallet transaction routing, payment integrations, and any system where 'events must not be lost' is a hard requirement."
What It Can't Do Yet
- No multi-tenant isolation (single-org deployment only)
- No admin dashboard for delivery monitoring (CLI/logs only)
- No webhook replay from UI (must re-enqueue manually)
- Rate limiting per destination endpoint not yet configurable
These are real limitations. They're also the next build targets.