High-throughput webhook delivery engine in Go
Hookvane
A distributed webhook delivery platform in Go: transactional outbox and NATS JetStream for at-least-once delivery, per-endpoint circuit breakers, SSRF protection and signed webhooks.
My role: Lead backend systems and infrastructure engineer
- Go
- NATS JetStream
- PostgreSQL
- gRPC / Connect
- HTMX
Overview
SaaS platforms often degrade or fail in cascades when they send webhooks to slow or unreliable customer endpoints. Hookvane decouples ingestion from delivery using the Transactional Outbox Pattern and NATS JetStream work queues. It guarantees at-least-once delivery, isolates failing endpoints with circuit breakers and rate limiters, blocks Server-Side Request Forgery at the TCP socket layer, and signs every webhook to the Standard Webhooks specification.
The challenge
Almost every SaaS product sends webhooks, and most send them from the same process that handles user requests. When a customer's endpoint is slow or down, retries pile up in memory, timers are lost on restart, other customers' deliveries slow down, and a malicious URL can point the sender at internal services.
Hookvane set out to make webhook delivery a dependable, isolated service: never lose an accepted event, retry politely, protect both the sender and the receivers, and give operators a clear view of what is happening.
How it works
- 1Producer (gRPC / JSON)
- 2API + outbox (one tx)
- 3Relay (SKIP LOCKED)
- 4NATS JetStream
- 5Dispatcher (breaker, limits, SSRF guard)
- 6Signed webhook delivered
Architecture
Producers call hookvane-api over Connect-Go, which serves gRPC and HTTP/JSON from one Protobuf definition. In a single PostgreSQL transaction the API records the message (ULID keys, partitioned), one delivery per subscribed endpoint and an outbox row, then returns 202 Accepted; idempotency keys suppress duplicate producer retries for 24 hours. hookvane-relay polls the outbox with FOR UPDATE SKIP LOCKED in batches of 500 and publishes to a NATS JetStream work-queue stream, using the delivery ID as the message ID for deduplication.
hookvane-dispatcher consumes with a durable pull consumer. Each endpoint has a token-bucket rate limiter, an optional FIFO lane lock and a circuit breaker that opens after 10 consecutive failures, sends one half-open probe every 5 minutes and disables the endpoint after 72 hours of failure. Failed attempts are retried on a 9-step schedule from 5 seconds to 10 hours with ±20% jitter, honouring Retry-After, through JetStream NakWithDelay; exhausted deliveries become dead letters that can be replayed.
Every request is signed with Standard Webhooks headers using HMAC-SHA256, and endpoint secrets are encrypted at rest with AES-256-GCM. The HTTP client uses a custom net.Dialer.Control hook that checks the resolved IP before connecting and refuses loopback, private, link-local, multicast and cloud-metadata addresses, with redirects disabled. An HTMX and Tailwind dashboard compiled into the binary shows live delivery history and replay controls, Prometheus exposes ingest, delivery, latency and open-breaker metrics, and Go and TypeScript SDKs wrap the API. Data access uses sqlc and Goose migrations, and the services ship as distroless Docker images.
What I built
Zero-loss ingest with an outbox
Connect-Go serves gRPC and HTTP/JSON from one Protobuf spec; the message, deliveries and outbox rows are written in one transaction and the API returns 202 at once, with 24-hour idempotency keys.
NATS JetStream queueing
A relay pulls the outbox with FOR UPDATE SKIP LOCKED in batches of 500 into a work-queue stream with message-ID deduplication; retries use NakWithDelay, so no in-memory timers are lost on restart.
Backoff that respects receivers
A 9-step schedule from 5 seconds to 10 hours (about 21 hours total) with ±20% jitter, honouring Retry-After headers, then dead letters that can be replayed with one click.
Circuit breakers and lanes
Endpoints trip open after 10 consecutive failures, get a half-open probe every 5 minutes and are disabled after 72 hours; token-bucket rate limits and optional FIFO lanes preserve ordering.
SSRF-safe and signed
A custom dialer checks the resolved IP before the TCP handshake and blocks loopback, private, link-local, multicast and cloud-metadata ranges; webhooks are HMAC-SHA256 signed and secrets encrypted with AES-256-GCM.
Dashboard, metrics and SDKs
An HTMX + Tailwind control center embedded in the binary with live delivery history and replay, Prometheus metrics, and client SDKs in Go and TypeScript.
Engineering decisions
- Write the event and its outbox row in the same transaction, so nothing is lost even if the queue is down.
- Use JetStream delayed redelivery for retries instead of in-memory timers, so restarts never drop scheduled attempts.
- Enforce SSRF protection at the socket level after DNS resolution, because URL validation alone can be bypassed by DNS tricks and redirects.
- Isolate every endpoint with its own breaker and rate limit, so one failing customer never slows delivery to everyone else.
Results
- At-least-once delivery with no accepted event lost, thanks to the transactional outbox.
- Retries survive restarts, with no in-memory timers.
- One failing endpoint is isolated by its own breaker and rate limit.
- SSRF blocked at the socket level, including cloud metadata endpoints.
Stack
- Language
- Go 1.24+
- APIs
- Connect-Go (gRPC / Protobuf and HTTP/JSON), Buf codegen
- Data
- PostgreSQL (Goose migrations, sqlc type-safe queries)
- Queueing
- NATS JetStream (work queue, deduplication, delayed redelivery)
- Dashboard & observability
- HTMX, Tailwind CSS, Prometheus metrics
- Security & delivery
- Standard Webhooks HMAC-SHA256, AES-256-GCM, SSRF-safe dialer, Docker distroless build, Go and TypeScript SDKs
Related services
Need something like this?
I can build a version of this for your product, your data and your stack.
Start a project