feat(api): implement HyperLogLog blast radius tracking for failing webhooks

[sodiqscript111]

What does this PR do?
This PR introduces a highly efficient "Blast Radius" metric to track the number of unique customer endpoints currently experiencing webhook delivery failures. It uses Redis's HyperLogLog (PFADD / PFCOUNT) data structure to calculate set cardinality in O(1) time using a maximum of 12KB of memory.

Use Cases & How it Helps

1. Incident Triage: During a massive outage, support teams can instantly see if 1,000 deliveries failing is affecting 1 customer (a single misconfigured endpoint) or 1,000 customers (a systemic Convoy issue).

2. System Health Dashboarding: The new API endpoint makes it trivial to plot "Unique Failing Endpoints" on a Grafana or UI dashboard to visually monitor the blast radius of network issues in real-time.

3. Alerting: Teams can set up alerts to trigger only when the blast radius exceeds a certain threshold, eliminating noise from a single customer taking down their own server and causing hundreds of delivery failures.

The Implementation

• Zero-Overhead Write Path: Injected the Redis client into the ProcessEventDelivery worker dependencies. When an HTTP delivery returns a non-2xx status code or encounters a network failure, we immediately execute a PFADD to record the failing EndpointID.

• Pipelining for Performance: The PFADD and Expire commands are combined in a single Redis Pipeline to prevent paying for two network round-trips per delivery failure.

• Centralized Key Generation: Created a new shared helper datastore.BlastRadiusKey() which pins the time buckets to UTC to prevent drift across server environments.

• New Analytics API: Added GET /api/v1/projects/{projectID}/dashboard/blast-radius which executes a PFCOUNT on the daily bucket and returns the unique count in JSON.

• Customizable Retention: Added BlastRadiusRetentionHours to the Analytics configuration. Users can now override the default 24-hour retention period using the CONVOY_ANALYTICS_BLAST_RADIUS_RETENTION_HOURS environment variable.

TODO

• UI once feature is approvesd

---

Note

Medium Risk
Adds Redis writes on the delivery failure hot path (pipelined, errors logged only) and makes the metric depend on Redis being configured for reads.

Overview
Adds a blast radius metric: how many distinct endpoints had at least one failed webhook delivery in the current UTC day bucket.

On non-2xx delivery outcomes, the event-delivery and retry-delivery workers PFADD the endpoint ID into a per-project daily Redis HyperLogLog (pipelined with EXPIRE). TTL comes from CONVOY_ANALYTICS_BLAST_RADIUS_RETENTION_HOURS (default 24h, minimum 24h on keys). datastore.BlastRadiusKey centralizes UTC day bucketing.

The control-plane UI exposes GET …/dashboard/blast-radius, which PFCOUNTs today’s bucket and returns blast_radius (503 if Redis is unavailable). Redis and retention are wired through internal/dataplane/worker into shared EventDeliveryProcessorDeps used by both delivery processors.

^{Reviewed by Cursor Bugbot for commit ed4ab8f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ed4ab8f. Configure here.}

[cursor]

Wrong Redis client wiring

High Severity

EventDeliveryProcessorDeps.Redis is set from opts.Redis.Client(), but opts.Redis is already a redis.UniversalClient and has no Client() method. Nearby deps correctly use rd.Client(), so blast-radius tracking is wired against the wrong Redis source and cannot initialize as intended.

 

^{Reviewed by Cursor Bugbot for commit ed4ab8f. Configure here.}

[cursor]

Retention below 24h ignored

Medium Severity

BlastRadiusRetentionHours is advertised as overriding the default 24-hour retention, but the write path clamps any TTL under 24 hours back to 24 hours. Values from CONVOY_ANALYTICS_BLAST_RADIUS_RETENTION_HOURS below 24 are therefore silently ignored.

Additional Locations (2)

• worker/task/process_retry_event_delivery.go#L357-L361
• config/config.go#L409-L410

 

^{Reviewed by Cursor Bugbot for commit ed4ab8f. Configure here.}