cloudflare-log-collector

Firewall Events HTTP Traffic Stats Prometheus Metrics Loki Log Streams OpenTelemetry Tracing

README Architecture Go API Grafana GitHub

Cloudflare analytics for your self-hosted stack

A lightweight Go service that polls the Cloudflare GraphQL Analytics API for firewall events and HTTP traffic statistics, ships them into a self-hosted observability stack, and traces every poll cycle with OpenTelemetry.

Firewall events are pushed to Loki as structured JSON log lines for querying in Grafana
HTTP traffic stats are exposed as Prometheus gauges and also pushed to Loki for raw detail
Every poll cycle gets its own OpenTelemetry trace with child spans, exported to Tempo via OTLP gRPC
Log-trace correlation is automatic — trace_id and span_id are injected into every structured log line

Key Features

Firewall Event Collection

Polls Cloudflare's firewallEventsAdaptive dataset for WAF events with full request detail.

Captures action, client IP, host, method, path, query, ray ID, rule ID, source, user agent, and country. Each event becomes a structured JSON log line in Loki.

HTTP Traffic Statistics

Aggregated request counts grouped by method, status code, and country.

Polls httpRequestsAdaptiveGroups for traffic aggregates. Data is exposed as Prometheus gauges for dashboarding and also pushed to Loki for raw queryability.

Prometheus Metrics

Rich metrics covering poll health, firewall events, HTTP traffic, and Loki push status.

Exposes poll counters and histograms, firewall event counts by action, HTTP request gauges by method/status/country, Loki push success/failure rates, and build info.

Loki Integration

Pushes structured JSON log streams directly to Loki's push API.

Two log streams: firewall events and HTTP traffic, each with distinct labels. Supports multi-tenant Loki via configurable X-Scope-OrgID header. Automatic batching and retry with exponential backoff.

OpenTelemetry Tracing

Every poll cycle is traced end-to-end with child spans for API calls and Loki pushes.

Exports traces to Tempo via OTLP gRPC with configurable sampling rate. Each trace captures the full poll lifecycle: Cloudflare API query, data transformation, Loki push, and metric updates.

Log-Trace Correlation

Automatic trace_id and span_id injection into every structured log line.

A custom slog handler injects OpenTelemetry context into all JSON log output. Enables one-click navigation between Loki logs and Tempo traces in Grafana.