2025-10-06: PubSub messages queuing on pubsub-rails-inf-gprd-sub causing Elastic log lag

PubSub messages queuing on pubsub-rails-inf-gprd-sub causing Elastic log lag (Severity 4 (Low))

Problem: A backlog of unacknowledged messages accumulated in the pubsub-rails-inf-gprd-sub subscription, delaying log data processing in Elasticsearch.

Impact: At peak, roughly 2 hours of rails log messages had accumulated as a backlog in the pubsub-rails-inf-gprd-sub pubsub topic. This delay in ingesting logs into Elasticsearch caused a temporary gap in observability.

Causes: A newly introduced pubsub topic added extra ingestion workload, causing Elasticsearch to become CPU starved. This reduced the ingestion rate for all other topics and led to a backlog in the pubsub queue.

Response strategy: We have removed the experimental log ingestion source to reduce Elasticsearch workload. The backlog has cleared, and new log events are again visible promptly in Elasticsearch.

---

This ticket was created to track INC-4527, by incident.io 🔥