2025-10-08: PubSub messages queuing on pubsub-rails-inf-gprd-sub causing Elastic log data lag

PubSub messages queuing on pubsub-rails-inf-gprd-sub causing Elastic log data lag (Severity 4 (Low))

Problem: A backlog of PubSub messages queued in the pubsub-rails-inf-gprd-sub subscription, causing delays in processing Rails log data in Elastic.

Impact: Rails log data in Elastic was delayed by up to 1.9 hours. Other log types were not affected. The backlog has now cleared and log data is processing in real time.

Causes: High CPU usage and search thread pool backups across several Elastic nodes caused log processing delays. An Elastic bug related to inefficient ES|QL queries on timeseries documents contributed to the slowness. Specific fields in log queries, like source.geo.location, caused slow field loading. A case has been opened with Elastic for vendor support.

Response strategy: We performed a zone-by-zone reboot of the Elastic cluster, which has completed. CPU usage is now more balanced across the hosts. We disabled ES|QL in Kibana to prevent recurrence, following Elastic's recommendations. The backlog has fully cleared. Elastic support will continue investigating slow query issues.

---

This ticket was created to track INC-4611, by incident.io 🔥