2025-10-17: Manifest read requests on registry manifest endpoints violating apdex SLO

Manifest read requests on registry manifest endpoints violating apdex SLO (Severity 3 (Medium))

Problem: Read requests (GET or HEAD) to the manifest endpoints on the registry's CNY stage violated the Apdex SLO.

Impact: Performance for manifest read requests on the registry's CNY stage dropped, with Apdex at 99.39% over the past 6 hours. Only the CNY stage was affected; main stage users were not impacted. Service performance has now improved after the rollback.

Causes: The issue began shortly after upgrading the registry's CNY stage to version 4.29.0. The deployment at 16:03 UTC was followed by a drop in Apdex from 16:06 UTC. The main stage was unaffected.

Response strategy: We rolled back the registry's CNY stage to the previous version. The deployment is complete, and performance has returned to normal levels.

---

This ticket was created to track INC-4894, by incident.io 🔥