2025-12-05: At least 10% of chef-runs failing in gprd environment, likely due to configuration error

At least 10% of chef-runs failing in gprd environment, likely due to configuration error (Severity 4)

Problem: At least 10% of Chef runs were failing on ci-runner nodes in the gprd environment.

Impact: Configuration management was disrupted for ci-runner nodes in the gprd environment, with at least 10% of Chef runs failing. No other node types have been confirmed as affected.

Causes: A configuration error related to the firewall cookbook version was contributing to Chef run failures.

Response strategy: We pinned the firewall cookbook to version 2.5.4 in the ci-prd environment to address the configuration error. A follow-up change is open to extend cookbook pinning to additional environments if needed. We are monitoring dashboards to confirm if failures are reduced.

---

This ticket was created to track INC-6107, by incident.io 🔥