2025-12-03: At least 10% of chef-runs are failing in gprd environment, likely due to configuration error

At least 10% of chef-runs are failing in gprd environment, likely due to configuration error (Severity 4)

Problem: At least 10% of Chef client runs were failing in the gprd environment due to configuration errors with multiple cookbooks.

Impact: Chef runs failed on at least 10% of nodes in the gprd environment, disrupting configuration management tasks. One VM remained unable to converge until cookbook pinning was corrected.

Causes: Unpinned cookbook versions led to failures in the gitlab-wiz-sensor, packagecloud, and ufw cookbooks during Chef runs. Pinning versions and reverting changes resolved most issues. A single VM continued to experience failures until the correct cookbook pin was applied.

Response strategy: We pinned the versions of gitlab-wiz-sensor, packagecloud, and ufw cookbooks across all environments and reverted problematic changes. Additional fixes were applied to the last affected VM, restoring successful Chef runs.

---

This ticket was created to track INC-6079, by incident.io 🔥