<!-- Change the title to be: [Group Name] FCL for Incident #### --> ## Summary On 2023-May-12 an [incident](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/14260/ "2023-05-12: Container registry pulls failing with unknown blob") was created as a result of a [change in the codebase](https://gitlab.com/gitlab-org/container-registry/-/blob/v3.73.0-gitlab/registry/storage/linkedblobstore.go#L76). Due to this incident being an feature-change-locks~3760139 incident on GitLab.com caused by a change from the development department we will implement a Feature Change Lock. **The FCL will start on `2023-05-22` and will end on `2023-06-04`.** . ## Artifacts * **Daily Stand Up Entries:** [Async daily standup](https://gitlab.com/groups/gitlab-com/-/epics/2179#note_1400215353 "Package:Container Registry FCL for Incident 14260") in comments below * **Related Incident:** https://gitlab.com/gitlab-com/gl-infra/production/-/issues/14260+ * **Incident Review:** https://gitlab.com/gitlab-com/gl-infra/production/-/issues/14263+ * **Breaking Code Change:** `add link to the related breaking change (MR)` * **Slack Channel:** `#fcl-incident-14260` **Post FCL** * **Closing Ceremony Agenda**: `add link to the agenda` * **Closing Ceremony Video**: `add link to the video` * **FCL Retrospective**: `add link to the retrospective` ## Prepare for the FCL Once it has been determined that the incident will lead to an FCL, the Engineering Manager for the team who caused the incident will create an FCL Issue and complete the "**Prepare for the FCL"** section. * [x] 1. Complete the written Incident Review documentation within the Incident Issue as the first priority after the incident is resolved. * [x] 2. Complete the Summary section of this document * [x] 3. Review the [FCL handbook page](https://about.gitlab.com/handbook/engineering/#feature-change-locks) in case updates to the process have been made * [x] 4. Define the scope of the FCL by creating a work plan referencing all the Issues, Epics, and/or involved MRs must be created and used to identify the scope of work for the FCL. The work plan itself should be an Issue or Epic. * [x] 5. Create the FCL issue Title: FCL: Date - Incident Title and promote to an Epic * [x] 6. Create a public slack channel called #fcl-incident-\[number\], with members * [x] 7. Add the dates to the timeline below * [x] 8. Add Labels to this Issue * `FYXX/QX` * `FCL` * `Group` * `Category` * [x] 9. Identify the team members who will be involved in the FCL * @hswimelar * @jaime * @suleimiahmed * @jdrpereira * @adie.po * [x] 10. Inform stable counterparts (UX, Product, Quality, Security and Product Management) who will be impacted by a team FCL * @trizzi * @svistas * @greg ## FCL Timeline _Complete the date section in the timeline below, based on the timeline_ [_here_](https://about.gitlab.com/handbook/engineering/#timeline)_._ - **Day 0**: 2023-May-12 Incident: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/14260+ - **Business Day 0**: `2023-May-16` - Prepare and updates to the daily reliability standup agenda and provide status during the meeting - **Business Day 0**: `2023-May-16` - Confirmation that an FCL is required for this incident and start planning - **Business Day 3-4**: `2023-May-22` - Planning time - **Business Day 5-9**: `2023-June-1` - Complete Planned Work - **Business Day 10-11**: `2023-June-2` - Closing ceremony, retrospective and report back to standup ## Work plan Goals of the FCL: 1. Address corrective actions as listed below. 2. Improve the deployment process and develop an implementation plan for automated deployments. 3. Accelerate the maintainership trainee progress for team members who are not yet maintainers. _Please here to the guidance given_ [_here_](https://about.gitlab.com/handbook/engineering/#scope-of-work-during-fcl) _regarding the Scope of Work for an FCL handbook page . List of links to all work that is included in the scope of this FCL_ * [x] https://gitlab.com/gitlab-org/container-registry/-/issues/1015+ - @hswimelar * [x] https://gitlab.com/gitlab-org/container-registry/-/issues/1016+ - @jaime * [x] https://gitlab.com/gitlab-org/container-registry/-/issues/1017+ - @svistas * [x] https://gitlab.com/gitlab-org/container-registry/-/issues/1018+ - @suleimiahmed * [x] https://gitlab.com/gitlab-org/container-registry/-/issues/1019+ - @svistas * [x] https://gitlab.com/gitlab-org/container-registry/-/issues/1020+ - @jaime * [x] https://gitlab.com/gitlab-org/container-registry/-/issues/1021+ - @suleimiahmed * [x] https://gitlab.com/gitlab-org/container-registry/-/issues/1034+ - @jdrpereira * [x] https://gitlab.com/gitlab-org/container-registry/-/issues/1031+ - @hswimelar _Complete the date section in the timeline below, based on the timeline_ [_here_](https://about.gitlab.com/handbook/engineering/#timeline)_._