2021-02-18: release-tools cannot create protected tags

Note: In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in out handbook page. Any of this confidential data will be in a linked issue, only visible internally. By default, all information we can share, will be public, in accordance to our transparency value.

Summary

release-tools is not able to create protected tags on gitlab-foss project, this blocks the monthly release preperation

📦 GitLab release

❌ The section "GitLab release" produced an error:

2021-02-18 10:54:55.620824 F Rake::Task -- Task failed -- Exception: Gitlab::Error::BadRequest: Server responded with code 400, message: You are not allowed to create this tag as it is protected.. Request URI: https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab-foss/repository/tags

Timeline

All times UTC.

2021-02-18

• 10:54 - 13.9.0-RC42 tagging failure
• 11:20 - @nolith declares incident in Slack.
• 11:25 - @nick.thomas identified that we are affected by gitlab-org/gitlab#321658 (closed) and gitlab-org/gitlab#321564 (closed)
• 11:25 - @nolith allows release-tools-bot to push in the * protected-branch rule to continue RC42 tagging, this mitigates the incident

Corrective Actions

• delivery#1576

---

Click to expand or collapse the Incident Review section.

Incident Review

---

Summary

1. Service(s) affected: ServiceGitLab Rails
2. Team attribution: groupsource code
3. Time to detection: 3 days since the first user report gitlab-org/gitlab#321564 (closed)
4. Minutes downtime or degradation: the regression lasted for about 3 days. It delayed the monthly release process by 30 minutes

Metrics

Customer Impact

1. Who was impacted by this incident? (i.e. external customers, internal customers)
   1. external users with multiple protected tags rules matching the same tag
   2. internal customer: teamDelivery
2. What was the customer experience during the incident? (i.e. preventing them from doing X, incorrect display of Y, ...)
   1. preventing them to create protected tags
3. How many customers were affected?
   1. N/A
4. If a precise customer impact number is unknown, what is the estimated impact (number and ratio of failed requests, amount of traffic drop, ...)?
   1. ...

What were the root causes?

"5 Whys"

Incident Response Analysis

1. How was the incident detected?
   1. ...
2. How could detection time be improved?
   1. ...
3. How was the root cause diagnosed?
   1. ...
4. How could time to diagnosis be improved?
   1. ...
5. How did we reach the point where we knew how to mitigate the impact?
   1. ...
6. How could time to mitigation be improved?
   1. ...
7. What went well?
   1. ...

Post Incident Analysis

1. Did we have other events in the past with the same root cause?
   1. ...
2. Do we have existing backlog items that would've prevented or greatly reduced the impact of this incident?
   1. ...
3. Was this incident triggered by a change (deployment of code or change to infrastructure)? If yes, link the issue.
   1. ...

Lessons Learned

1. There is an high pressure on release managers by the time they have to tag RC42. We don't exercise tagging pipelines that oftens, and many things could change between a failure and the previuous successfull run of the same pipeline - (see also #3652 (closed) as an example)

Guidelines

• Blameless RCA Guideline

Resources

1. If the Situation Zoom room was utilised, recording will be automatically uploaded to Incident room Google Drive folder (private)

Incident Review Stakeholders