2021-09-16 - Users with private commit emails cannot create issues or MRs assigned to themselves in Canary
Current Status
We have drained canary, and reverted MR gitlab-org/gitlab!70473 (merged) which has fixed the bug.
Users who enabled private commit emails in their profile settings were impacted by this if they were using the Canary environment. For those users they were unable to create Issues or MRs assigned to themselves.
More information will be added as we investigate the issue.
Timeline
View recent production deployment and configuration events / gcp events (internal only)
All times UTC.
2021-09-16
-
03:46
- @engwan declares incident in Slack. -
04:02
- @cmiskell drains Canary; users are no longer affected. -
04:03
- @engwan prepares MR (gitlab-org/gitlab!70473 (merged)) to revert change.
Corrective Actions
Corrective actions should be put here as soon as an incident is mitigated, ensure that all corrective actions mentioned in the notes below are included.
- ...
Note: In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in out handbook page. Any of this confidential data will be in a linked issue, only visible internally. By default, all information we can share, will be public, in accordance to our transparency value.
Click to expand or collapse the Incident Review section.
Incident Review
-
Ensure that the exec summary is completed at the top of the incident issue, the timeline is updated and relevant graphs are included in the summary -
If there are any corrective action items mentioned in the notes on the incident, ensure they are listed in the "Corrective Action" section -
Fill out relevant sections below or link to the meeting review notes that cover these topics
Customer Impact
-
Who was impacted by this incident? (i.e. external customers, internal customers)
- Users who enabled private commit emails in their profile settings.
-
What was the customer experience during the incident? (i.e. preventing them from doing X, incorrect display of Y, ...)
- Creating an issue / MR that was assigned to themselves resulted in a "Assignees is invalid" error message.
-
How many customers were affected?
- Based on https://log.gprd.gitlab.net/goto/3b53fc70e582d454f2ed766b03592549, there are 59 unique users. (Note that this query filters by status 200 because a successful creation here results in a 302, while a validation error results in a 200)
What were the root causes?
- Code change from gitlab-org/gitlab!69648 (merged)
Incident Response Analysis
-
How was the incident detected?
-
@marcel.amirault reported the issue in Slack
#is-this-known
.
-
@marcel.amirault reported the issue in Slack
-
How could detection time be improved?
- ...
-
How was the root cause diagnosed?
- ...
-
How could time to diagnosis be improved?
- ...
-
How did we reach the point where we knew how to mitigate the impact?
- When we could replicate the validate failure locally and found the MR to revert.
-
How could time to mitigation be improved?
- ...
-
What went well?
- ...
Post Incident Analysis
-
Did we have other events in the past with the same root cause?
- Not exactly the same but very similar. In #5473 (closed), a change to the
User
validations prevented some users from signing in.
- Not exactly the same but very similar. In #5473 (closed), a change to the
-
Do we have existing backlog items that would've prevented or greatly reduced the impact of this incident?
- ...
-
Was this incident triggered by a change (deployment of code or change to infrastructure)? If yes, link the issue.
- Code change. See summary in gitlab-org/gitlab!69648 (comment 679084943)
Lessons Learned
- ...
Guidelines
Resources
- If the Situation Zoom room was utilised, recording will be automatically uploaded to Incident room Google Drive folder (private)