2021-02-18: NPM builds broken on gitlab.com

Note: In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in out handbook page. Any of this confidential data will be in a linked issue, only visible internally. By default, all information we can share, will be public, in accordance to our transparency value.

Summary

More information will be added as we investigate the issue.

Timeline

All times UTC.

View recent production deployment and configuration events (internal only)

2021-02-18

2021-02-18

Corrective Actions


Incident Review

Summary

For a period of 6 hours and 24 minutes, the instance level API of the npm package registry was disrupted and could under specific conditions not find the proper package.

By not finding the package, the request would then be redirected to npmjs.org due to the npm request forward that is enabled on gitlab.com. npmjs.org would likely reply that the package was not found 404.

This disruption affected all npm commands pulling packages from the npm packages registry such as $ npm install or $ yarn install. Those commands being used frequently in CI pipelines, those pipelines failed.

The source of this disruption was the work from gitlab-org/gitlab!53266 (merged). At the instance level, the backend has to "locate" the project out of a merely the package name. This locator has been updated to keep the same behavior at the instance level. This update did not take into account a specific set of conditions.

The MR fixing the bug was in maintainer review when the production incident was declared. It was decided to deliver it as a hotfix to production.

  1. Service(s) affected: The npm packages registry (ServiceAPI)
  2. Team attribution: The package team
  3. Time to detection: The bug was detected on staging before it was deployed on production
  4. Minutes downtime or degradation: 6 hours and 24 minutes

Interaction schemas

Expected behavior

For packages existing on gitlab.com (assuming that the user has the proper permissions to see those):

sequenceDiagram
    npm->>GitLab: Give me information about package `x`
    GitLab->>npm: Certainly, here is the information about package `x`

For packages not existing on gitlab.com (assuming that the package doesn't exist on npmjs.org):

sequenceDiagram
    npm->>GitLab: Give me information about package `x`
    GitLab->>npm: I don't have that package, but here is the url of `npmjs.org` (redirect response)
    npm->>npmjs.org: Give me information about package `x`
    npmjs.org->>npm: I don't know that package (not found response)

Behavior during the production incident

For some packages existing on gitlab.com under specific conditions (assuming that the user has the proper permissions to see those):

sequenceDiagram
    npm->>GitLab: Give me information about package `x`
    GitLab->>npm: I don't have that package, but here is the url of `npmjs.org` (redirect response)
    npm->>npmjs.org: Give me information about package `x`
    npmjs.org->>npm: I don't know that package (not found response)

For packages not existing on gitlab.com

sequenceDiagram
    npm->>GitLab: Give me information about package `x`
    GitLab->>npm: I don't have that package, but here is the url of `npmjs.org` (redirect response)
    npm->>npmjs.org: Give me information about package `x`
    npmjs.org->>npm: I don't know that package (not found response)

Metrics

Because the bug made the requests to be forwarded to npmjs.org, the only visible effect on metrics is the number of redirected responses answered by the npm packages regsitry:

Screenshot_2021-02-22_at_11.35.45

https://log.gprd.gitlab.net/goto/ff317bd86fd66c5070d70bd97f72756c

As we can see it is impossible to distinguish the valid redirect responses from the ones generated by the bugs. See the activity after the fix was deployed, it is really similar to what we have when the bug is present in production.

Customer Impact

  1. Who was impacted by this incident? (i.e. external customers, internal customers)
    1. Any customer pulling npm packages from the GitLab package registry under specific conditions. When the production incident was declared, a large external customer was having issues with its CI pipelines due to this.
  2. What was the customer experience during the incident? (i.e. preventing them from doing X, incorrect display of Y, ...)
    1. Prevent them to pull npm packages from the GitLab package registry.
  3. How many customers were affected?
    1. ...
  4. If a precise customer impact number is unknown, what is the estimated impact (number and ratio of failed requests, amount of traffic drop, ...)?
    1. ...

What were the root causes?

"5 Whys"

  1. Why was this not discovered in development or review?
    1. Because the bug only happens when a specific set of conditions are met: the scope of the package has to match the path of a subgroup created before the root namespace of the package. Those conditions were not tested.
  2. Why are those conditions not tested.
    1. They were missing from the original MR.
      • Those conditions were added to the MR with the fix and we opened gitlab-org/gitlab#322150 (closed) to expand the existing test conditions with package managers that do have an instance or a group level API.
  3. If the bug was detected on staging, why it has landed on production?
    1. Due to a lack of guidelines and because the bug was only happening under specific conditions, the MR DRI wrongly classified the severity as low. At that time, the path for moving forward was to create a fix and get it deployed as usual with auto deploy.
      • We opened gitlab-org/gitlab#322152 (closed) for adding guidelines when classifying bugs on the Package team. Among others, bug impacted the most used packages registry should have a higher severity.
  4. If it landed on production, why the monitoring tools didn't trigger any alarm?
    1. The bug will make the API reply a 301 instead of a 200. 301 is not considered as an erroneous response code. Moreover, some of these 301 are valid ones and some are from the bug

Incident Response Analysis

  1. How was the incident detected?
    1. When communicating the bug to EM/PM of the package team. It was realized that this could impact major external customers. Major external customers were also reporting issues while using the npm packages registry.
  2. How could detection time be improved?
    1. As we can see from the metrics above, we can't distinguish erroneous volumes of 301 responses from normal volumes. As such, we don't think there is any improvement we could do here
  3. How was the root cause diagnosed?
    1. On staging, see https://gitlab.com/gitlab-org/release/tasks/-/issues/2243#grouppackage-grouppackage.
      • Note that this was before the bug was deployed on production.
  4. How could time to diagnosis be improved?
    1. This was diagnosed on staging which is the very first environment on which changes are deployed. We don't think there is any improvement we could do here.
  5. How did we reach the point where we knew how to mitigate the impact?
    1. From the bug analysis on staging.
  6. How could time to mitigation be improved?
    1. By having better guidelines when estimating the impact of a bug found on staging. See gitlab-org/gitlab#322152 (closed).
    2. Engineers should have better knowledge on how to properly create a hotfix (see #3661 (comment 512274778))
  7. What went well?
    1. Detecting the bug on staging played a part in having the fix already ready by the time the production incident was declared. Without it, the bug would have been detected on production and it's root cause analysis would have take some time while the bug was affecting users.

Post Incident Analysis

  1. Did we have other events in the past with the same root cause?
    1. Given that the root cause is from a refactoring of an existing logic, we don't have any other events.
  2. Do we have existing backlog items that would've prevented or greatly reduced the impact of this incident?
    1. No
  3. Was this incident triggered by a change (deployment of code or change to infrastructure)? If yes, link the issue.
    1. gitlab-org/gitlab!53266 (merged)

Lessons Learned

  1. Backend tests on package manager APIs that support the Group or Instance level should have better conditions coverage: gitlab-org/gitlab#322150 (closed)
  2. Have a guideline to measure the impact of a bug found on staging and how to react to it: gitlab-org/gitlab#322152 (closed)
  3. We need engineer training for hot patching process. See #3661 (comment 512274868).

Guidelines

Resources

  1. If the Situation Zoom room was utilised, recording will be automatically uploaded to Incident room Google Drive folder (private)

Incident Review Stakeholders

Edited by John Jarvis