Skip to content

Add error budget exception for Workspace group

Gosia Ksionek requested to merge mk-add-error-budget-exceptionfor-workspace into master

Why is this change being made?

I would like to request the error budget exception for Manage::Workspace group.

For the last 2 months, our error budget was week to week between 99.84% and 99.91%. The biggest offenders are the API endpoints responsible for listing projects (api/:version/projects and its variations, like /groups/:id/projects or user/:id/projects).

We have noticed several problems with those endpoints:

  • Projects endpoints suffer from payload bloat: when a user is listing projects, each project has around 80 (87-96) attributes. Many of those attributes are connected to the individual project’s settings; there are also nested resources included, like compliance frameworks.

  • With a high number of different options available (https://docs.gitlab.com/ee/api/projects.html#list-all-projects) optimizing sql queries gets complicated. For example, just adding the limit parameter to one query changes the db query times significantly - from 13 ms to 8 minutes. Finding all permutations with the number of parameters allowed in this endpoint - it’s definitely a lot of work.

  • Another problem is that some part of response time from those endpoints are gitaly times.

  • already there were several iterations to address some of the issues. But with number of permutations with parameters, options that those endpoints are accepting, all my points above: it was not possible for the team to make a significant dent in the number of slow responses in the last quarter.

I would like to point out as well that the Manage::Workspace group is covering 244 endpoints.

All efforts around this is centralized in this epic: gitlab-org&8164 and despite many comments, participants and 12 merged MRs (please see here) - this endpoint is still slow for many users.

The exception would allow us to address this problem and also collaborate more with the API vision working group on how to treat those endpoints that were developed by multiple teams and over time begun to be hard to maintain.

Author Checklist

  • Provided a concise title for this Merge Request (MR)
  • Added a description to this MR explaining the reasons for the proposed change, per say why, not just what
    • Copy/paste the Slack conversation to document it for later, or upload screenshots. Verify that no confidential data is added.
  • Assign reviewers for this MR to the correct Directly Responsible Individual/s (DRI)
    • If the DRI for the page/s being updated isn’t immediately clear, then assign it to one of the people listed in the Maintained by section on the page being edited
    • If your manager does not have merge rights, please ask someone to merge it AFTER it has been approved by your manager in #mr-buddies
  • If the changes affect team members, or warrant an announcement in another way, please consider posting an update in #whats-happening-at-gitlab linking to this MR
    • If this is a change that directly impacts the majority of global team members, it should be a candidate for #company-fyi. Please work with internal communications and check the handbook for examples.

Edited by Gosia Ksionek

Merge request reports