502, 503, and 504 when using Projects API

We have been seeing sporadic timeouts (502), connection errors (504), and service unavailable (503) codes for 2 sample projects throughout the past month. On those projects, there are between 200 and 500 repositories with approximately 400 GitLab users. These error codes are being thrown in the API when following an automated API flow that hits several different endpoints and paginates through them.

• Slack channel

API flow

There are 3 flows being followed where these errors surface:

• Workflow 1
  1. Get all groups /groups?page=XXX&per_page=YYY&min_access_level=ZZZ
  2. Iterate over each group to retrieve projects /groups/:id/projects?page=XXX&per_page=YYY&min_access_level=ZZZ?archived=false
  3. Iterate over each project and get all members /projects/%s/members/all?page=XXX&per_page=YYY
     • Failures happen here, seemingly at random
• Workflow 2
  1. Get all groups /groups?page=XXX&per_page=YYY&min_access_level=ZZZ
  2. Iterate over each group to retrieve projects /groups/:id/projects?page=XXX&per_page=YYY&min_access_level=ZZZ?archived=false
  3. Iterate through each project and look for
     • Approval Rules /projects/%projectId/approval_rules
     • Project configuration /projects/%projectId/approvals
     • Protected Branches /projects/:projectId/protected_branches/:projectDefaultBranch
     • Failures happen on these endpoints
• Workflow 3
  1. This is the combination of the above 2 workflows, where Project Members is called last and sees failures

Task	Workflow	Status	URL
Get Protected Branches	2	502	https://gitlab.com/api/v4/projects/:id/protected_branches/main
Get Project approvals	2	502	https://gitlab.com/api/v4/projects/:id/approval_rules
Get Project configuration	2	502	https://gitlab.com/api/v4/projects/:id/approvals
Get Project members	1, 3	502	https://gitlab.com/api/v4/projects/:id/members/all?per_page=100&page=1