API Group/:id/Projects endpoint is broken

Summary

In the latest Gitlab-EE 17.3 Docker-image, it seems like 90% of group’s projects api endpoints are broken.

Mose of them (located under /groups/:id/projects) are retuning 500 errors, instead of the usual JSON.

The weird thing is, there are still 3 groups, out of 31 total groups where this endpoint is working normally.

Steps to reproduce

Unknown.

We recently purchased a Gitlab Premium subscribtion and switched from to the self hosted "CE" to "EE" docker image. The image-switch worked flawlessly, and the new Epic-planning features are really appreciated.

Then we started restructuring all of our groups and planning, by creating subgroups several layers deep, moving projects around, renaming them and etc... Replicating the exact chain of events that led here is probably impossible and we cannot exactly pinpoint the problem.

Example Project

I did a simple test, where I created a group, created a project, then renamed the project once:

https://gitlab.com/test-if-the-projects-endpoint-is-broken/new-blank-project

But it seems that on gitlab.com by default this is not a problem.

The api for this group still works: https://gitlab.com/api/v4/groups/92084700/projects

Probably my test-case was not complex enough to trigger these problems.

What is the current bug behavior?

Most group's projects' api endpoints are broken like here:

(Yes, we use a custom Error-page middleware via Traefik, disabling it did not help)

What is the expected correct behavior?

The group-projects endpoint should deliver normal json format, as you can see with these groups where it is still working:

So our reverse proxy is probably not the problem in this case.

Relevant logs and/or screenshots

Client script where we wanted to clone all repos fails on most groups:

.\clone_all_gitlab_repos.ps1
Processing group: 154 = Archive
Error: Der Remoteserver hat einen Fehler zurückgegeben: (500) Interner Serverfehler. for URI: https://gitlab.server.com/api/v4/groups/58/projects
Error: Der Remoteserver hat einen Fehler zurückgegeben: (500) Interner Serverfehler. for URI: https://gitlab.server.com/api/v4/groups/62/projects
Error: Der Remoteserver hat einen Fehler zurückgegeben: (500) Interner Serverfehler. for URI: https://gitlab.server.com/api/v4/groups/51/projects
Error: Der Remoteserver hat einen Fehler zurückgegeben: (500) Interner Serverfehler. for URI: https://gitlab.server.com/api/v4/groups/7/projects
Error: Der Remoteserver hat einen Fehler zurückgegeben: (500) Interner Serverfehler. for URI: https://gitlab.server.com/api/v4/groups/95/projects
Error: Der Remoteserver hat einen Fehler zurückgegeben: (500) Interner Serverfehler. for URI: https://gitlab.server.com/api/v4/groups/10/projects
Error: Der Remoteserver hat einen Fehler zurückgegeben: (500) Interner Serverfehler. for URI: https://gitlab.server.com/api/v4/groups/15/projects
Processing group: 129 = Backend
Error: Der Remoteserver hat einen Fehler zurückgegeben: (500) Interner Serverfehler. for URI: https://gitlab.server.com/api/v4/groups/129/projects
Processing group: 137 = Backend
Error: Der Remoteserver hat einen Fehler zurückgegeben: (500) Interner Serverfehler. for URI: https://gitlab.server.com/api/v4/groups/137/projects
Processing group: 146 = Backend
Error: Der Remoteserver hat einen Fehler zurückgegeben: (500) Interner Serverfehler. for URI: https://gitlab.server.com/api/v4/groups/146/projects
Processing group: 152 = Base Images
Cloning into 'D:\Groups\dev\base-images\gitlab-release-base'...

As you can see, most of the groups fail, but then some of them are working again. Like group 152.

This script ran without any problem last week. So either it was a regression between the 12.08.2024 - 19.08.2024, or our internal group & project restructuring severely broke something else in the meantime.

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info


(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

```
# docker exec -it gitlab bash

root@gitlab:/# gitlab-rake gitlab:env:info

System information
System:
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   3.1.5p253
Gem Version:    3.5.11
Bundler Version:2.5.11
Rake Version:   13.0.6
Redis Version:  7.0.15
Sidekiq Version:7.1.6
Go Version:     unknown

GitLab information
Version:        17.3.0-ee
Revision:       5aad128b1de
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     14.11
URL:            https://gitlab.server.com
HTTP Clone URL: https://gitlab.server.com/some-group/some-project.git
SSH Clone URL:  ssh://git@gitlab.server.com:2222/some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: saml

GitLab Shell
Version:        14.38.0
Repository storages:
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address:      unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version:      17.3.0
- default Git Version:  2.45.2
root@gitlab:/#
```

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of: sudo gitlab-rake gitlab:check SANITIZE=true)

root@gitlab:/# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 14.38.0 ? ... OK (14.38.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Database config exists? ... yes
Tables are truncated? ... skipped
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
6/2 ... yes
7/4 ... yes
7/5 ... yes
10/8 ... yes
10/9 ... yes
6/10 ... yes
10/11 ... yes
10/12 ... yes
15/14 ... yes
15/16 ... yes
15/17 ... yes
15/20 ... yes
15/21 ... yes
15/22 ... yes
15/24 ... yes
129/26 ... yes
51/27 ... yes
7/28 ... yes
7/29 ... yes
7/30 ... yes
58/32 ... yes
130/33 ... yes
6/34 ... yes
146/35 ... yes
62/36 ... yes
143/38 ... yes
137/39 ... yes
151/41 ... yes
131/42 ... yes
139/43 ... yes
131/45 ... yes
136/46 ... yes
136/47 ... yes
131/48 ... yes
150/49 ... yes
133/50 ... yes
136/51 ... yes
155/52 ... yes
135/53 ... yes
140/54 ... yes
151/55 ... yes
151/56 ... yes
135/57 ... yes
141/58 ... yes
7/59 ... yes
7/60 ... yes
95/61 ... yes
95/62 ... yes
150/64 ... yes
150/65 ... yes
150/66 ... yes
150/67 ... yes
149/68 ... yes
95/69 ... yes
147/70 ... yes
51/72 ... yes
142/73 ... yes
138/74 ... yes
138/76 ... yes
6/77 ... yes
152/79 ... yes
6/80 ... yes
129/81 ... yes
150/82 ... yes
6/83 ... yes
152/84 ... yes
151/85 ... yes
Redis version >= 6.2.14? ... yes
Ruby version >= 3.0.6 ? ... yes (3.1.5)
Git user has default SSH configuration? ... yes
Active users: ... 7
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled)
All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled)

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

root@gitlab:/#

(we will only investigate if the tests are passing)