Jobs stuck in "created" state

Since the upgrade to v16.6.2 a few project's CI does no longer work! It has been like this for two weeks now.

Basically, I can no longer run the CI. When I manually start a pipeline from the UI, I'm being redirected to its detail page and see all jobs created. However, this pipeline does not show up in the pipelines list for that project. The jobs show up on the GitLab admin page, though. Neither are these jobs ever started, they remain in the "created" state forever. So they also don't time out. The job detail page displays "This job has not been triggered yet".

I believe these projects are bugged, for some reason. Maybe through a failed (database) migration? Forking the project and running the pipelines in the fork perfectly works!

I don't know how to further debug this issue. There don't seem to be any related errors in the logs. There are no queued nor failed background jobs. There are some CI related dead sidekiq tasks, but the same show up for the working projects and pipelines as well:

Queue	Job	Arguments	Error
default	Ci::InitialPipelineProcessWorker	4119	NameError: uninitialized constant Ci
default	Ci::ArchiveTraceWorker	11844	NameError: uninitialized constant Ci
default	PipelineNotificationWorker	4119, {"ref_status"=>"failed"}	NameError: uninitialized constant PipelineNotificationWorker
default	PipelineHooksWorker	4119	NameError: uninitialized constant PipelineHooksWorker
default	PipelineProcessWorker	4117	NameError: uninitialized constant PipelineProcessWorker

My ratio of succeeded and failed sidekiq tasks is 3,329,067 (processed) vs. 101,351 (failed). I'm not sure whether that's "healthy".

There are similar issues (#341293 (closed), #433404 (closed), #367078 (closed), gitlab-foss#37797 (closed)), but they all have in common, that the pipeline/jobs are started. They also don't mention the pipeline not showing up in the list view.

Checks

gitlab-ctl status

$ sudo gitlab-ctl status
run: gitaly: (pid 3846047) 593726s; run: log: (pid 1967) 12799947s
run: gitlab-exporter: (pid 3846038) 593727s; run: log: (pid 1965) 12799947s
run: gitlab-workhorse: (pid 3846021) 593728s; run: log: (pid 1975) 12799947s
run: logrotate: (pid 819629) 3319s; run: log: (pid 1978) 12799947s
run: node-exporter: (pid 3846049) 593726s; run: log: (pid 1968) 12799947s
run: puma: (pid 3846060) 593726s; run: log: (pid 1964) 12799947s
run: registry: (pid 209653) 312817s; run: log: (pid 1963) 12799947s
run: sidekiq: (pid 3846078) 593725s; run: log: (pid 1969) 12799947s

gitlab-rake

$ sudo gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 14.30.0 ? ... OK (14.30.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Database config exists? ... yes
Tables are truncated? ... skipped
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
// a lot of projects, all of which answered with 'yes'

Redis version >= 6.0.0? ... yes
Ruby version >= 3.0.6 ? ... yes (3.0.6)
Git user has default SSH configuration? ... yes
Active users: ... 13
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... no
  Try fixing it:
  Please migrate all projects to hashed storage
  as legacy storage is deprecated in 13.0 and support will be removed in 14.0.
  For more information see:
  doc/administration/repository_storage_types.md

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished