CI/CD jobs are getting stuck in "starting" in version 16.0.1

Summary

CI/CD jobs getting stuck after using GitLab v16.0.1.

What is the current bug behavior?

The job does not start and job page shows the message:

This job has not been triggered yet
This job depends on upstream jobs that need to succeed in order for this job to be triggered

What is the expected correct behavior?

The job should start.

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Possible workaround

For high sidekiq loads as a result of indexing jobs:

1. Pause indexing using gitlab:elastic:pause_indexing rake task or Admin UI
2. Allow system to stabilize

If the strict_dynamic_mapping_exception error is present in the elasticsearch.log, that means a major version upgrade has occurred but the Advanced Search migrations were not finished before the upgrade. To fix the mapping exception, a full reindex from scratch is needed.

1. Reindex everything from scratch using gitlab:elastic:index rake task
2. Unpause indexing (if it's still paused from the previous steps) using a rake task or Admin UI