500 error after turning on CI catalog on a repo

Summary

After recent update of GitLab CE to v16.6.0 (Omnibus), I've turned on CI Catalog feature for a repo that I haven't prepared for a usage as a CI Component catalog repo according to docs (haven't created folders/files as specified) and got 500 error on that repo's URI as well as other paths of my installation, like GitLab site root.

Steps to reproduce

As my installation is currently impaired with 500 error, I'm unable to reproduce the error, but here's how I got to it:

  • Upgrade to Omnibus v16.6.0
  • Turn on CI/CD Catalog resource in Settings > General > Visibility, project features, permissions of the repo

Example Project

Here're the contents of the repo in question:

-rw-r--r--   1 501  20  2129 Oct 28  2020 .App-Version-Update.gitlab-ci.yml
-rw-r--r--   1 501  20     7 Aug 14  2020 .gitignore
-rw-r--r--   1 501  20   329 Nov 29 14:51 .gitlab-ci.yml
-rw-r--r--   1 501  20  1776 Sep  7  2020 README.md
drwxr-xr-x   3 501  20    96 Nov 29 14:46 templates
-rw-r--r--  1 501  20  2129 Oct 28  2020 templates/app-version-update.yml

gitlab-ci.yml:

stages:
  - deploy

create-release:
  stage: deploy
  image: registry.gitlab.com/gitlab-org/release-cli:latest
  rules:
    - if: $CI_COMMIT_TAG =~ /^v\d+/
  script: echo "Creating release $CI_COMMIT_TAG"
  release:
    tag_name: $CI_COMMIT_TAG
    description: "Release $CI_COMMIT_TAG of components repository $CI_PROJECT_PATH"

templates/app-version-update.yml

# This template job updates remote Terraform application's version store with given values
# Environment variables expected:
#   - ENV (should contain target environment name, if none `dev` is assumed)
#   - VERSION_TAGS (specially crafted string, that points to a certain line in the target version
#                   file and replace it with a new version value)
#
# USAGE EXAMPLE:
# Include this file in the upstream pipeline with:
# stages:
#   - dev-version-update
# include
#   - project: 'v-ticket-env/templates'
#     ref: master
#     file: .App-Version-Update.gitlab-ci.yml
#
# Dev-Version-Update:
#   extends: .app-version-update
#   stage: dev-version-update
#   variables:
#     ENV: "dev"
#     VERSION_TAGS: "concertua-app_docker_image_version=${CI_COMMIT_TAG};parterua-app_docker_image_version=${CI_COMMIT_TAG}"

.app-version-update:
  cache: {}
  variables:
    ENV: dev
  tags:
    - release-tools
  only:
    - tags
  script:
    - git clone --depth 1 git@gitlab.***.com:v-ticket-env/infra-versions.git
    - cd infra-versions
    - if [ -z "$VERSION_TAGS" ]; then echo "No $VERSION_TAGS variable or it is empty! Can't proceed." && exit 1; fi
    - if [ -z "$ENV" ]; then echo "No $ENV variable or it is empty! Can't proceed." && exit 1; fi
    - echo "Updating $ENV-versions with $VERSION_TAGS"
    # Below we split the incoming data into an array by `;` symbol and than use params separated by `=` symbol
    # Example of the incoming data with 2 pairs of params to be parsed correctly:
    # concertua-app_docker_image_version=v3.1.185;parterua-app_docker_image_version=v3.1.185
    - |
      IFS=\; read -a CMD_ARRAY <<<"$VERSION_TAGS"
      for CMD in "${CMD_ARRAY[@]}" ; do
        IFS=\= read -a PARAMS <<<"$CMD"
        sed -i -e "s/${PARAMS[0]}\(.*\)/${PARAMS[0]} = \"${PARAMS[1]}\"/g" ${ENV}/${ENV}-app-versions.tfvars;
        echo "Updated ${PARAMS[0]} with version tag ${PARAMS[1]}"
      done
    - git add ${ENV}/${ENV}-app-versions.tfvars
    - git commit -m "Version tags update $VERSION_TAGS in $ENV-environment"
    - git push -u git@gitlab.***.com:v-ticket-env/infra-versions.git master

What is the current bug behavior?

500 error in browser on some pages, most notable on GitLab site root and project that was marked as CI component

What is the expected correct behavior?

Pages display correctly

Relevant logs and/or screenshots

gitlab-ctl tail gitlab-rails shows this error:

==> /var/log/gitlab/gitlab-rails/production.log <==

ActionView::Template::Error (undefined method `project_ci_catalog_resource_path' for #<ActionView::Base:0x000000000ffeb0>
Did you mean?  project_feature_flags_path):
    38:         = visibility_level_content(project, css_class: 'gl-mr-2')
    39:
    40:         - if project.catalog_resource
    41:           = render partial: 'shared/ci_catalog_badge', locals: { href: project_ci_catalog_resource_path(project, project.catalog_resource), css_class: 'gl-mr-2' }
    42:
    43:         - if explore_projects_tab? && project_license_name(project)
    44:           %span.gl-display-inline-flex.gl-align-items-center.gl-mr-3

app/views/shared/projects/_project.html.haml:41
app/views/shared/projects/_list.html.haml:39
app/views/shared/projects/_list.html.haml:37:in `each_with_index'
app/views/shared/projects/_list.html.haml:37
app/views/dashboard/projects/_projects.html.haml:1
app/views/dashboard/projects/index.html.haml:14
app/controllers/application_controller.rb:162:in `render'
app/controllers/dashboard/projects_controller.rb:64:in `block in render_projects'
lib/gitlab/gitaly_client.rb:455:in `allow_n_plus_1_calls'
app/controllers/dashboard/projects_controller.rb:63:in `render_projects'
app/controllers/dashboard/projects_controller.rb:23:in `block (2 levels) in index'
app/controllers/dashboard/projects_controller.rb:21:in `index'
app/controllers/root_controller.rb:25:in `block in index'
lib/gitlab/gitaly_client.rb:455:in `allow_n_plus_1_calls'
app/controllers/root_controller.rb:23:in `index'
app/controllers/application_controller.rb:498:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:489:in `set_session_storage'
lib/gitlab/i18n.rb:114:in `with_locale'
lib/gitlab/i18n.rb:120:in `with_user_locale'
app/controllers/application_controller.rb:480:in `set_locale'
app/controllers/application_controller.rb:473:in `set_current_context'
lib/gitlab/middleware/memory_report.rb:13:in `call'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
lib/gitlab/database/query_analyzer.rb:37:in `within'
lib/gitlab/middleware/query_analyzer.rb:11:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/path_traversal_check.rb:48:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:15:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:44:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'

Output of checks

These commands show no errors and all migrations up

$ sudo gitlab-rake gitlab:check --trace
$ sudo gitlab-rake db:migrate:status --trace

Turning off feature flag Feature.disable(:global_ci_catalog) does not help

Results of GitLab environment info

Expand for output related to GitLab environment info 
(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

root@gitlab:/etc/gitlab# sudo gitlab-rake gitlab:env:info

System information
System:		Ubuntu 18.04
Current User:	git
Using RVM:	no
Ruby Version:	3.0.6p216
Gem Version:	3.4.21
Bundler Version:2.4.21
Rake Version:	13.0.6
Redis Version:	7.0.14
Sidekiq Version:6.5.12
Go Version:	unknown

GitLab information
Version:	16.6.0
Revision:	6d558d71eba
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	13.11
URL:		https://gitlab.***.com
HTTP Clone URL:	https://gitlab.***.com/some-group/some-project.git
SSH Clone URL:	git@gitlab.***.com:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: bitbucket

GitLab Shell
Version:	14.30.0
Repository storages:
- default: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version: 	16.6.0
- default Git Version: 	2.42.0

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


root@gitlab:/etc/gitlab# sudo gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...


Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 14.30.0 ? ... OK (14.30.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Database config exists? ... yes
Tables are truncated? ... skipped
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
15/1 ... yes
15/2 ... yes
14/5 ... yes
15/6 ... yes
14/7 ... yes
14/8 ... yes
14/9 ... yes
13/10 ... yes
14/11 ... yes
15/12 ... yes
13/15 ... yes
13/17 ... yes
15/19 ... yes
15/22 ... yes
15/24 ... yes
15/25 ... yes
15/27 ... yes
29/28 ... yes
30/29 ... yes
30/32 ... yes
29/33 ... yes
15/34 ... yes
29/35 ... yes
29/36 ... yes
30/37 ... yes
29/38 ... yes
29/39 ... yes
29/40 ... yes
29/41 ... yes
30/42 ... yes
29/43 ... yes
29/44 ... yes
29/45 ... yes
15/46 ... yes
35/47 ... yes
35/48 ... yes
35/49 ... yes
35/50 ... yes
35/51 ... yes
15/52 ... yes
30/53 ... yes
29/54 ... yes
29/55 ... yes
29/57 ... yes
15/58 ... yes
15/59 ... yes
30/60 ... yes
35/62 ... yes
41/63 ... yes
29/64 ... yes
30/65 ... yes
29/66 ... yes
29/68 ... yes
52/69 ... yes
29/70 ... yes
34/71 ... yes
34/72 ... yes
29/73 ... yes
14/74 ... yes
30/75 ... yes
29/76 ... yes
29/77 ... yes
35/78 ... yes
35/79 ... yes
38/80 ... yes
29/81 ... yes
40/82 ... yes
52/83 ... yes
30/84 ... yes
58/86 ... yes
15/87 ... yes
29/88 ... yes
15/89 ... yes
29/90 ... yes
15/91 ... yes
40/92 ... yes
30/93 ... yes
35/94 ... yes
40/95 ... yes
30/98 ... yes
29/99 ... yes
15/100 ... yes
15/101 ... yes
15/102 ... yes
15/103 ... yes
15/104 ... yes
35/105 ... yes
73/106 ... yes
29/107 ... yes
34/108 ... yes
29/110 ... yes
15/111 ... yes
15/112 ... yes
34/113 ... yes
82/114 ... yes
83/118 ... yes
83/121 ... yes
82/122 ... yes
38/123 ... yes
38/126 ... yes
83/127 ... yes
82/128 ... yes
29/129 ... yes
29/130 ... yes
83/131 ... yes
83/132 ... yes
83/133 ... yes
83/134 ... yes
29/135 ... yes
30/136 ... yes
30/137 ... yes
15/138 ... yes
30/139 ... yes
29/140 ... yes
30/141 ... yes
30/142 ... yes
15/143 ... yes
30/144 ... yes
30/145 ... yes
221/147 ... yes
29/148 ... yes
225/149 ... yes
225/150 ... yes
225/151 ... yes
29/152 ... yes
30/153 ... yes
95/154 ... yes
30/155 ... yes
30/156 ... yes
29/157 ... yes
29/158 ... yes
Redis version >= 6.0.0? ... yes
Ruby version >= 3.0.6 ? ... yes (3.0.6)
Git user has default SSH configuration? ... yes
Active users: ... 23
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished


(we will only investigate if the tests are passing)

Possible fixes

Unknown

Edited by Yevgen Lysenko