Jira Cloud app - 401 error when installing manually

Summary

When installing the GitLab for Jira Cloud app manually, users will see a 401 error:

The app host returned HTTP response code 401 when we tried to contact it during installation. Please try again later or contact the app vendor.

Due to a change (introduced about 17th Nov 2024) in how Atlassian sends us the JWT token during the app install, this has broken how we validate the JWT token and this causes the 401 error.

This merge request !173110 (merged) fixed the problem for users that installed the GitLab for Jira Cloud app via the Atlassian Marketplace but to fix this for manual installs, it would require this MR to be backported to previous releases.

This problem currently affects all GitLab self-managed versions.

Steps to reproduce

Follow the instructions to install the GitLab for Jira Cloud app manually

What is the current bug behavior?

Users see a 401 error when installing the GitLab for Jira Cloud app manually.

What is the expected correct behavior?

Users should not see a 401 error when installing the GitLab for Jira Cloud app manually.

Relevant logs and/or screenshots

# /var/log/gitlab/nginx/gitlab_access.log
104.192.143.247 - - [20/Nov/2024:10:09:15 +1300] "POST /-/jira_connect/events/installed HTTP/1.1" 401 5 "" "Atlassian HttpClient unknown / JIRA-1001.0.0-SNAPSHOT (100276) / Atlassian-Connect/1001.0.0-SNAPSHOT" -

# /var/log/gitlab/gitlab-rails/exceptions_json.log
{"severity":"ERROR","time":"2024-11-19T22:47:19.645Z","correlation_id":"01JD39QPSR0SQ4G3FWWESA18GR","meta.caller_id":"JiraConnect::EventsController#installed","meta.feature_category":"integrations","meta.remote_ip":"104.192.143.241","meta.client_id":"ip/104.192.143.241","exception.class":"Atlassian::JiraConnect::Jwt::Asymmetric::KeyFetchError","exception.message":"Atlassian::JiraConnect::Jwt::Asymmetric::KeyFetchError","exception.backtrace":["lib/atlassian/jira_connect/jwt/asymmetric.rb:66:in `retrieve_public_key'","lib/atlassian/jira_connect/jwt/asymmetric.rb:40:in `block in claims'","gems/gitlab-utils/lib/gitlab/utils/strong_memoize.rb:34:in `strong_memoize'","lib/atlassian/jira_connect/jwt/asymmetric.rb:38:in `claims'","lib/atlassian/jira_connect/jwt/asymmetric.rb:26:in `valid?'","app/controllers/jira_connect/events_controller.rb:55:in `verify_asymmetric_atlassian_jwt!'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:400:in `block in make_lambda'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:199:in `block (2 levels) in halting'","actionpack (7.0.8.4) lib/abstract_controller/callbacks.rb:34:in `block (2 levels) in \u003cmodule:Callbacks\u003e'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:200:in `block in halting'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:595:in `block in invoke_before'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:595:in `each'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:595:in `invoke_before'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:116:in `block in run_callbacks'","lib/gitlab/ip_address_state.rb:11:in `with'","ee/app/controllers/ee/application_controller.rb:45:in `set_current_ip_address'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","app/controllers/application_controller.rb:484:in `set_current_admin'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:475:in `set_session_storage'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","lib/gitlab/i18n.rb:114:in `with_locale'","app/controllers/application_controller.rb:468:in `set_locale'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","marginalia (1.11.1) lib/marginalia.rb:109:in `record_query_comment'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","app/controllers/application_controller.rb:459:in `set_current_context'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","sentry-rails (5.19.0) lib/sentry/rails/controller_transaction.rb:30:in `block in sentry_around_action'","sentry-ruby (5.19.0) lib/sentry/hub.rb:102:in `with_child_span'","sentry-ruby (5.19.0) lib/sentry-ruby.rb:498:in `with_child_span'","sentry-rails (5.19.0) lib/sentry/rails/controller_transaction.rb:16:in `sentry_around_action'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:138:in `run_callbacks'","actionpack (7.0.8.4) lib/abstract_controller/callbacks.rb:233:in `process_action'","actionpack (7.0.8.4) lib/action_controller/metal/rescue.rb:23:in `process_action'","actionpack (7.0.8.4) lib/action_controller/metal/instrumentation.rb:67:in `block in process_action'","activesupport (7.0.8.4) lib/active_support/notifications.rb:206:in `block in instrument'","activesupport (7.0.8.4) lib/active_support/notifications/instrumenter.rb:24:in `instrument'","activesupport (7.0.8.4) lib/active_support/notifications.rb:206:in `instrument'","actionpack (7.0.8.4) lib/action_controller/metal/instrumentation.rb:66:in `process_action'","actionpack (7.0.8.4) lib/action_controller/metal/params_wrapper.rb:259:in `process_action'","activerecord (7.0.8.4) lib/active_record/railties/controller_runtime.rb:27:in `process_action'","actionpack (7.0.8.4) lib/abstract_controller/base.rb:151:in `process'","actionview (7.0.8.4) lib/action_view/rendering.rb:39:in `process'","actionpack (7.0.8.4) lib/action_controller/metal.rb:188:in `dispatch'","actionpack (7.0.8.4) lib/action_controller/metal.rb:249:in `block in dispatch'","lib/gitlab/middleware/action_controller_static_context.rb:23:in `call'","actionpack (7.0.8.4) lib/action_controller/metal.rb:249:in `dispatch'","actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:49:in `dispatch'","actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:32:in `serve'","actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:50:in `block in serve'","actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:32:in `each'","actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:32:in `serve'","actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:852:in `call'","gitlab-experiment (0.9.1) lib/gitlab/experiment/middleware.rb:19:in `call'","omniauth (2.1.0) lib/omniauth/strategy.rb:202:in `call!'","omniauth (2.1.0) lib/omniauth/strategy.rb:169:in `call'","omniauth (2.1.0) lib/omniauth/strategy.rb:202:in `call!'","omniauth (2.1.0) lib/omniauth/strategy.rb:169:in `call'","omniauth (2.1.0) lib/omniauth/strategy.rb:202:in `call!'","omniauth (2.1.0) lib/omniauth/strategy.rb:169:in `call'","omniauth (2.1.0) lib/omniauth/strategy.rb:470:in `call_app!'","ee/lib/omni_auth/strategies/group_saml.rb:41:in `other_phase'","omniauth (2.1.0) lib/omniauth/strategy.rb:195:in `call!'","omniauth (2.1.0) lib/omniauth/strategy.rb:169:in `call'","omniauth (2.1.0) lib/omniauth/strategy.rb:202:in `call!'","omniauth (2.1.0) lib/omniauth/strategy.rb:169:in `call'","omniauth (2.1.0) lib/omniauth/strategy.rb:202:in `call!'","omniauth (2.1.0) lib/omniauth/strategy.rb:169:in `call'","omniauth (2.1.0) lib/omniauth/strategy.rb:470:in `call_app!'","omniauth_openid_connect (0.8.0) lib/omniauth/strategies/openid_connect.rb:163:in `other_phase'","omniauth (2.1.0) lib/omniauth/strategy.rb:195:in `call!'","omniauth (2.1.0) lib/omniauth/strategy.rb:169:in `call'","omniauth (2.1.0) lib/omniauth/strategy.rb:470:in `call_app!'","omniauth-saml (2.2.1) lib/omniauth/strategies/saml.rb:86:in `other_phase'","omniauth (2.1.0) lib/omniauth/strategy.rb:195:in `call!'","omniauth (2.1.0) lib/omniauth/strategy.rb:169:in `call'","omniauth (2.1.0) lib/omniauth/strategy.rb:470:in `call_app!'","omniauth-saml (2.2.1) lib/omniauth/strategies/saml.rb:86:in `other_phase'","omniauth (2.1.0) lib/omniauth/strategy.rb:195:in `call!'","omniauth (2.1.0) lib/omniauth/strategy.rb:169:in `call'","omniauth (2.1.0) lib/omniauth/strategy.rb:202:in `call!'","omniauth (2.1.0) lib/omniauth/strategy.rb:169:in `call'","flipper (0.26.2) lib/flipper/middleware/memoizer.rb:72:in `memoized_call'","flipper (0.26.2) lib/flipper/middleware/memoizer.rb:37:in `call'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call'","lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'","lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call'","lib/gitlab/middleware/memory_report.rb:13:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:46:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/middleware/go.rb:21:in `call'","lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'","lib/gitlab/database/query_analyzer.rb:40:in `within'","lib/gitlab/middleware/query_analyzer.rb:11:in `call'","batch-loader (2.0.5) lib/batch_loader/middleware.rb:11:in `call'","rack-attack (6.7.0) lib/rack/attack.rb:103:in `call'","apollo_upload_server (2.1.6) lib/apollo_upload_server/middleware.rb:19:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","rack-attack (6.7.0) lib/rack/attack.rb:127:in `call'","warden (1.2.9) lib/warden/manager.rb:36:in `block in call'","warden (1.2.9) lib/warden/manager.rb:34:in `catch'","warden (1.2.9) lib/warden/manager.rb:34:in `call'","rack-cors (2.0.2) lib/rack/cors.rb:102:in `call'","rack (2.2.9) lib/rack/tempfile_reaper.rb:15:in `call'","rack (2.2.9) lib/rack/etag.rb:27:in `call'","rack (2.2.9) lib/rack/conditional_get.rb:40:in `call'","rack (2.2.9) lib/rack/head.rb:12:in `call'","actionpack (7.0.8.4) lib/action_dispatch/http/permissions_policy.rb:38:in `call'","actionpack (7.0.8.4) lib/action_dispatch/http/content_security_policy.rb:36:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'","rack (2.2.9) lib/rack/session/abstract/id.rb:266:in `context'","rack (2.2.9) lib/rack/session/abstract/id.rb:260:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/cookies.rb:704:in `call'","lib/gitlab/middleware/strip_cookies.rb:29:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:99:in `run_callbacks'","actionpack (7.0.8.4) lib/action_dispatch/middleware/callbacks.rb:26:in `call'","sentry-rails (5.19.0) lib/sentry/rails/rescued_exception_interceptor.rb:12:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'","lib/gitlab/middleware/path_traversal_check.rb:35:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","sentry-ruby (5.19.0) lib/sentry/rack/capture_exceptions.rb:30:in `block (2 levels) in call'","sentry-ruby (5.19.0) lib/sentry/hub.rb:258:in `with_session_tracking'","sentry-ruby (5.19.0) lib/sentry-ruby.rb:411:in `with_session_tracking'","sentry-ruby (5.19.0) lib/sentry/rack/capture_exceptions.rb:21:in `block in call'","sentry-ruby (5.19.0) lib/sentry/hub.rb:59:in `with_scope'","sentry-ruby (5.19.0) lib/sentry-ruby.rb:391:in `with_scope'","sentry-ruby (5.19.0) lib/sentry/rack/capture_exceptions.rb:20:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/show_exceptions.rb:29:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'","railties (7.0.8.4) lib/rails/rack/logger.rb:25:in `block in call'","activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:99:in `block in tagged'","activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:37:in `tagged'","activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:99:in `tagged'","railties (7.0.8.4) lib/rails/rack/logger.rb:25:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/remote_ip.rb:93:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:15:in `call'","lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'","request_store (1.5.1) lib/request_store/middleware.rb:19:in `call'","rack (2.2.9) lib/rack/method_override.rb:24:in `call'","rack (2.2.9) lib/rack/runtime.rb:22:in `call'","rack-timeout (0.7.0) lib/rack/timeout/core.rb:154:in `block in call'","rack-timeout (0.7.0) lib/rack/timeout/support/timeout.rb:19:in `timeout'","rack-timeout (0.7.0) lib/rack/timeout/core.rb:153:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/compressed_json.rb:44:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/executor.rb:14:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","rack (2.2.9) lib/rack/sendfile.rb:110:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'","gitlab-labkit (0.36.1) lib/labkit/middleware/rack.rb:22:in `block in call'","gitlab-labkit (0.36.1) lib/labkit/context.rb:35:in `with_context'","gitlab-labkit (0.36.1) lib/labkit/middleware/rack.rb:21:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/request_id.rb:26:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/host_authorization.rb:131:in `call'","railties (7.0.8.4) lib/rails/engine.rb:530:in `call'","railties (7.0.8.4) lib/rails/railtie.rb:226:in `public_send'","railties (7.0.8.4) lib/rails/railtie.rb:226:in `method_missing'","lib/gitlab/middleware/release_env.rb:13:in `call'","rack (2.2.9) lib/rack/urlmap.rb:74:in `block in call'","rack (2.2.9) lib/rack/urlmap.rb:58:in `each'","rack (2.2.9) lib/rack/urlmap.rb:58:in `call'","puma (6.4.0) lib/puma/configuration.rb:272:in `call'","puma (6.4.0) lib/puma/request.rb:100:in `block in handle_request'","puma (6.4.0) lib/puma/thread_pool.rb:378:in `with_force_shutdown'","puma (6.4.0) lib/puma/request.rb:99:in `handle_request'","puma (6.4.0) lib/puma/server.rb:443:in `process_client'","puma (6.4.0) lib/puma/server.rb:241:in `block in run'","puma (6.4.0) lib/puma/thread_pool.rb:155:in `block in spawn_thread'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":"integrations","tags.correlation_id":"01JD39QPSR0SQ4G3FWWESA18GR"}

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info


(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Possible fixes

Workaround

To resolve this, you will need to apply a patch to your self-managed instance.

When to Apply the Patch

Only if installing GitLab for Jira Cloud app manually
Must reapply after GitLab upgrades (only if reinstalling the GitLab for Jira Cloud app)
Not needed if app is already installed and functioning, or you have installed the GitLab for Jira Cloud app from the Atlassian Marketplace

Patch instructions

These instructions are for Omnibus installs. If you have multiple nodes, this needs to be executed on all app/Rails nodes.

See the general patching an instance doc for more details.

# Use https://gitlab.com/gitlab-org/gitlab/-/merge_requests/173110 as the patch source
mr_iid=173110

curl -o /tmp/$mr_iid.patch https://gitlab.com/gitlab-org/gitlab/-/merge_requests/$mr_iid.patch
cd /opt/gitlab/embedded/service/gitlab-rails
patch -p1 -b -f < /tmp/$mr_iid.patch
gitlab-ctl restart

Edited Nov 20, 2024 by Magdalena Frankiewicz