GitLab is still showing "Can't verify CSRF token authenticity" in production.log.

Summary

One of our customers is seeing this error in production.log after doing a backup and restore. There's no other related error and GitLab is working normally without any issues. They are seeing this on GitLab 16.8.1.

As per my discussion with @stanhu, the CSRF warning is coming from:

=== CSRF error here:
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:400:in `block in make_lambda'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:180:in `block (2 levels) in halting_and_conditional'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/abstract_controller/callbacks.rb:34:in `block (2 levels) in <module:Callbacks>'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:181:in `block in halting_and_conditional'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:595:in `block in invoke_before'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:595:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:595:in `invoke_before'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:116:in `block in run_callbacks'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/ip_address_state.rb:10:in `with'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/controllers/ee/application_controller.rb:45:in `set_current_ip_address'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
/opt/gitlab/embedded/service/gitlab-rails/app/controllers/application_controller.rb:468:in `set_current_admin'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/session.rb:11:in `with_session'
/opt/gitlab/embedded/service/gitlab-rails/app/controllers/application_controller.rb:459:in `set_session_storage'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/i18n.rb:114:in `with_locale'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/i18n.rb:120:in `with_user_locale'
/opt/gitlab/embedded/service/gitlab-rails/app/controllers/application_controller.rb:450:in `set_locale'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/marginalia-1.11.1/lib/marginalia.rb:109:in `record_query_comment'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-raven-3.1.2/lib/raven/integrations/rails/controller_transaction.rb:7:in `block in included'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:127:in `instance_exec'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
/opt/gitlab/embedded/service/gitlab-rails/app/controllers/application_controller.rb:443:in `set_current_context'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-rails-5.10.0/lib/sentry/rails/controller_transaction.rb:28:in `block in sentry_around_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/hub.rb:102:in `with_child_span'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry-ruby.rb:456:in `with_child_span'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-rails-5.10.0/lib/sentry/rails/controller_transaction.rb:14:in `sentry_around_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:138:in `run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/abstract_controller/callbacks.rb:233:in `process_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_controller/metal/rescue.rb:23:in `process_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_controller/metal/instrumentation.rb:67:in `block in process_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:206:in `block in instrument'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:206:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_controller/metal/instrumentation.rb:66:in `process_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_controller/metal/params_wrapper.rb:259:in `process_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/abstract_controller/base.rb:151:in `process'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionview-7.0.8/lib/action_view/rendering.rb:39:in `process'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_controller/metal.rb:188:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_controller/metal.rb:251:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/routing/route_set.rb:49:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/routing/route_set.rb:32:in `serve'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/journey/router.rb:50:in `block in serve'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/journey/router.rb:32:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/journey/router.rb:32:in `serve'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/routing/route_set.rb:852:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/gitlab-experiment-0.9.1/lib/gitlab/experiment/middleware.rb:19:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/flipper-0.26.2/lib/flipper/middleware/memoizer.rb:72:in `memoized_call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/flipper-0.26.2/lib/flipper/middleware/memoizer.rb:37:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/memory_report.rb:13:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/speedscope.rb:13:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/etag_caching/middleware.rb:21:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/web_transaction.rb:46:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/rack_middleware.rb:16:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/go.rb:20:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/query_analyzer.rb:11:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/batch-loader-2.0.1/lib/batch_loader/middleware.rb:11:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-attack-6.7.0/lib/rack/attack.rb:103:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/apollo_upload_server-2.1.5/lib/apollo_upload_server/middleware.rb:19:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/multipart.rb:173:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-attack-6.7.0/lib/rack/attack.rb:127:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/warden-1.2.9/lib/warden/manager.rb:36:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/warden-1.2.9/lib/warden/manager.rb:34:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/warden-1.2.9/lib/warden/manager.rb:34:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-cors-2.0.1/lib/rack/cors.rb:102:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/etag.rb:27:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/conditional_get.rb:40:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/head.rb:12:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/http/permissions_policy.rb:38:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/http/content_security_policy.rb:36:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only/controller.rb:50:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only.rb:18:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:266:in `context'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:260:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/middleware/cookies.rb:704:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:99:in `run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-rails-5.10.0/lib/sentry/rails/rescued_exception_interceptor.rb:12:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/path_traversal_check.rb:35:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/rack/capture_exceptions.rb:28:in `block (2 levels) in call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/hub.rb:227:in `with_session_tracking'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry-ruby.rb:385:in `with_session_tracking'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/rack/capture_exceptions.rb:19:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/hub.rb:59:in `with_scope'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry-ruby.rb:365:in `with_scope'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/rack/capture_exceptions.rb:18:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/middleware/show_exceptions.rb:29:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/basic_health_check.rb:25:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/lograge-0.11.2/lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/rack/logger.rb:25:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/tagged_logging.rb:99:in `block in tagged'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/tagged_logging.rb:37:in `tagged'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/tagged_logging.rb:99:in `tagged'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/rack/logger.rb:25:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/middleware/remote_ip.rb:93:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/request_context.rb:15:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/request_store-1.5.1/lib/request_store/middleware.rb:19:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/method_override.rb:24:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/runtime.rb:22:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-timeout-0.6.3/lib/rack/timeout/core.rb:148:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-timeout-0.6.3/lib/rack/timeout/support/timeout.rb:19:in `timeout'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-timeout-0.6.3/lib/rack/timeout/core.rb:147:in `call'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/fix_local_cache_middleware.rb:11:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/compressed_json.rb:44:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/sendfile.rb:110:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/middleware/host_authorization.rb:131:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/gitlab-labkit-0.35.0/lib/labkit/middleware/rack.rb:22:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/gitlab-labkit-0.35.0/lib/labkit/context.rb:35:in `with_context'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/gitlab-labkit-0.35.0/lib/labkit/middleware/rack.rb:21:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/actionpack-7.0.8/lib/action_dispatch/middleware/request_id.rb:26:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-raven-3.1.2/lib/raven/integrations/rack.rb:51:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/engine.rb:530:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/railtie.rb:226:in `public_send'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/railtie.rb:226:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/release_env.rb:13:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/urlmap.rb:74:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/urlmap.rb:58:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-2.2.8/lib/rack/urlmap.rb:58:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/puma-6.4.0/lib/puma/configuration.rb:272:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/puma-6.4.0/lib/puma/request.rb:100:in `block in handle_request'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/puma-6.4.0/lib/puma/thread_pool.rb:378:in `with_force_shutdown'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/puma-6.4.0/lib/puma/request.rb:99:in `handle_request'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/puma-6.4.0/lib/puma/server.rb:443:in `process_client'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/puma-6.4.0/lib/puma/server.rb:241:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/puma-6.4.0/lib/puma/thread_pool.rb:155:in `block in spawn_thread'
Can't verify CSRF token authenticity.

What is the current bug behavior?

production.log is being spammed with unhelpful error message about CSRF.

What is the expected correct behavior?

Hide CSRF warnings in production.log

Possible fixes

@stanhu suggests to set ActionController::Base.log_warning_on_csrf_failure = false