Can't access #30229
I am currently unable to access #30229 (closed) and just get a 502
error.
Steps to reproduce:
- Visit Can't access #30229 (closed)
- Wait
- See
502
warning page.
Calling rb_backtrace()
from a Rails console yields:
from /opt/gitlab/embedded/service/gitlab-rails/app/views/projects/issues/show.html.haml:70:in `_app_views_projects_issues_show_html_haml___496534223925085040_70113034640840'
from /opt/gitlab/embedded/service/gitlab-rails/ee/app/helpers/ee/issuables_helper.rb:17:in `issuable_initial_data'
from /opt/gitlab/embedded/service/gitlab-rails/app/helpers/issuables_helper.rb:276:in `issuable_initial_data'
from /opt/gitlab/embedded/service/gitlab-rails/app/helpers/markup_helper.rb:112:in `markdown_field'
from /opt/gitlab/embedded/service/gitlab-rails/lib/banzai.rb:16:in `render_field'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:161:in `render_field'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/method_call.rb:36:in `measure'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:161:in `block in render_field'
from /opt/gitlab/embedded/service/gitlab-rails/lib/banzai/renderer.rb:42:in `render_field'
from /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cache_markdown_field.rb:58:in `refresh_markdown_cache!'
from /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cache_markdown_field.rb:46:in `refresh_markdown_cache'
from /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cache_markdown_field.rb:46:in `map'
from /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cache_markdown_field.rb:49:in `block in refresh_markdown_cache'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:161:in `cacheless_render_field'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/method_call.rb:36:in `measure'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:161:in `block in cacheless_render_field'
from /opt/gitlab/embedded/service/gitlab-rails/lib/banzai/renderer.rb:52:in `cacheless_render_field'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:161:in `cacheless_render'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/method_call.rb:36:in `measure'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:161:in `block in cacheless_render'
from /opt/gitlab/embedded/service/gitlab-rails/lib/banzai/renderer.rb:160:in `cacheless_render'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/influx_db.rb:103:in `measure'
from /opt/gitlab/embedded/service/gitlab-rails/lib/banzai/renderer.rb:161:in `block in cacheless_render'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:161:in `render_result'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/method_call.rb:36:in `measure'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:161:in `block in render_result'
from /opt/gitlab/embedded/service/gitlab-rails/lib/banzai/renderer.rb:130:in `render_result'
from /opt/gitlab/embedded/service/gitlab-rails/lib/banzai/pipeline/base_pipeline.rb:23:in `block (2 levels) in singleton class'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/html-pipeline-2.8.4/lib/html/pipeline.rb:113:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/html-pipeline-2.8.4/lib/html/pipeline.rb:169:in `instrument'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/html-pipeline-2.8.4/lib/html/pipeline.rb:115:in `block in call'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/html-pipeline-2.8.4/lib/html/pipeline.rb:115:in `inject'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/html-pipeline-2.8.4/lib/html/pipeline.rb:115:in `each'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/html-pipeline-2.8.4/lib/html/pipeline.rb:116:in `block (2 levels) in call'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/html-pipeline-2.8.4/lib/html/pipeline.rb:130:in `perform_filter'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/html-pipeline-2.8.4/lib/html/pipeline.rb:169:in `instrument'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/html-pipeline-2.8.4/lib/html/pipeline.rb:131:in `block in perform_filter'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/html-pipeline-2.8.4/lib/html/pipeline/filter.rb:127:in `call'
from /opt/gitlab/embedded/service/gitlab-rails/lib/banzai/filter/syntax_highlight_filter.rb:17:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/nokogiri-1.10.4/lib/nokogiri/xml/node_set.rb:237:in `each'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/nokogiri-1.10.4/lib/nokogiri/xml/node_set.rb:237:in `upto'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/nokogiri-1.10.4/lib/nokogiri/xml/node_set.rb:238:in `block in each'
from /opt/gitlab/embedded/service/gitlab-rails/lib/banzai/filter/syntax_highlight_filter.rb:18:in `block in call'
from /opt/gitlab/embedded/service/gitlab-rails/lib/banzai/filter/syntax_highlight_filter.rb:38:in `highlight_node'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rouge-3.10.0/lib/rouge/formatter.rb:46:in `format'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rouge-3.10.0/lib/rouge/formatter.rb:74:in `format'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:161:in `stream'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/method_call.rb:36:in `measure'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:161:in `block in stream'
from /opt/gitlab/embedded/service/gitlab-rails/lib/rouge/formatters/html_gitlab.rb:18:in `stream'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rouge-3.10.0/lib/rouge/formatter.rb:96:in `token_lines'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rouge-3.10.0/lib/rouge/formatter.rb:96:in `each'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rouge-3.10.0/lib/rouge/formatter.rb:58:in `filter_escapes'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rouge-3.10.0/lib/rouge/formatter.rb:58:in `each'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rouge-3.10.0/lib/rouge/lexer.rb:448:in `lex'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rouge-3.10.0/lib/rouge/lexer.rb:458:in `continue_lex'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rouge-3.10.0/lib/rouge/regex_lexer.rb:272:in `stream_tokens'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rouge-3.10.0/lib/rouge/regex_lexer.rb:291:in `step'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rouge-3.10.0/lib/rouge/regex_lexer.rb:291:in `each'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rouge-3.10.0/lib/rouge/regex_lexer.rb:294:in `block in step'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rouge-3.10.0/lib/rouge/regex_lexer.rb:291:in `step'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rouge-3.10.0/lib/rouge/regex_lexer.rb:291:in `each'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rouge-3.10.0/lib/rouge/regex_lexer.rb:306:in `block in step'
from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rouge-3.10.0/lib/rouge/regex_lexer.rb:306:in `skip'
This the problematic JSON:
{"severity":"WARN","time":"2019-07-18T14:39:51.091Z","class":"BackgroundMigrationWorker","retry":3,"queue":"background_migration","jid":"7e716ce7e40b6c6a1fc1a647","created_at":"2019-07-18T14:15:32.453Z","correlation_id":"5c06ecbd-21da-4906-a35d-f3e51f4e0048","enqueued_at":"2019-07-18T14:39:51.000Z","error_message":"Sidekiq::JobRetry::Skip","error_class":"NameError","failed_at":"2019-07-18T14:17:02.563Z","retry_count":0,"pid":30739,"message":"BackgroundMigrationWorker JID-7e716ce7e40b6c6a1fc1a647: fail: 0.09 sec","job_status":"fail","duration":0.09,"completed_at":"2019-07-18T14:39:51.091Z","error":{},"error_backtrace":["/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:127:in `rescue in local'","/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:108:in `local'","/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'","/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:43:in `block in call'","/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.1.7/lib/active_support/execution_wrapper.rb:85:in `wrap'","/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.1.7/lib/active_support/reloader.rb:68:in `block in wrap'","/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.1.7/lib/active_support/execution_wrapper.rb:85:in `wrap'","/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.1.7/lib/active_support/reloader.rb:67:in `wrap'","/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:42:in `call'","/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'","/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'","/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'","/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_logging/structured_logger.rb:16:in `call'","/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'","/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'","/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'","/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'","/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'","/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'","/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'","/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'","/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'","/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'","/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'"]}{"severity":"WARN","time":"2019-07-18T14:39:51.092Z","message":"{\"context\":\"Job raised exception\",\"job\":{\"class\":\"BackgroundMigrationWorker\",\"args\":[\"Gitlab::BackgroundMigration::PruneOrphanedGeoEvents\",\"geo_repositories_changed_events\"],\"retry\":3,\"queue\":\"background_migration\",\"jid\":\"7e716ce7e40b6c6a1fc1a647\",\"created_at\":1563459332.4537215,\"correlation_id\":\"5c06ecbd-21da-4906-a35d-f3e51f4e0048\",\"enqueued_at\":1563460791.0008795,\"error_message\":\"uninitialized constant Gitlab::BackgroundMigration::PruneOrphanedGeoEvents\\nDid you mean? PruneOldEventsWorker\",\"error_class\":\"NameError\",\"failed_at\":1563459422.5637114,\"retry_count\":0},\"jobstr\":\"{\\\"class\\\":\\\"BackgroundMigrationWorker\\\",\\\"args\\\":[\\\"Gitlab::BackgroundMigration::PruneOrphanedGeoEvents\\\",\\\"geo_repositories_changed_events\\\"],\\\"retry\\\":3,\\\"queue\\\":\\\"background_migration\\\",\\\"jid\\\":\\\"7e716ce7e40b6c6a1fc1a647\\\",\\\"created_at\\\":1563459332.4537215,\\\"correlation_id\\\":\\\"5c06ecbd-21da-4906-a35d-f3e51f4e0048\\\",\\\"enqueued_at\\\":1563460791.0008795,\\\"error_message\\\":\\\"uninitialized constant Gitlab::BackgroundMigration::PruneOrphanedGeoEvents\\\\nDid you mean? PruneOldEventsWorker\\\",\\\"error_class\\\":\\\"NameError\\\",\\\"failed_at\\\":1563459422.5637114,\\\"retry_count\\\":0}\"}"}
{"severity":"WARN","time":"2019-07-18T14:39:51.092Z","message":"NameError: uninitialized constant Gitlab::BackgroundMigration::PruneOrphanedGeoEvents\nDid you mean? PruneOldEventsWorker"}
{"severity":"WARN","time":"2019-07-18T14:39:51.092Z","message":"/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:81:in `const_get'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:81:in `migration_class_for'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:58:in `perform'\n/opt/gitlab/embedded/service/gitlab-rails/app/workers/background_migration_worker.rb:26:in `perform'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/correlation_logger.rb:10:in `block in call'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.3.0/lib/labkit/correlation/correlation_id.rb:18:in `use_id'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/correlation_logger.rb:9:in `call'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:8:in `call'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/memory_killer.rb:18:in `call'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sentry-raven-2.9.0/lib/raven/integrations/sidekiq.rb:9:in `call'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:43:in `block in call'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.1.7/lib/active_support/execution_wrapper.rb:85:in `wrap'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.1.7/lib/active_support/reloader.rb:68:in `block in wrap'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.1.7/lib/active_support/execution_wrapper.rb:85:in `wrap'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.1.7/lib/active_support/reloader.rb:67:in `wrap'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:42:in `call'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_logging/structured_logger.rb:16:in `call'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'\n/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'"}
Edited by Stan Hu