Sidekiq exception backtrace not showing useful lines
In https://gitlab.com/gitlab-org/gitlab-ce/issues/63115, the backtrace isn't showing the right job or the actual exception for the problem. Sentry made it possible to find the actual backtrace, but we shouldn't have to rely on that:
"error_backtrace":[
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:127:in `rescue in local'",
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:108:in `local'",
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'",
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:43:in `block in call'",
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.1.7/lib/active_support/execution_wrapper.rb:85:in `wrap'",
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.1.7/lib/active_support/reloader.rb:68:in `block in wrap'",
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.1.7/lib/active_support/execution_wrapper.rb:85:in `wrap'",
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.1.7/lib/active_support/reloader.rb:67:in `wrap'",
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:42:in `call'",
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'",
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'",
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'",
"/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_logging/structured_logger.rb:16:in `call'",
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'",
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'",
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'",
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'",
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'",
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'",
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'",
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'",
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'",
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'",
"/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'"
]Note we're missing the exact line that caused the problem:
TypeError: type mismatch: Array given
from gitlab/ci/pipeline/expression/lexeme/matches.rb:20:in `index'
from gitlab/ci/pipeline/expression/lexeme/matches.rb:20:in `evaluate'
from gitlab/ci/pipeline/expression/statement.rb:22:in `evaluate'
from gitlab/ci/pipeline/expression/statement.rb:26:in `truthful?'
from gitlab/ci/build/policy/variables.rb:20:in `any?'
from gitlab/ci/build/policy/variables.rb:20:in `satisfied_by?'
from gitlab/ci/pipeline/seed/build.rb:25:in `block (2 levels) in included?'
from gitlab/ci/pipeline/seed/build.rb:25:in `none?'
from gitlab/ci/pipeline/seed/build.rb:25:in `block in included?'
from gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
from gitlab/ci/pipeline/seed/build.rb:23:in `included?'
from gitlab/ci/pipeline/seed/stage.rb:31:in `select'
from gitlab/ci/pipeline/seed/stage.rb:31:in `block in seeds'
from gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
from gitlab/ci/pipeline/seed/stage.rb:30:in `seeds'
from gitlab/ci/pipeline/seed/stage.rb:36:in `included?'
from ci/pipeline.rb:502:in `select'
from ci/pipeline.rb:502:in `block in stage_seeds'
from gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
from ci/pipeline.rb:497:in `stage_seeds'
from gitlab/metrics/instrumentation.rb:161:in `block in stage_seeds'
from gitlab/metrics/method_call.rb:36:in `measure'
from gitlab/metrics/instrumentation.rb:161:in `stage_seeds'
from gitlab/ci/pipeline/chain/populate.rb:28:in `perform!'
from gitlab/ci/pipeline/chain/sequence.rb:19:in `block in build!'
from gitlab/ci/pipeline/chain/sequence.rb:16:in `each'
from gitlab/ci/pipeline/chain/sequence.rb:16:in `build!'Edited by Stan Hu