Errors during indexing gitlab-ee repo
as part of setting up elk indexing for staging (see: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6537 ) I manually triggered indexing of repos in gitlab-org
namespace using a rake task. Until gitlab-ee
, all repos were succesfully indexed. See below for the trace
[2019-04-10 11:18:41] I, [2019-04-10T11:18:41.953077 #1050] INFO -- : Indexing GitLab.org / GitLab Enterprise Edition (ID=278964)...
[2019-04-10 11:18:57] W, [2019-04-10T11:18:41.992849 #1050] WARN -- : panic: runtime error: invalid memory address or nil pointer dereference
[2019-04-10 11:18:57] [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x9e1163]
[2019-04-10 11:18:57]
[2019-04-10 11:18:57] goroutine 1 [running]:
[2019-04-10 11:18:57] gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/git.(*gitalyClient).lookUpHEAD(0xc0000a0480, 0xbf6090, 0x28, 0x0, 0x2)
[2019-04-10 11:18:57] /var/cache/omnibus/src/gitlab-elasticsearch-indexer/.GOPATH/src/gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/git/gitaly.go:194 +0x193
[2019-04-10 11:18:57] gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/git.NewGitalyClient(0xc0000ca0a0, 0xc00003e149, 0x28, 0x0, 0x0, 0x4663d8, 0x14cedd0, 0xc0000306c0)
[2019-04-10 11:18:57] /var/cache/omnibus/src/gitlab-elasticsearch-indexer/.GOPATH/src/gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/git/gitaly.go:88 +0x404
[2019-04-10 11:18:57] gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/git.NewGitalyClientFromEnv(0x7ffe75ca645a, 0x52, 0xc00003e149, 0x28, 0x0, 0x0, 0x14b1c80, 0xc00013fea0, 0xa1f38a)
[2019-04-10 11:18:57] /var/cache/omnibus/src/gitlab-elasticsearch-indexer/.GOPATH/src/gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/git/gitaly.go:109 +0x1af
[2019-04-10 11:18:57] main.main()
[2019-04-10 11:18:57] /var/cache/omnibus/src/gitlab-elasticsearch-indexer/.GOPATH/src/gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/main.go:26 +0x19e
[2019-04-10 11:18:57] indexing GitLab.org / GitLab Enterprise Edition (ID=278964), trace - ["/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/elastic/indexer.rb:88:in `run_indexer!'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:163:in `run_indexer!'", "/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/elastic/indexer.rb:51:in `run'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:163:in `run'", "/opt/gitlab/embedded/service/gitlab-rails/ee/app/workers/elastic_batch_project_indexer_worker.rb:31:in `run_indexer'", "/opt/gitlab/embedded/service/gitlab-rails/ee/app/workers/elastic_batch_project_indexer_worker.rb:16:in `block in perform'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/batches.rb:59:in `block (2 levels) in find_each'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/batches.rb:59:in `each'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/batches.rb:59:in `block in find_each'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/batches.rb:121:in `block in find_in_batches'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/batches.rb:214:in `block in in_batches'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/batches.rb:198:in `loop'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/batches.rb:198:in `in_batches'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/batches.rb:120:in `find_in_batches'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/batches.rb:58:in `find_each'", "/opt/gitlab/embedded/service/gitlab-rails/ee/app/workers/elastic_batch_project_indexer_worker.rb:16:in `perform'", "/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/gitlab/elastic.rake:44:in `block (4 levels) in <top (required)>'", "/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/gitlab/elastic.rake:196:in `block in project_id_batches'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/batches.rb:214:in `block in in_batches'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/batches.rb:198:in `loop'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/batches.rb:198:in `in_batches'", "/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/gitlab/elastic.rake:193:in `project_id_batches'", "/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/gitlab/elastic.rake:43:in `block (3 levels) in <top (required)>'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `block in execute'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `each'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `execute'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:214:in `block in invoke_with_call_chain'", "/opt/gitlab/embedded/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:194:in `invoke_with_call_chain'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:183:in `invoke'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:160:in `invoke_task'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `block (2 levels) in top_level'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `each'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `block in top_level'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:125:in `run_with_threads'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:110:in `top_level'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:83:in `block in run'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:186:in `standard_exception_handling'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:80:in `run'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'", "/opt/gitlab/embedded/bin/rake:23:in `load'", "/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'", "/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'", "/opt/gitlab/embedded/bin/bundle:23:in `load'", "/opt/gitlab/embedded/bin/bundle:23:in `<main>'"]
Edited by Valery Sizov