Make Faraday timeout configurable for ES indexing
Description
Right now if we try to index a project with lots of files (not big in terms of size, just a lot of files and folders), we are getting a Faraday time out on the Ruby ElasticSearch Client:
Indexing project repositories...I, [2018-04-23T09:09:24.964117 #11379] INFO -- : Indexing XYZ / XYZ (ID=28844)...
W, [2018-04-23T09:14:08.070078 #11379] WARN -- : /opt/gitlab/embedded/lib/ruby/2.3.0/net/protocol.rb:176:in `rbuf_fill': Net::ReadTimeout (Faraday::TimeoutError)
from /opt/gitlab/embedded/lib/ruby/2.3.0/net/protocol.rb:154:in `readuntil'
from /opt/gitlab/embedded/lib/ruby/2.3.0/net/protocol.rb:164:in `readline'
from /opt/gitlab/embedded/lib/ruby/2.3.0/net/http/response.rb:40:in `read_status_line'
from /opt/gitlab/embedded/lib/ruby/2.3.0/net/http/response.rb:29:in `read_new'
from /opt/gitlab/embedded/lib/ruby/2.3.0/net/http.rb:1423:in `block in transport_request'
from /opt/gitlab/embedded/lib/ruby/2.3.0/net/http.rb:1420:in `catch'
from /opt/gitlab/embedded/lib/ruby/2.3.0/net/http.rb:1420:in `transport_request'
from /opt/gitlab/embedded/lib/ruby/2.3.0/net/http.rb:1393:in `request'
from /opt/gitlab/embedded/lib/ruby/2.3.0/net/http.rb:1386:in `block in request'
from /opt/gitlab/embedded/lib/ruby/2.3.0/net/http.rb:853:in `start'
from /opt/gitlab/embedded/lib/ruby/2.3.0/net/http.rb:1384:in `request'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/faraday-0.12.2/lib/faraday/adapter/net_http.rb:80:in `perform_request'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/faraday-0.12.2/lib/faraday/adapter/net_http.rb:38:in `block in call'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/faraday-0.12.2/lib/faraday/adapter/net_http.rb:85:in `with_net_http_connection'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/faraday-0.12.2/lib/faraday/adapter/net_http.rb:33:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/faraday-0.12.2/lib/faraday/rack_builder.rb:141:in `build_response'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/faraday-0.12.2/lib/faraday/connection.rb:386:in `run_request'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/elasticsearch-transport-5.0.3/lib/elasticsearch/transport/transport/http/faraday.rb:23:in `block in perform_request'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/elasticsearch-transport-5.0.3/lib/elasticsearch/transport/transport/base.rb:261:in `perform_request'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/elasticsearch-transport-5.0.3/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/elasticsearch-transport-5.0.3/lib/elasticsearch/transport/client.rb:131:in `perform_request'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/elasticsearch-api-5.0.3/lib/elasticsearch/api/actions/bulk.rb:95:in `bulk'
from /opt/gitlab/embedded/service/gitlab-rails/ee/lib/elasticsearch/git/repository.rb:112:in `perform_bulk'
from /opt/gitlab/embedded/service/gitlab-rails/ee/lib/elasticsearch/git/repository.rb:99:in `block in index_blobs'
from /opt/gitlab/embedded/service/gitlab-rails/ee/lib/elasticsearch/git/repository.rb:84:in `each'
from /opt/gitlab/embedded/service/gitlab-rails/ee/lib/elasticsearch/git/repository.rb:84:in `each_slice'
from /opt/gitlab/embedded/service/gitlab-rails/ee/lib/elasticsearch/git/repository.rb:84:in `index_blobs'
from /opt/gitlab/embedded/service/gitlab-rails/bin/elastic_repo_indexer:91:in `block in <main>'
from /opt/gitlab/embedded/lib/ruby/2.3.0/benchmark.rb:293:in `measure'
from /opt/gitlab/embedded/service/gitlab-rails/bin/elastic_repo_indexer:89:in `<main>'
XYZ / XYZ (ID=28844), trace - ["/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/elastic/indexer.rb:64:in `run_indexer!'", "/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/elastic/indexer.rb:38:in `run'", "/opt/gitlab/embedded/service/gitlab-rails/ee/app/workers/elastic_batch_project_indexer_worker.rb:23:in `run_indexer'", "/opt/gitlab/embedded/service/gitlab-rails/ee/app/workers/elastic_batch_project_indexer_worker.rb:14:in `block in perform'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `block (2 levels) in find_each'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `each'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `block in find_each'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:124:in `find_in_batches'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:50:in `find_each'", "/opt/gitlab/embedded/service/gitlab-rails/ee/app/workers/elastic_batch_project_indexer_worker.rb:14:in `perform'", "/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/gitlab/elastic.rake:39:in `block (4 levels) in <top (required)>'", "/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/storage.rake:72:in `block in project_id_batches'", "/opt/gitlab/embedded/service/gitlab-rails/config/initializers/ar5_batching.rb:34:in `block in in_batches'", "/opt/gitlab/embedded/service/gitlab-rails/config/initializers/ar5_batching.rb:18:in `loop'", "/opt/gitlab/embedded/service/gitlab-rails/config/initializers/ar5_batching.rb:18:in `in_batches'", "/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/storage.rake:69:in `project_id_batches'", "/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/gitlab/elastic.rake:38:in `block (3 levels) in <top (required)>'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:251:in `each'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:251:in `execute'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:195:in `block in invoke_with_call_chain'", "/opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:188:in `invoke_with_call_chain'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/task.rb:181:in `invoke'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:160:in `invoke_task'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:116:in `block (2 levels) in top_level'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:116:in `each'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:116:in `block in top_level'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:125:in `run_with_threads'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:110:in `top_level'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:83:in `block in run'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:186:in `standard_exception_handling'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/lib/rake/application.rb:80:in `run'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'", "/opt/gitlab/embedded/bin/rake:23:in `load'", "/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `load'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `kernel_load'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:27:in `run'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:332:in `exec'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor.rb:359:in `dispatch'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:20:in `dispatch'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/base.rb:440:in `start'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:11:in `start'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/exe/bundle:34:in `block in <top (required)>'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/friendly_errors.rb:100:in `with_friendly_errors'", "/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/exe/bundle:26:in `<top (required)>'", "/opt/gitlab/embedded/bin/bundle:23:in `load'", "/opt/gitlab/embedded/bin/bundle:23:in `<main>'"]
Proposal
This could be fixed by adding request_timeout
configurable https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/gitlab/elastic/client.rb#L13
In the end, we could having something like (extracting the values from config):
base_config = {
urls: config[:url],
randomize_hosts: true,
retry_on_failure: true,
request_timeout: 10*60
}
This worked in all the problematic cases.
Links / references
https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/gitlab/elastic/client.rb
Edited by Daniel Juarez