Increase timeout for importing Jira issues.
Summary
While using Jira import. Customer is experiencing timeout while importing Jira issues. From our discussion on slack.. It seems like the timeout is hard-coded and cannot be configured.
Summary of findings.
-
We call our Gitlab::HTTP wrapper from the Jira importer at https://gitlab.com/gitlab-org/gitlab/blob/1f9cea0234129104ab502e8a2d1b75b73601eb27/lib/gitlab/jira/http_client.rb#L38, and aren't passing any custom timeouts.
-
Inside Gitlab::HTTP we apply the default timeouts at https://gitlab.com/gitlab-org/gitlab/blob/1f9cea0234129104ab502e8a2d1b75b73601eb27/lib/gitlab/http.rb#L42, which are defined further up at https://gitlab.com/gitlab-org/gitlab/blob/1f9cea0234129104ab502e8a2d1b75b73601eb27/lib/gitlab/http.rb#L23-28.
Zendesk
https://gitlab.zendesk.com/agent/tickets/266984
https://gitlab.zendesk.com/agent/tickets/275820
Steps to reproduce
Setup Jira issue and import a large project.
Example Project
What is the current bug behavior?
What is the expected correct behavior?
Relevant logs and/or screenshots
{
"severity": "ERROR",
"time": "2022-02-15T13:40:31.344Z",
"correlation_id": "01FVYSSXJG5EYXC9MKZN6DCF9C",
"exception.class": "JIRA::HTTPError",
"exception.message": "{\"message\":\"null for uri: https://xxxx/rest/api/2/label?maxResults=500&startAt=0\",\"status-code\":404}",
"exception.backtrace": [
"lib/gitlab/jira/http_client.rb:15:in `request'",
"lib/gitlab/jira_import/labels_importer.rb:42:in `process_jira_page'",
"lib/gitlab/jira_import/labels_importer.rb:32:in `block in import_jira_labels'",
"lib/gitlab/jira_import/labels_importer.rb:31:in `loop'",
"lib/gitlab/jira_import/labels_importer.rb:31:in `import_jira_labels'",
"lib/gitlab/jira_import/labels_importer.rb:17:in `execute'",
"app/workers/gitlab/jira_import/stage/import_labels_worker.rb:12:in `import'",
"app/workers/concerns/gitlab/jira_import/import_worker.rb:21:in `perform'",
"lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:24:in `call'",
"lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in `perform'",
"lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:58:in `perform'",
"lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call'",
"lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context'",
"lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in `block in call'",
"lib/gitlab/application_context.rb:74:in `block in use'",
"lib/gitlab/application_context.rb:74:in `use'",
"lib/gitlab/application_context.rb:27:in `with_context'",
"lib/gitlab/sidekiq_middleware/worker_context/server.rb:15:in `call'",
"lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call'",
"lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call'",
"lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call'",
"lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call'",
"lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'",
"lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call'",
"lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call'",
"lib/gitlab/with_request_store.rb:17:in `enabling_request_store'",
"lib/gitlab/with_request_store.rb:10:in `with_request_store'",
"lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call'",
"lib/gitlab/sidekiq_middleware/arguments_logger.rb:8:in `call'",
"lib/gitlab/sidekiq_middleware/server_metrics.rb:66:in `block in call'",
"lib/gitlab/sidekiq_middleware/server_metrics.rb:89:in `block in instrument'",
"lib/gitlab/metrics/background_transaction.rb:33:in `run'",
"lib/gitlab/sidekiq_middleware/server_metrics.rb:89:in `instrument'",
"lib/gitlab/sidekiq_middleware/server_metrics.rb:65:in `call'",
"lib/gitlab/sidekiq_middleware/monitor.rb:8:in `block in call'",
"lib/gitlab/sidekiq_daemon/monitor.rb:49:in `within_job'",
This later results to
{
"severity": "ERROR",
"time": "2022-02-15T13:41:12.307Z",
"correlation_id": "01FVYSSXJG5EYXC9MKZN6DCF9C",
"exception.class": "Net::ReadTimeout",
"exception.message": "Net::ReadTimeout with #<TCPSocket:(closed)>",
"exception.backtrace": [
"lib/gitlab/http.rb:56:in `perform_request'",
"lib/gitlab/jira/http_client.rb:38:in `public_send'",
"lib/gitlab/jira/http_client.rb:38:in `make_request'",
"lib/gitlab/jira/http_client.rb:13:in `request'",
"lib/gitlab/jira_import/issues_importer.rb:87:in `fetch_issues'",
"lib/gitlab/jira_import/issues_importer.rb:30:in `import_issues'",
"lib/gitlab/jira_import/issues_importer.rb:22:in `execute'",
"app/workers/gitlab/jira_import/stage/import_issues_worker.rb:12:in `import'",
"app/workers/concerns/gitlab/jira_import/import_worker.rb:21:in `perform'",
"lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:24:in `call'",
"lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in `perform'",
"lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:58:in `perform'",
"lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call'",
"lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context'",
"lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in `block in call'",
"lib/gitlab/application_context.rb:74:in `block in use'",
"lib/gitlab/application_context.rb:74:in `use'",
"lib/gitlab/application_context.rb:27:in `with_context'",
"lib/gitlab/sidekiq_middleware/worker_context/server.rb:15:in `call'",
"lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call'",
"lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call'",
"lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call'",
"lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call'",
"lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'",
"lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call'",
"lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call'",
"lib/gitlab/with_request_store.rb:17:in `enabling_request_store'",
"lib/gitlab/with_request_store.rb:10:in `with_request_store'",
"lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call'",
"lib/gitlab/sidekiq_middleware/arguments_logger.rb:8:in `call'",
"lib/gitlab/sidekiq_middleware/server_metrics.rb:66:in `block in call'",
"lib/gitlab/sidekiq_middleware/server_metrics.rb:89:in `block in instrument'",
"lib/gitlab/metrics/background_transaction.rb:33:in `run'",
"lib/gitlab/sidekiq_middleware/server_metrics.rb:89:in `instrument'",
"lib/gitlab/sidekiq_middleware/server_metrics.rb:65:in `call'",
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`) ``` System information System: Proxy: PROD_GITLAB_PAGES_SERVICE_PORT_HTTP_GITLAB_PAGES_PROXY: 8090 Current User: git Using RVM: no Ruby Version: 2.7.4p191 Gem Version: 3.2.19 Bundler Version:2.2.19 Rake Version: 13.0.6 Redis Version: unknown Git Version: unknown Sidekiq Version:6.2.2 Go Version: unknown GitLab information Version: 14.5.0-ee Revision: 490a8efda84 Directory: /srv/gitlab DB Adapter: PostgreSQL DB Version: 12.4 URL: https://xxxx HTTP Clone URL: https://xxxx/some-group/some-project.git SSH Clone URL: git@xxxxx:some-group/some-project.git Elasticsearch: yes Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers: saml GitLab Shell Version: 13.22.0 Repository storage paths: - default: /var/opt/gitlab/repo GitLab Shell path: /home/git/gitlab-shell Git: /usr/bin/git ```
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 13.22.0 ? ... OK (13.22.0) Running /home/git/gitlab-shell/bin/check gitlab-shell self-check failed Try fixing it: Make sure GitLab is running; Check the gitlab-shell configuration file: sudo -u git -H editor /home/git/gitlab-shell/config.yml Please fix the error above and rerun the checks. Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... no Try fixing it: sudo -u git -H RAILS_ENV=production bin/background_jobs start For more information see: doc/install/installation.md in section "Install Init Script" see log/sidekiq.log for possible errors Please fix the error above and rerun the checks. Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab App ... Git configured correctly? ... no Trying to fix error automatically. ...Failed Try fixing it: sudo -u git -H "/usr/bin/git" config --global core.autocrlf "input" For more information see: doc/install/installation.md in section "GitLab" Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Systemd unit files or init script exist? ... no Try fixing it: Install the Service For more information see: doc/install/installation.md in section "Install the Service" Please fix the error above and rerun the checks. Systemd unit files or init script up-to-date? ... can't check because of previous errors Projects have namespace: ... 5/3 ... yes 5/4 ... yes (omitted similar lines…) 5/5902 ... yes 2131/5903 ... yes 308/5904 ... yes 600/5905 ... yes 1227/5906 ... yes 218/5907 ... yes 308/5908 ... yes 411/5909 ... yes 696/5932 ... yes Redis version >= 5.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.4) Git version >= 2.33.0 ? ... no Your git bin path is "/usr/bin/git" Try fixing it: Update your git to a version >= 2.33.0 from Unknown Please fix the error above and rerun the checks. Git user has default SSH configuration? ... yes Active users: ... xxx Is authorized keys file accessible? ... skipped (authorized keys not enabled) GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... yes (7.12.0) Checking GitLab App ... Finished Checking GitLab subtasks ... Finished
(we will only investigate if the tests are passing)
Possible fixes
Increase the Jira client read timeout limit, but only within the Jira issue importer, rather than all Jira API requests (https://gitlab.com/gitlab-org/gitlab/-/issues/343174).