Error adding or updating issue, adding or editing comment, or updating merge request
Summary
Since we upgraded our self managed GitLab instance to EE we've been seeing errors all over the place.
Our GitLab instance contains hundreds of projects so performing a clean installation and start from scratch is out of the question.
Steps to reproduce
The easiest to reproduce is updating an issue. Just change the issue description and save it.
What is the current bug behavior?
A message will be displayed telling the user something went wrong: Error updating issue. Request failed with status code 500. But the issue is in fact updated with the changes. After reloading the issue in the browser or viewing the issue on another device the changed contents are visible as usual.
The funny thing is that these errors started appearing after our upgrade to EE. And after seeing the "Faraday::ConnectionFailed: Connection refused" message in the logs we found out that Elasticsearch was enabled in the GitLab configuration, even though it's not installed. I am the administrator of the instance and have not configured the instance to use elasticsearch, wich adds to my confusion.
What is the expected correct behavior?
As before the upgrade, managing issues, comments and merge requests should work without errors.
Relevant logs and/or screenshots
I've manage to find this in the logs when monitoring gitlab-ctl tail:
==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"PUT","path":"/xxx/xxx/-/issues/877.json","format":"json","controller":"Projects::IssuesController","action":"update","status":500,"time":"2021-04-15T08:19:32.055Z","params":[{"key":"title","value":"[FILTERED]"},{"key":"description","value":"[FILTERED]"},{"key":"lockedWarningVisible","value":false},{"key":"updateLoading","value":true},{"key":"lock_version","value":0},{"key":"issuableTemplates","value":[{"name":"Bug","id":"Bug","key":"[FILTERED]","project_id":48},{"name":"Enhancement","id":"Enhancement","key":"[FILTERED]","project_id":48},{"name":"Feature","id":"Feature","key":"[FILTERED]","project_id":48},{"name":"Integration Test","id":"Integration Test","key":"[FILTERED]","project_id":48}]},{"key":"namespace_id","value":"ptc"},{"key":"project_id","value":"werkstap"},{"key":"id","value":"877"},{"key":"issue","value":{"title":"[FILTERED]","description":"[FILTERED]","lock_version":0}}],"remote_ip":"xxx.xxx.xxx.xxx","user_id":2,"username":"bob","ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36","correlation_id":"01F3A9T9E0D0M794THA4BQSMYG","meta.user":"bob","meta.project":"xxx/xxx","meta.root_namespace":"ptc","meta.caller_id":"Projects::IssuesController#update","meta.remote_ip":"xxx.xxx.xxx.xxx","meta.feature_category":"issue_tracking","meta.client_id":"user/2","redis_calls":4,"redis_duration_s":0.002931,"redis_read_bytes":390,"redis_write_bytes":988,"redis_cache_calls":3,"redis_cache_duration_s":0.002104,"redis_cache_read_bytes":7,"redis_cache_write_bytes":233,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000827,"redis_shared_state_read_bytes":383,"redis_shared_state_write_bytes":755,"elasticsearch_calls":1,"elasticsearch_duration_s":0.007971,"elasticsearch_timed_out_count":0,"db_count":24,"db_write_count":2,"db_cached_count":3,"external_http_count":4,"external_http_duration_s":0.005193396471440792,"rack_attack_redis_count":1,"rack_attack_redis_duration_s":0.0018823621794581413,"cpu_s":0.131115,"mem_objects":32688,"mem_bytes":2554480,"mem_mallocs":7797,"queue_duration_s":0.020406,"exception.class":"Faraday::ConnectionFailed","exception.message":"Connection refused - Connection refused - connect(2) for \"localhost\" port 9200 (localhost:9200)","exception.backtrace":["lib/gitlab/instrumentation/elasticsearch_transport.rb:12:in `perform_request'","ee/app/models/elastic/migration_record.rb:35:in `load_from_index'","ee/app/services/elastic/data_migration_service.rb:41:in `migration_has_finished_uncached?'","ee/app/services/elastic/data_migration_service.rb:34:in `block in migration_has_finished?'","ee/app/services/elastic/data_migration_service.rb:33:in `migration_has_finished?'","ee/app/models/concerns/elastic/application_versioned_search.rb:94:in `use_separate_indices?'","ee/app/models/concerns/elastic/application_versioned_search.rb:11:in `__elasticsearch__'","ee/app/models/concerns/elastic/application_versioned_search.rb:37:in `es_id'","ee/lib/gitlab/elastic/document_reference.rb:66:in `serialize_record'","ee/lib/gitlab/elastic/document_reference.rb:57:in `serialize'","ee/app/services/elastic/process_bookkeeping_service.rb:27:in `block in track!'","ee/app/services/elastic/process_bookkeeping_service.rb:27:in `map!'","ee/app/services/elastic/process_bookkeeping_service.rb:27:in `track!'","ee/app/models/concerns/elastic/application_versioned_search.rb:59:in `maintain_elasticsearch_update'","ee/app/models/ee/issue.rb:138:in `maintain_elasticsearch_update'","config/initializers/forbid_sidekiq_in_transactions.rb:52:in `block in committed!'","config/initializers/forbid_sidekiq_in_transactions.rb:10:in `skipping_transaction_check'","config/initializers/forbid_sidekiq_in_transactions.rb:52:in `committed!'","lib/gitlab/database.rb:342:in `block in transaction'","lib/gitlab/database.rb:341:in `transaction'","app/services/issuable_base_service.rb:214:in `update'","app/services/issues/update_service.rb:20:in `update'","lib/gitlab/metrics/instrumentation.rb:160:in `block in update'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `update'","app/services/issues/update_service.rb:14:in `execute'","ee/app/services/ee/issues/update_service.rb:23:in `execute'","app/controllers/concerns/issuable_actions.rb:30:in `update'","ee/lib/gitlab/ip_address_state.rb:10:in `with'","ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'","app/controllers/application_controller.rb:486:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:477:in `set_session_storage'","lib/gitlab/i18n.rb:73:in `with_locale'","lib/gitlab/i18n.rb:79:in `with_user_locale'","app/controllers/application_controller.rb:471:in `set_locale'","app/controllers/application_controller.rb:464:in `block in set_current_context'","lib/gitlab/application_context.rb:63:in `block in use'","lib/gitlab/application_context.rb:63:in `use'","lib/gitlab/application_context.rb:24:in `with_context'","app/controllers/application_controller.rb:455:in `set_current_context'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/transaction.rb:56:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:172:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"db_duration_s":0.02125,"view_duration_s":0.0,"duration_s":0.10982}
==> /var/log/gitlab/gitlab-rails/production.log <==
Faraday::ConnectionFailed (Connection refused - Connection refused - connect(2) for "localhost" port 9200 (localhost:9200)):
lib/gitlab/instrumentation/elasticsearch_transport.rb:12:in `perform_request'
ee/app/models/elastic/migration_record.rb:35:in `load_from_index'
ee/app/services/elastic/data_migration_service.rb:41:in `migration_has_finished_uncached?'
ee/app/services/elastic/data_migration_service.rb:34:in `block in migration_has_finished?'
ee/app/services/elastic/data_migration_service.rb:33:in `migration_has_finished?'
ee/app/models/concerns/elastic/application_versioned_search.rb:94:in `use_separate_indices?'
ee/app/models/concerns/elastic/application_versioned_search.rb:11:in `__elasticsearch__'
ee/app/models/concerns/elastic/application_versioned_search.rb:37:in `es_id'
ee/lib/gitlab/elastic/document_reference.rb:66:in `serialize_record'
ee/lib/gitlab/elastic/document_reference.rb:57:in `serialize'
ee/app/services/elastic/process_bookkeeping_service.rb:27:in `block in track!'
ee/app/services/elastic/process_bookkeeping_service.rb:27:in `map!'
ee/app/services/elastic/process_bookkeeping_service.rb:27:in `track!'
ee/app/models/concerns/elastic/application_versioned_search.rb:59:in `maintain_elasticsearch_update'
ee/app/models/ee/issue.rb:138:in `maintain_elasticsearch_update'
config/initializers/forbid_sidekiq_in_transactions.rb:52:in `block in committed!'
config/initializers/forbid_sidekiq_in_transactions.rb:10:in `skipping_transaction_check'
config/initializers/forbid_sidekiq_in_transactions.rb:52:in `committed!'
lib/gitlab/database.rb:342:in `block in transaction'
lib/gitlab/database.rb:341:in `transaction'
app/services/issuable_base_service.rb:214:in `update'
app/services/issues/update_service.rb:20:in `update'
lib/gitlab/metrics/instrumentation.rb:160:in `block in update'
lib/gitlab/metrics/method_call.rb:27:in `measure'
lib/gitlab/metrics/instrumentation.rb:160:in `update'
app/services/issues/update_service.rb:14:in `execute'
ee/app/services/ee/issues/update_service.rb:23:in `execute'
app/controllers/concerns/issuable_actions.rb:30:in `update'
ee/lib/gitlab/ip_address_state.rb:10:in `with'
ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'
app/controllers/application_controller.rb:486:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:477:in `set_session_storage'
lib/gitlab/i18n.rb:73:in `with_locale'
lib/gitlab/i18n.rb:79:in `with_user_locale'
app/controllers/application_controller.rb:471:in `set_locale'
app/controllers/application_controller.rb:464:in `block in set_current_context'
lib/gitlab/application_context.rb:63:in `block in use'
lib/gitlab/application_context.rb:63:in `use'
lib/gitlab/application_context.rb:24:in `with_context'
app/controllers/application_controller.rb:455:in `set_current_context'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/transaction.rb:56:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:172:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'
==> /var/log/gitlab/gitlab-workhorse/current <==
{"content_type":"text/html; charset=utf-8","correlation_id":"01F3A9T9E0D0M794THA4BQSMYG","duration_ms":157,"host":"xxx","level":"info","method":"PUT","msg":"access","proto":"HTTP/1.1","referrer":"https://vcs.unitracer.io/ptc/werkstap/-/issues/877","remote_addr":"xxx.xxx.xxx.xxx:0","remote_ip":"xxx.xxx.xxx.xxx","route":"","status":500,"system":"http","time":"2021-04-15T10:19:32+02:00","ttfb_ms":157,"uri":"/xxx/xxx/-/issues/877.json","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36","written_bytes":2926}
==> /var/log/gitlab/nginx/gitlab_access.log <==
xxx.xxx.xxx.xxx - - [15/Apr/2021:10:19:32 +0200] "PUT /xxx/xxx/-/issues/877.json HTTP/2.0" 500 2926 "https://xxx/xxx/xxx/-/issues/877" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36" -Output of checks
This bug happens in our self-managed GitLab instance.
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Debian 10 Proxy: no Current User: git Using RVM: no Ruby Version: 2.7.2p137 Gem Version: 3.1.4 Bundler Version:2.1.4 Rake Version: 13.0.3 Redis Version: 6.0.10 Git Version: 2.29.0 Sidekiq Version:5.2.9 Go Version: unknown GitLab information Version: 13.10.3-ee Revision: db2e358dba4 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 12.6 URL: https://vcs.unitracer.io HTTP Clone URL: https://vcs.unitracer.io/some-group/some-project.git SSH Clone URL: git@vcs.unitracer.io:some-group/some-project.git Elasticsearch: yes Geo: no Using LDAP: yes Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 13.17.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.17.0 ? ... OK (13.17.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 14 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 3/2 ... yes 3/5 ... yes 4/19 ... yes 2/20 ... yes 14/21 ... yes 3/22 ... yes 3/24 ... yes 3/25 ... yes 3/26 ... yes 3/28 ... yes 18/32 ... yes 2/34 ... yes 2/35 ... yes 2/36 ... yes 21/37 ... yes 21/38 ... yes 21/39 ... yes 3/40 ... yes 10/41 ... yes 20/42 ... yes 7/43 ... yes 8/44 ... yes 22/46 ... yes 23/47 ... yes 5/48 ... yes 3/50 ... yes 5/51 ... yes 27/53 ... yes 27/54 ... yes 27/55 ... yes 27/56 ... yes 27/57 ... yes 27/58 ... yes 27/59 ... yes 27/60 ... yes 27/61 ... yes 27/62 ... yes 27/63 ... yes 27/64 ... yes 27/65 ... yes 27/66 ... yes 8/69 ... yes 14/71 ... yes 2/72 ... yes 10/73 ... yes 4/81 ... yes 8/87 ... yes 30/92 ... yes 8/93 ... yes 8/94 ... yes 7/95 ... yes 23/96 ... yes 3/98 ... yes 34/99 ... yes 34/100 ... yes 34/101 ... yes 34/102 ... yes 36/103 ... yes 35/104 ... yes 37/105 ... yes 36/106 ... yes 38/107 ... yes 38/108 ... yes 38/109 ... yes 38/110 ... yes 39/111 ... yes 40/112 ... yes 40/113 ... yes 4/114 ... yes 39/115 ... yes 39/116 ... yes 41/117 ... yes 3/118 ... yes 20/119 ... yes 2/120 ... yes 3/122 ... yes 3/123 ... yes 3/124 ... yes 34/125 ... yes 34/126 ... yes 20/127 ... yes 4/128 ... yes 3/129 ... yes 34/130 ... yes 21/131 ... yes 8/132 ... yes 5/133 ... yes 44/134 ... yes 23/135 ... yes 45/136 ... yes 20/137 ... yes 49/139 ... yes 46/140 ... yes 4/141 ... yes 2/142 ... yes 4/143 ... yes 51/144 ... yes 2/145 ... yes 46/148 ... yes 4/149 ... yes 52/150 ... yes 53/151 ... yes 53/153 ... yes 4/155 ... yes 5/156 ... yes 3/157 ... yes 55/159 ... yes 3/160 ... yes 57/161 ... yes 59/162 ... yes 8/163 ... yes 4/164 ... yes 44/165 ... yes 5/166 ... yes 57/167 ... yes 57/168 ... yes 2/169 ... yes 2/170 ... yes 2/171 ... yes 63/172 ... yes 4/173 ... yes 5/174 ... yes 3/175 ... yes 3/176 ... yes 3/178 ... yes 14/179 ... yes 3/180 ... yes 67/181 ... yes 68/182 ... yes 68/183 ... yes 68/184 ... yes 68/185 ... yes 68/186 ... yes 68/187 ... yes 68/188 ... yes 82/189 ... yes 81/190 ... yes 83/191 ... yes 80/192 ... yes 4/193 ... yes 14/194 ... yes 80/195 ... yes 82/196 ... yes 83/197 ... yes 83/198 ... yes 83/199 ... yes 68/200 ... yes 68/201 ... yes 68/202 ... yes 68/203 ... yes 68/204 ... yes 3/205 ... yes 5/206 ... yes 23/207 ... yes 5/208 ... yes 20/209 ... yes 3/210 ... yes 98/211 ... yes 98/212 ... yes 100/213 ... yes 100/214 ... yes 23/215 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.2) Git version >= 2.29.0 ? ... yes (2.29.0) Git user has default SSH configuration? ... yes Active users: ... 5 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... no Try fixing it: Please migrate all projects to hashed storage as legacy storage is deprecated in 13.0 and support will be removed in 14.0. For more information see: doc/administration/repository_storage_types.md Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... Exception: Connection refused - Connection refused - connect(2) for "::1" port 9200 (localhost:9200)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
Unknown
Possible workarounds
I see two options:
- Install and configure elasticsearch.
- Disable usage of elasticsearch.
But elasticsearch should never have been enabled in the first place.
Release Notes
Add as Bug Fix