Internal Server Error (500) when attempting gradle publish to Maven package registry
Summary
customer Receiving 500 error when attempting to publish a package via Gradle to the project's Maven package registry.
Steps to reproduce
Problem is intermittent.
Relevant logs and/or screenshots
https://sentry.gitlab.net/gitlab/gitlabcom/issues/4103474/?referrer=gitlab_plugin
PG::QueryCanceled: ERROR: canceling statement due to statement timeout
lib/gitlab/database/load_balancing/connection_proxy.rb:104:in `public_send'
connection.public_send(...)
lib/gitlab/database/load_balancing/connection_proxy.rb:104:in `block in read_using_load_balancer'
connection.public_send(...)
lib/gitlab/database/load_balancing/load_balancer.rb:55:in `read'
return yield connection
lib/gitlab/database/load_balancing/connection_proxy.rb:103:in `read_using_load_balancer'
@load_balancer.read do |connection|
lib/gitlab/database/load_balancing/connection_proxy.rb:48:in `select_all'
read_using_load_balancer(:select_all, arel, name, binds)
...
(204 additional frame(s) were not displayed)
ActiveRecord::QueryCanceled: PG::QueryCanceled: ERROR: canceling statement due to statement timeout
JSON snippet from logs
"correlation_id": "f09e1635abfae58005e1d8e8f8ae199e",
"cpu_s": 0.164199,
"db_count": 15,
"db_duration_s": 15.01859,
"duration_s": 15.5008,
"exception.backtrace": [
"lib/gitlab/database/load_balancing/connection_proxy.rb:104:in `public_send'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:104:in `block in read_using_load_balancer'",
"lib/gitlab/database/load_balancing/load_balancer.rb:55:in `read'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:103:in `read_using_load_balancer'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:48:in `select_all'",
"app/finders/packages/maven/package_finder.rb:7:in `execute'",
"lib/api/maven_packages.rb:117:in `fetch_package'",
"lib/api/maven_packages.rb:274:in `block (2 levels) in <class:MavenPackages>'",
"ee/lib/gitlab/middleware/ip_restrictor.rb:14:in `block in call'",
"ee/lib/gitlab/ip_address_state.rb:10:in `with'",
"ee/lib/gitlab/middleware/ip_restrictor.rb:13:in `call'",
"lib/api/api_guard.rb:222:in `call'",
"ee/lib/omni_auth/strategies/group_saml.rb:41:in `other_phase'",
"lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'",
"lib/gitlab/middleware/memory_report.rb:13:in `call'",
"lib/gitlab/middleware/speedscope.rb:13:in `call'",
"lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
"lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'",
"lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'",
"lib/gitlab/metrics/web_transaction.rb:46:in `run'",
"lib/gitlab/metrics/rack_middleware.rb:16:in `call'",
"lib/gitlab/jira/middleware.rb:19:in `call'",
"lib/gitlab/middleware/go.rb:20:in `call'",
"lib/gitlab/etag_caching/middleware.rb:21:in `call'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'",
"lib/gitlab/database/query_analyzer.rb:37:in `within'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `call'",
"lib/gitlab/middleware/multipart.rb:173:in `call'",
"lib/gitlab/middleware/read_only/controller.rb:50:in `call'",
"lib/gitlab/middleware/read_only.rb:18:in `call'",
"lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
"lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
"lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
"lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
"lib/gitlab/middleware/request_context.rb:21:in `call'",
"lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",
"config/initializers/fix_local_cache_middleware.rb:11:in `call'",
"lib/gitlab/middleware/compressed_json.rb:37:in `call'",
"lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
"lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
"lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'",
"lib/gitlab/middleware/release_env.rb:13:in `call'"
],
"exception.cause_class": "PG::QueryCanceled",
"exception.class": "ActiveRecord::QueryCanceled",
"exception.message": "PG::QueryCanceled: ERROR: canceling statement due to statement timeout\n",
"level": "error",
"mem_bytes": 9862634,
"mem_mallocs": 21653,
"mem_objects": 90169,
"mem_total_bytes": 13469394,
"meta.caller_id": "GET /api/:version/projects/:id/packages/maven/*path/:file_name",
"method": "GET",
"params": [],
"status": 500,
Output of checks
/label reproduced on GitLab.com
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)
Possible fixes
Edited by Matthew Badeau