API timeout when searching for group issues encounters a specific record
Summary
See gitlab-org/quality/triage-ops#1246 (comment 1301730403)
Hitting https://gitlab.com/api/v4/groups/9970/issues?per_page=100&labels=devops%3A%3Averify¬%5Blabels%5D=failure%3A%3Aflaky-test%2Cfailure%3A%3Anew&state=opened&page=18 times out after 60seconds. Example correlation/request id: 5dc0e87f47214b5725c903a7b0b9a8ee
Previously ages return very quickly (1 second) so I suspect there is a specific issue causing the timeout.
Steps to reproduce
Click the API link
Example Project
gitlab-org
What is the current bug behavior?
Rack::Timeout::RequestTimeoutException
Request ran for longer than 60000ms
app/models/concerns/taskable.rb:40:in `scan'
app/models/concerns/taskable.rb:40:in `get_tasks'
app/models/concerns/taskable.rb:67:in `task_list_items'
app/models/concerns/taskable.rb:76:in `tasks?'
lib/api/entities/issue.rb:13:in `block in <class:Issue>'
lib/gitlab/json.rb:130:in `dump'
lib/gitlab/json.rb:130:in `adapter_dump'
lib/gitlab/json.rb:52:in `dump'
lib/gitlab/json.rb:194:in `call'
ee/lib/gitlab/middleware/ip_restrictor.rb:14:in `block in call'
ee/lib/gitlab/ip_address_state.rb:10:in `with'
ee/lib/gitlab/middleware/ip_restrictor.rb:13:in `call'
lib/api/api_guard.rb:222:in `call'
ee/lib/omni_auth/strategies/group_saml.rb:41:in `other_phase'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/memory_report.rb:13:in `call'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/web_transaction.rb:46:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
lib/gitlab/database/query_analyzer.rb:37:in `within'
lib/gitlab/middleware/query_analyzer.rb:11:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:37:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
What is the expected correct behavior?
Return page of issues
Relevant logs and/or screenshots
5dc0e87f47214b5725c903a7b0b9a8ee
Output of checks
/label reproduced on GitLab.com
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)
Possible fixes
/cc @tkuah