500 in canary and production on GET /api/v4/groups/:group_id API call | 100% CPU spike
Summary
A few end-to-end tests are failing intermittently in canary production and canary with 500 on GET /api/v4/groups/:group_id
API call
Jobs:
- Production: https://ops.gitlab.net/gitlab-org/quality/production/-/jobs/1406665
- Canary: https://ops.gitlab.net/gitlab-org/quality/canary/-/jobs/1406660
Stack trace
project = Resource::Project.fabricate_via_api! do |project|
project.name = 'project-to-test-mention'
project.visibility = 'private'
end
QA::Resource::ApiFabricator::InternalServerError:
Failed to GET https://gitlab.com/api/v4/groups/gitlab-qa-sandbox-group%2Fqa-test-2020-07-08-12-09-33-5e52851d3e538db6?private_token=[****] - (500): `{"message":"500 Internal Server Error"}`.
# ./qa/resource/api_fabricator.rb:90:in `api_get_from'
# ./qa/resource/api_fabricator.rb:82:in `api_get'
Logs from Kibana:
https://log.gprd.gitlab.net/goto/6a685b3fc68929cd31b3a8271d3f51df (Thanks @smcgivern)
Sentry logs
Rails Stacktrace:
"exception.class": "GRPC::DeadlineExceeded",
"exception.message": "4:Deadline Exceeded",
"lib/gitlab/gitaly_client.rb:179:in `execute'",
"lib/gitlab/gitaly_client/call.rb:18:in `block in call'",
"lib/gitlab/gitaly_client/call.rb:55:in `recording_request'",
"lib/gitlab/gitaly_client/call.rb:17:in `call'",
"lib/gitlab/gitaly_client.rb:169:in `call'",
"lib/gitlab/gitaly_client/ref_service.rb:39:in `default_branch_name'",
"lib/gitlab/git/repository.rb:90:in `root_ref'",
"app/models/repository.rb:510:in `root_ref'",
"lib/gitlab/repository_cache_adapter.rb:84:in `block (2 levels) in cache_method_asymmetrically'",
"lib/gitlab/repository_cache.rb:44:in `fetch_without_caching_false'",
"lib/gitlab/repository_cache_adapter.rb:179:in `block (2 levels) in cache_method_output_asymmetrically'",
"lib/gitlab/safe_request_store.rb:12:in `fetch'",
"lib/gitlab/repository_cache.rb:25:in `fetch'",
"lib/gitlab/repository_cache_adapter.rb:178:in `block in cache_method_output_asymmetrically'",
"lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'",
"lib/gitlab/repository_cache_adapter.rb:192:in `block in memoize_method_output'",
"lib/gitlab/repository_cache_adapter.rb:201:in `no_repository_fallback'",
"lib/gitlab/repository_cache_adapter.rb:191:in `memoize_method_output'",
"lib/gitlab/repository_cache_adapter.rb:177:in `cache_method_output_asymmetrically'",
"lib/gitlab/repository_cache_adapter.rb:83:in `block in cache_method_asymmetrically'",
"app/models/concerns/has_repository.rb:79:in `default_branch'",
"ee/lib/gitlab/ip_address_state.rb:10:in `with'",
"ee/lib/omni_auth/strategies/group_saml.rb:41:in `other_phase'",
"ee/lib/gitlab/jira/middleware.rb:19:in `call'"
Possible fixes
Edited by Sanad Liaquat