Skip to content

Track elasticsearch_timed_out_count for all Elasticsearch requests in log

Dylan Griffith requested to merge 301146-log-timeouts-from-elasticsearch into master

What does this MR do?

Since we've recently introduced timeouts in !53435 (merged) we need some way to track this in logs. The reason it needs special handling is that Elasticsearch timeouts by default will just return early and return partial results. For our use case this is what we want as the user still gets something but we don't starve Elasticsearch resources but this means there is no error propagated in the stack or the logs for us to measure so we need to explicitly log this detail that is returned from Elasticsearch.

Screenshots (strongly suggested)

{"method":"GET","path":"/search/count","format":"json","controller":"SearchController","action":"count","status":200,"time":"2021-02-11T06:28:55.473Z","params":[{"key":"utf8","value":"✓"},{"key":"search","value":"[FILTERED]"},{"key":"group_id","value":""},{"key":"project_id","value":""},{"key":"snippets","value":"false"},{"key":"repository_ref","value":""},{"key":"scope","value":"wiki_blobs"}],"remote_ip":"127.0.0.1","user_id":1,"username":"root","ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36","correlation_id":"01EY7WEEJ15QTRYEARMRSPHCJN","meta.user":"root","meta.caller_id":"SearchController#count","meta.remote_ip":"127.0.0.1","meta.feature_category":"global_search","meta.search.group_id":"","meta.search.project_id":"","meta.search.scope":"wiki_blobs","meta.search.filters.confidential":null,"meta.search.filters.state":null,"meta.search.force_search_results":null,"elasticsearch_calls":1,"elasticsearch_duration_s":0.012702,"elasticsearch_timed_out_count":0,"db_count":2,"db_write_count":0,"db_cached_count":0,"external_http_count":1,"external_http_duration_s":0.00879903700115392,"cpu_s":0.480161,"queue_duration_s":0.964331,"db_duration_s":0.00218,"view_duration_s":0.00023,"duration_s":0.03873}

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Related to #301146 (closed)

Edited by Dylan Griffith

Merge request reports