Skip to content

Log extra metadata about searches

Dylan Griffith requested to merge 227884-log-search-related-keywords into master

What does this MR do?

Logging these extra fields as top level keys will ensure that we can query/filter them in Kibana which will help us to answer questions about common search terms or understand usage of search within individual groups. Currently these query params are already logged but they are nested keys they are not indexed and cannot be filtered in any way in Kibana.

This change adds to the existing logic in ApplicationController which in term relies on the append_info_to_payload being invoked after a request. This metadata is finally extracted and logged by our lograge configuration.

Screenshots

{"method":"GET","path":"/search","format":"*/*","controller":"SearchController","action":"show","status":200,"unpermitted_params":["utf8","search_code","nav_source","utf8","search_code","nav_source","utf8","search_code","nav_source","utf8","search_code","nav_source","utf8","search_code","nav_source","utf8","search_code","nav_source","utf8","search_code","nav_source","utf8","search_code","nav_source"],"time":"2020-07-28T05:18:54.081Z","params":[{"key":"utf8","value":"✓"},{"key":"search","value":"hello world"},{"key":"group_id","value":"456"},{"key":"project_id","value":"13"},{"key":"scope","value":""},{"key":"search_code","value":"true"},{"key":"snippets","value":"false"},{"key":"repository_ref","value":"master"},{"key":"nav_source","value":"navbar"}],"remote_ip":"127.0.0.1","user_id":null,"username":null,"ua":"curl/7.64.1","correlation_id":"pGc23kDjaia","meta.project":"root/wget2","meta.root_namespace":"root","meta.caller_id":"SearchController#show","meta.search.group_id":"456","meta.search.project_id":"13","meta.search.search":"hello world","gitaly_calls":5,"gitaly_duration_s":0.204777,"redis_calls":18,"redis_duration_s":0.034875,"redis_read_bytes":2041,"redis_write_bytes":90337,"redis_cache_calls":16,"redis_cache_duration_s":0.028589,"redis_cache_read_bytes":2039,"redis_cache_write_bytes":90290,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.006286,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":47,"queue_duration_s":0.288697,"cpu_s":3.05,"db_duration_s":0.10424,"view_duration_s":1.87052,"duration_s":3.29513,"db_count":12,"db_write_count":0,"db_cached_count":3}

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Closes #227884 (closed)

Edited by Dylan Griffith

Merge request reports