Skip to content

Improve Advanced Search counts queries

Dmitry Gruzd requested to merge 299099-update-es-counts-query into master

What does this MR do?

This MR changes ES queries that we use for counts (e.g. /search/count?group_id=&project_id=&repository_ref=&scope=blobs&search=update) to make them more performant.

  • Move the Clause from MUST to FILTER this will save the "score" calculation.
  • Remove Boosting for simple_query_string fields. This will also prevent calculation scores.
  • Set the Size 0 , This will prevent fetching the documents.

#299099 (closed)

Query benchmark with hyperfine

This is a simple benchmark comparing old and new issues query

❯ hyperfine './old.sh' './new.sh'
Benchmark #1: ./old.sh
  Time (mean ± σ):      17.0 ms ±   1.8 ms    [User: 5.1 ms, System: 2.4 ms]
  Range (min … max):    12.9 ms …  24.1 ms    122 runs

Benchmark #2: ./new.sh
  Time (mean ± σ):      10.0 ms ±   1.0 ms    [User: 5.1 ms, System: 2.3 ms]
  Range (min … max):     6.8 ms …  13.7 ms    251 runs

Summary
  './new.sh' ran
    1.70 ± 0.24 times faster than './old.sh'
scripts
cat old.sh
#!/bin/sh

curl -XPOST -s -H "Content-Type: application/json" http://localhost:9200/gitlab-development-issues/_search --data-binary "@old.json" > /dev/null
cat old.json
{"query":{"bool":{"must":[{"simple_query_string":{"_name":"issue:match:search_terms","fields":["iid^3","title^2","description"],"query":"update","lenient":true,"default_operator":"and"}}],"filter":[{"term":{"type":{"_name":"doc:is_a:issue","value":"issue"}}},{"bool":{"should":[{"bool":{"filter":[{"term":{"visibility_level":{"_name":"issue:authorized:project:any","value":0}}},{"terms":{"_name":"issue:authorized:project:issues:enabled_or_private","issues_access_level":[20,10]}}]}},{"bool":{"_name":"issue:authorized:project:visibility:10:issues:access_level","filter":[{"term":{"visibility_level":{"_name":"issue:authorized:project:visibility:10","value":10}}},{"terms":{"_name":"issue:authorized:project:visibility:10:issues:access_level:enabled_or_private","issues_access_level":[20,10]}}]}},{"bool":{"_name":"issue:authorized:project:visibility:20:issues:access_level","filter":[{"term":{"visibility_level":{"_name":"issue:authorized:project:visibility:20","value":20}}},{"terms":{"_name":"issue:authorized:project:visibility:20:issues:access_level:enabled_or_private","issues_access_level":[20,10]}}]}}]}}]}},"highlight":{"fields":{"iid":{},"title":{},"description":{}},"number_of_fragments":0,"pre_tags":["gitlabelasticsearch→"],"post_tags":["←gitlabelasticsearch"]}}
cat new.sh
#!/bin/sh

curl -XPOST -s -H "Content-Type: application/json" http://localhost:9200/gitlab-development-issues/_search --data-binary "@new.json" > /dev/null
cat new.json
{"query":{"bool":{"filter":[{"term":{"type":{"_name":"doc:is_a:issue","value":"issue"}}},{"simple_query_string":{"_name":"issue:match:search_terms","fields":["iid","title","description"],"query":"update","lenient":true,"default_operator":"and"}},{"bool":{"should":[{"bool":{"filter":[{"term":{"visibility_level":{"_name":"issue:authorized:project:any","value":0}}},{"terms":{"_name":"issue:authorized:project:issues:enabled_or_private","issues_access_level":[20,10]}}]}},{"bool":{"_name":"issue:authorized:project:visibility:10:issues:access_level","filter":[{"term":{"visibility_level":{"_name":"issue:authorized:project:visibility:10","value":10}}},{"terms":{"_name":"issue:authorized:project:visibility:10:issues:access_level:enabled_or_private","issues_access_level":[20,10]}}]}},{"bool":{"_name":"issue:authorized:project:visibility:20:issues:access_level","filter":[{"term":{"visibility_level":{"_name":"issue:authorized:project:visibility:20","value":20}}},{"terms":{"_name":"issue:authorized:project:visibility:20:issues:access_level:enabled_or_private","issues_access_level":[20,10]}}]}}]}}]}},"size":0}

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Related to #299099 (closed)

Edited by Dmitry Gruzd

Merge request reports