<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Label this issue](https://contributors.gitlab.com/manage-issue?action=label&projectId=278964&issueIid=416286) - [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=416286) </details> <!--IssueSummary end--> ## Summary When you use Advanced Search, GitLab paginates results. But due to the mechanics of how result ordering works in Elasticsearch, GitLab will often duplicate some results across pages (but not within a single page). GitLab also will only ever return the number of results equal to the number of unique matching documents in Elasticsearch. This leads to a situation where duplicate results "crowd out" legitimate results, leading to missing data in the search result. For example, say you search for something in GitLab. It gets sent to Elasticsearch, which tells GitLab that there are 20 results. GitLab takes the result count and calculates pagination across those results. Let's say GitLab decides 20 results equals two pages, each with 10 results. But page 1 has search result ``x``, and page 2 also happens to have search result ``x``. Since GitLab only calculated pagination for 20 results, and not 21, this duplicate effectively "crowds out" one legitimate result. These "false negatives" lead the user to believe some of what they searched for doesn't exist in the instance, even though it does. ## Steps to reproduce Although I originally encountered this issue in an on-premise GitLab cluster, I also reproduced it on GitLab SaaS as follows. Set ``GITLAB_TOKEN`` to an access token with the ``read_api`` scope. Install the ``jq`` program. Run the following script: ``` page=1 while [ $page -ne 20 ]; do curl -H "Authorization: Bearer $GITLAB_TOKEN" "https://gitlab.com/api/v4/groups/9970/search?per_page=20&page=${page}&scope=blobs&search=filename%3A.gitlab-ci.yml" -o "page${page}.json" page=$((page+1)) done cat page*.json | jq -S -r '.[] | (.ref + "|" + .path)' | sort > before.txt cat page*.json | jq -S -r '.[] | (.ref + "|" + .path)' | sort |uniq > after.txt wc -l before.txt after.txt ``` When you do this, you get results like the following: ``` 380 before.txt 367 after.txt 747 total ``` This seems to show there are 13 dupes just in the subset of search results we scanned (the first 20 pages). If you dive into a specific dupe, it does look like the exact same data, e.g.: ``` $ cat before.txt | sort | uniq -c|grep -v ' 1 ' 2 0e22730da0423b63956d76cc285541d4560d17c6|qa/qa/ee/fixtures/secure_license_files/.gitlab-ci.yml 2 0e22730da0423b63956d76cc285541d4560d17c6|qa/qa/ee/fixtures/secure_premade_reports/.gitlab-ci.yml 2 109448b4e46a5692bf49913be46cbb2f1685e54d|.gitlab-ci.yml 2 4e542411229dc7041932a5192a2829a68c717ecd|.gitlab-ci.yml 2 54cfc448ad12fa931d02d7b1bbbaaa2b044741ee|.gitlab-ci.yml 2 5536068c8f35af2fecb947fcd2aff6229a48c792|.gitlab-ci.yml 2 65391e7c69ba2350e8335720bc6a3990a3775c5a|.gitlab-ci.yml 2 673a7d0090a2238787adaaa801e6f29ed92e1050|.gitlab-ci.yml 2 9be979af51c25f85731aaae8e79ae4e66d27f1da|template/.gitlab-ci.yml 2 c9a2ed5409efbdbb18cf0b88b1c91cc5790fa4be|.gitlab-ci.yml 4 cdeda7ae724a332e008d17245209d5edd9ba6499|.gitlab-ci.yml $ grep 0e22730da0423b63956d76cc285541d4560d17c6 page*.json | cut -d: -f2- | jq -S -r '.[]|select(.filename=="qa/qa/ee/fixtures/secure_license_files/.gitlab-ci.yml")|select(.ref=="0e22730da0423b63956d76cc285541d4560d17c6")' { "basename": "qa/qa/ee/fixtures/secure_license_files/.gitlab-ci", "data": "include:\n template: License-Scanning.gitlab-ci.yml\n\n", "filename": "qa/qa/ee/fixtures/secure_license_files/.gitlab-ci.yml", "id": null, "path": "qa/qa/ee/fixtures/secure_license_files/.gitlab-ci.yml", "project_id": 42278181, "ref": "0e22730da0423b63956d76cc285541d4560d17c6", "startline": 1 } { "basename": "qa/qa/ee/fixtures/secure_license_files/.gitlab-ci", "data": "include:\n template: License-Scanning.gitlab-ci.yml\n\n", "filename": "qa/qa/ee/fixtures/secure_license_files/.gitlab-ci.yml", "id": null, "path": "qa/qa/ee/fixtures/secure_license_files/.gitlab-ci.yml", "project_id": 42278181, "ref": "0e22730da0423b63956d76cc285541d4560d17c6", "startline": 1 } ``` ## What is the current *bug* behavior? 1. (Less important) The search results contain dupes 2. (More important) The dupes are crowding out legitimate results ## What is the expected *correct* behavior? The ideal correct behavior is to have zero duplicate results and zero missing results. The next best thing would be to accept the dupes, but to paginate correctly so that legitimate results still appear. Ideally, the search results page also warns you that there may be duplicates, and ideally this behavior is documented in the Advanced Search documentation so users are aware of it. **At the very least**, if this issue is not addressed, the Advanced Search documentation should be updated to point out this flaw. ## Relevant logs and/or screenshots More detail in zd#407505 ## Possible fixes More detail in zd#407505. I also believe there's an internal ticket for this problem already. ## Possible workarounds You may have better luck changing the sort option to something more stable than "Most relevant" and then paginating through results using this sort option. Today this won't work for code search, however, because we don't have any other sorting options. For code search you may have more luck just trying to narrow down searches and perhaps you can still exhaustively get data by multiple different narrowed down searches that cover all the data you are looking for.