Use traversal_id auth optimization in search project confidential filters
What does this MR do and why?
This MR moves the advanced search filters for confidentiality (for project level confidentiality checks only) to using a combination of project_id and traversal_ids.
This is being done to reduce query complexity for the Elasticsearch query. Today the filter uses a potentially very large (10k+ in some cases) list of project_ids.
This query approach is already used in the authorization checks.
Ai summary
This code change introduces a new feature flag to improve how GitLab's search system handles confidential content permissions. The main change replaces the old method of checking user access to confidential items with a new approach that uses "traversal IDs" - a more efficient way to determine what groups and projects a user can access.
The new system creates more sophisticated search filters that can better handle complex permission scenarios, especially when users have access through group memberships rather than direct project access. Instead of just checking individual project permissions, it now considers the hierarchical structure of groups and subgroups.
The old logic is preserved as a "legacy" fallback method that gets used when the feature flag is disabled, ensuring the system can safely roll back if needed. The change also includes extensive test files that verify the new search filters work correctly for different user scenarios - anonymous users, users with access, users without access, and admin users.
This is essentially a behind-the-scenes performance and accuracy improvement for GitLab's search functionality, making it better at determining what confidential content each user should be able to see in their search results.
References
Screenshots or screen recordings
Global search
before ES query
{
"query": {
"bool": {
"must": [
{
"simple_query_string": {
"_name": "note:match:search_terms",
"fields": [
"note"
],
"query": "*",
"lenient": true,
"default_operator": "and"
}
}
],
"should": [],
"filter": [
{
"bool": {
"should": [
{
"bool": {
"should": [
{
"prefix": {
"traversal_ids": {
"_name": "filters:permissions:global:private_access:ancestry_filter:descendants",
"value": "24-"
}
}
},
{
"prefix": {
"traversal_ids": {
"_name": "filters:permissions:global:private_access:ancestry_filter:descendants",
"value": "105-"
}
}
}
],
"filter": [
{
"terms": {
"_name": "filters:permissions:global:private_access:issues_access_level:enabled_or_private",
"issues_access_level": [
20,
10
]
}
}
],
"minimum_should_match": 1
}
},
{
"bool": {
"should": [
{
"prefix": {
"traversal_ids": {
"_name": "filters:permissions:global:private_access:ancestry_filter:descendants",
"value": "24-"
}
}
},
{
"prefix": {
"traversal_ids": {
"_name": "filters:permissions:global:private_access:ancestry_filter:descendants",
"value": "105-"
}
}
}
],
"filter": [
{
"terms": {
"_name": "filters:permissions:global:private_access:merge_requests_access_level:enabled_or_private",
"merge_requests_access_level": [
20,
10
]
}
}
],
"minimum_should_match": 1
}
},
{
"bool": {
"should": [
{
"prefix": {
"traversal_ids": {
"_name": "filters:permissions:global:private_access:ancestry_filter:descendants",
"value": "24-"
}
}
},
{
"prefix": {
"traversal_ids": {
"_name": "filters:permissions:global:private_access:ancestry_filter:descendants",
"value": "105-"
}
}
}
],
"filter": [
{
"terms": {
"_name": "filters:permissions:global:private_access:snippets_access_level:enabled_or_private",
"snippets_access_level": [
20,
10
]
}
}
],
"minimum_should_match": 1
}
},
{
"bool": {
"should": [
{
"prefix": {
"traversal_ids": {
"_name": "filters:permissions:global:private_access:ancestry_filter:descendants",
"value": "24-"
}
}
},
{
"prefix": {
"traversal_ids": {
"_name": "filters:permissions:global:private_access:ancestry_filter:descendants",
"value": "105-"
}
}
}
],
"filter": [
{
"terms": {
"_name": "filters:permissions:global:private_access:repository_access_level:enabled_or_private",
"repository_access_level": [
20,
10
]
}
}
],
"minimum_should_match": 1
}
},
{
"bool": {
"should": [
{
"terms": {
"_name": "filters:permissions:global:issues_access_level:enabled",
"issues_access_level": [
20
]
}
},
{
"terms": {
"_name": "filters:permissions:global:merge_requests_access_level:enabled",
"merge_requests_access_level": [
20
]
}
},
{
"terms": {
"_name": "filters:permissions:global:snippets_access_level:enabled",
"snippets_access_level": [
20
]
}
},
{
"terms": {
"_name": "filters:permissions:global:repository_access_level:enabled",
"repository_access_level": [
20
]
}
}
],
"filter": [
{
"terms": {
"_name": "filters:permissions:global:visibility_level:public_and_internal",
"visibility_level": [
20,
10
]
}
}
],
"minimum_should_match": 1
}
}
],
"minimum_should_match": 1
}
},
{
"bool": {
"should": [
{
"bool": {
"must": [
{
"bool": {
"_name": "note:confidentiality:issue:not_confidential",
"should": [
{
"bool": {
"must_not": [
{
"exists": {
"field": "issue"
}
}
]
}
},
{
"term": {
"issue.confidential": false
}
}
]
}
},
{
"bool": {
"_name": "note:confidentiality:not_confidential",
"should": [
{
"bool": {
"must_not": [
{
"exists": {
"field": "confidential"
}
}
]
}
},
{
"term": {
"confidential": false
}
}
]
}
}
]
}
},
{
"bool": {
"must": [
{
"bool": {
"_name": "note:confidentiality:issue:confidential",
"should": {
"term": {
"issue.confidential": true
}
}
}
},
{
"bool": {
"_name": "note:confidentiality:not_confidential",
"should": [
{
"bool": {
"must_not": [
{
"exists": {
"field": "confidential"
}
}
]
}
},
{
"term": {
"confidential": false
}
}
]
}
},
{
"bool": {
"_name": "note:confidentiality:user:issue_author:issue_assignee:project_membership",
"should": [
{
"term": {
"issue.author_id": 70
}
},
{
"term": {
"issue.assignee_id": 70
}
},
{
"terms": {
"project_id": [
2,
3,
22,
25,
27,
28
]
}
}
]
}
}
]
}
},
{
"bool": {
"must": [
{
"bool": {
"_name": "note:confidentiality:confidential",
"should": {
"term": {
"confidential": true
}
}
}
},
{
"bool": {
"_name": "note:confidentiality:user:project_membership",
"should": {
"terms": {
"_name": "note:confidentiality:project:membership:id",
"project_id": [
2,
3,
22,
25,
27,
28
]
}
}
}
}
]
}
}
]
}
},
{
"bool": {
"_name": "note:archived:non_archived",
"should": [
{
"bool": {
"filter": {
"term": {
"archived": {
"value": false
}
}
}
}
},
{
"bool": {
"must_not": {
"exists": {
"field": "archived"
}
}
}
}
]
}
},
{
"term": {
"noteable_type": {
"_name": "filters:related:issue",
"value": "Issue"
}
}
}
]
}
},
"highlight": {
"fields": {
"note": {}
},
"number_of_fragments": 0,
"pre_tags": [
"gitlabelasticsearch→"
],
"post_tags": [
"←gitlabelasticsearch"
]
},
"_source": [
"noteable_id"
],
"size": 100
}after ES query
{
"query": {
"bool": {
"should": [
{
"simple_query_string": {
"_name": "work_item:match:search_terms",
"fields": [
"iid^50",
"title^2",
"description"
],
"query": "*",
"lenient": true,
"default_operator": "and"
}
},
{
"terms": {
"_name": "work_item:related:ids",
"id": [
57,
58,
59,
60,
61,
62,
63,
64,
38,
39,
40,
41
],
"boost": 0.7
}
}
],
"filter": [
{
"bool": {
"should": [
{
"bool": {
"filter": [
{
"bool": {
"should": [
{
"bool": {
"should": [
{
"prefix": {
"traversal_ids": {
"_name": "filters:permissions:global:private_access:ancestry_filter:descendants",
"value": "24-"
}
}
},
{
"prefix": {
"traversal_ids": {
"_name": "filters:permissions:global:private_access:ancestry_filter:descendants",
"value": "105-"
}
}
}
],
"filter": [
{
"terms": {
"_name": "filters:permissions:global:private_access:issues_access_level:enabled_or_private",
"issues_access_level": [
20,
10
]
}
}
],
"minimum_should_match": 1
}
},
{
"bool": {
"should": [
{
"terms": {
"_name": "filters:permissions:global:issues_access_level:enabled",
"issues_access_level": [
20
]
}
}
],
"filter": [
{
"terms": {
"_name": "filters:permissions:global:project_visibility_level:public_and_internal",
"project_visibility_level": [
20,
10
]
}
}
],
"minimum_should_match": 1
}
}
],
"minimum_should_match": 1
}
}
]
}
}
],
"minimum_should_match": 1
}
},
{
"bool": {
"should": [
{
"bool": {
"filter": [
{
"term": {
"confidential": {
"_name": "filters:confidentiality:projects:user_filter",
"value": true
}
}
},
{
"bool": {
"should": [
{
"term": {
"confidential": {
"_name": "filters:confidentiality:projects:non_confidential",
"value": false
}
}
},
{
"bool": {
"should": [
{
"term": {
"author_id": {
"_name": "filters:confidentiality:projects:confidential:as_author",
"value": 70
}
}
},
{
"term": {
"assignee_id": {
"_name": "filters:confidentiality:projects:confidential:as_assignee",
"value": 70
}
}
},
{
"prefix": {
"traversal_ids": {
"_name": "filters:confidentiality:projects:ancestry_filter:descendants",
"value": "24-"
}
}
},
{
"prefix": {
"traversal_ids": {
"_name": "filters:confidentiality:projects:ancestry_filter:descendants",
"value": "105-"
}
}
}
],
"filter": [
{
"term": {
"confidential": {
"_name": "filters:confidentiality:projects:confidential",
"value": true
}
}
}
],
"minimum_should_match": 1
}
}
],
"minimum_should_match": 1
}
}
]
}
}
],
"minimum_should_match": 1
}
},
{
"term": {
"hidden": {
"_name": "filters:not_hidden",
"value": false
}
}
},
{
"bool": {
"_name": "filters:non_archived",
"should": [
{
"bool": {
"filter": {
"term": {
"archived": {
"value": false
}
}
}
}
},
{
"bool": {
"must_not": {
"exists": {
"field": "archived"
}
}
}
}
]
}
},
{
"bool": {
"must_not": {
"terms": {
"_name": "filters:not_work_item_type_ids",
"work_item_type_id": [
8
]
}
}
}
}
],
"minimum_should_match": 1
}
},
"highlight": {
"fields": {
"iid": {},
"title": {},
"description": {}
},
"number_of_fragments": 0,
"pre_tags": [
"gitlabelasticsearch→"
],
"post_tags": [
"←gitlabelasticsearch"
]
},
"_source": [
"id"
],
"from": 0,
"size": 20,
"sort": {}
}How to set up and validate locally
the new behavior is behind a FF: search_project_confidentiality_use_traversal_ids
Recommend testing with a non-admin user because admins bypass the auth checks for confidential. I also tested with anonymous user - they should never seen confidential issues.
- enable elasticsearch in gdk
- leave the FF turned off for now
- load up a global search for issues
- check performance bar for
esand verify the auth for confidential is there - ensure the user is only seeing confidential issues they should see
- repeat for group and project search for issues
- turn the FF on:
Feature.enable(:search_project_confidentiality_use_traversal_ids) - repeat the tests
- make sure user sees the same number of issues per search level (global, group, project)
- make sure in performance bar query does not contain a huge list of projects for confidential query
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Edited by Terri Chu