Skip to content

Add attributes related filters support

What does this MR do?

Adds security attribute filtering capabilities to the namespace security projects resolver, to allow users to filter projects in the security inventory by their assigned security attributes using IS_ONE_OF and IS_NOT_ONE_OF operators.

Filter Logic:

  • IS_ONE_OF filters: Within a single filter, attributes use OR logic. Multiple filters are combined with AND logic (project must match at least one attribute from each filter group).
  • IS_NOT_ONE_OF filters: Projects must not have any of the specified attributes.

Performance Considerations:

  • Uses PG EXISTS and NOT EXISTS subqueries to avoid cartesian product issues that would occur with multiple JOINs (up to 20 filter groups × 20 values each).
  • Leverages existing index on project_id and security_attribute_id for efficient scans.
  • Query plans show execution with nested loop joins and no sequential scans

Query Plans

Filter by IS_ONE_OF and IS_NOT_ONE_OF

Raw SQL
SELECT "security_inventory_filters".* 
FROM "security_inventory_filters" 
WHERE (security_inventory_filters.traversal_ids >= '{115756369}' 
  AND '{115756370}' > security_inventory_filters.traversal_ids) 
  AND "security_inventory_filters"."archived" = FALSE 
  AND (EXISTS ( 
    SELECT 1 
    FROM project_to_security_attributes 
    WHERE project_id = security_inventory_filters.project_id 
      AND security_attribute_id = ANY(ARRAY[1,2,3,4,5,6,7,8,9]::bigint[]) 
  )) 
  AND (NOT EXISTS ( 
    SELECT 1 
    FROM project_to_security_attributes 
    WHERE project_id = security_inventory_filters.project_id 
      AND security_attribute_id = ANY(ARRAY[16,65]::bigint[]) 
  )) 
ORDER BY "security_inventory_filters"."traversal_ids" ASC, 
  "security_inventory_filters"."project_id" ASC, 
  "security_inventory_filters"."id" DESC 
LIMIT 21
Query plan

Note: replica has limited project_to_security_attributes records currently, so it might be harder to predict exact performance.

See details here

Limit  (cost=9.96..10.01 rows=2 width=138) (actual time=2.963..2.965 rows=1 loops=1)
   Buffers: shared hit=19 read=9
   I/O Timings: read=2.836 write=0.000
   ->  Incremental Sort  (cost=9.96..10.01 rows=2 width=138) (actual time=2.962..2.963 rows=1 loops=1)
         Sort Key: security_inventory_filters.traversal_ids, security_inventory_filters.project_id, security_inventory_filters.id DESC
         Buffers: shared hit=19 read=9
         I/O Timings: read=2.836 write=0.000
         ->  Nested Loop Anti Join  (cost=0.84..9.95 rows=1 width=138) (actual time=2.930..2.931 rows=1 loops=1)
               Buffers: shared hit=8 read=9
               I/O Timings: read=2.836 write=0.000
               ->  Nested Loop Semi Join  (cost=0.71..6.78 rows=1 width=138) (actual time=2.904..2.918 rows=2 loops=1)
                     Buffers: shared hit=3 read=9
                     I/O Timings: read=2.836 write=0.000
                     ->  Index Scan using index_security_inventory_filters_on_traversal_ids on public.security_inventory_filters  (cost=0.55..3.57 rows=1 width=138) (actual time=2.822..2.833 rows=2 loops=1)
                           Index Cond: ((security_inventory_filters.traversal_ids >= '{115756369}'::bigint[]) AND (security_inventory_filters.traversal_ids < '{115756370}'::bigint[]))
                           Filter: ((NOT security_inventory_filters.archived) AND (security_inventory_filters.sast = 1))
                           Rows Removed by Filter: 1
                           Buffers: shared read=7
                           I/O Timings: read=2.796 write=0.000
                     ->  Index Only Scan using index_project_security_attributes_project_id_unique on public.project_to_security_attributes  (cost=0.16..1.68 rows=1 width=8) (actual time=0.040..0.040 rows=1 loops=2)
                           Index Cond: (project_to_security_attributes.project_id = security_inventory_filters.project_id)
                           Heap Fetches: 2
                           Filter: (project_to_security_attributes.security_attribute_id = ANY ('{1,2,3,4,5,6,7,8,9}'::bigint[]))
                           Rows Removed by Filter: 0
                           Buffers: shared hit=3 read=2
                           I/O Timings: read=0.040 write=0.000
               ->  Index Only Scan using index_project_security_attributes_project_id_unique on public.project_to_security_attributes project_to_security_attributes_1  (cost=0.14..1.66 rows=1 width=8) (actual time=0.005..0.005 rows=0 loops=2)
                     Index Cond: (project_to_security_attributes_1.project_id = security_inventory_filters.project_id)
                     Heap Fetches: 10
                     Filter: (project_to_security_attributes_1.security_attribute_id = ANY ('{16,65}'::bigint[]))
                     Rows Removed by Filter: 4
                     Buffers: shared hit=5
                     I/O Timings: read=0.000 write=0.000
Settings: effective_cache_size = '338688MB', jit = 'off', seq_page_cost = '4', work_mem = '100MB', random_page_cost = '1.5'

How to validate locally

  1. Create security categories and attributes under a root namespace.
  2. Assign different security attributes to different projects using the from the security inventory screen.
  3. Use the GraphQL query below to test filtering:
   query testSecurityAttributeFilters {
     namespaceSecurityProjects(
       namespaceId: "gid://gitlab/Group/<NAMESPACE_ID>"
       attributeFilters: [
         {
           operator: IS_ONE_OF
           attributes: [
             "gid://gitlab/Security::Attribute/<ID>"
             "gid://gitlab/Security::Attribute/<ID>"
           ]
         }
         {
           operator: IS_NOT_ONE_OF
           attributes: [
             "gid://gitlab/Security::Attribute/<ID>"
           ]
         }
       ]
     ) {
       edges {
         node {
           id
           name
           securityAttributes {
             nodes {
               id
               name
               securityCategory {
                 name
               }
             }
           }
         }
       }
     }
   }

Related to Add attributes related filters support (#578046) • Gal Katz, Nicolae Rotaru • 18.6

Edited by Nicolae Rotaru

Merge request reports

Loading