Enforce the tracked context quota per root namespace

Gregory Havengarequested to merge
ghavenga-20475-root-namespace-quota into master

What does this MR do and why?

This MR enforces the tracked security context quota for each root namespace. It is part of the quota work in &20475.

Today the quota is enforced only for each organization. Most root namespaces on GitLab.com are still in the default organization. So an organization quota cannot tell customers apart, and the GitLab.com default quota of 2 applies to the whole default organization. We decided to enforce the quota for each root namespace until customers are in their own organizations (#627145).

Changes

  1. A new nullable column namespace_settings.security_tracked_context_quota stores the quota of a root namespace. NamespaceSetting validates that the value is an integer of 1 or more.

  2. A new service Security::RootNamespaceTrackedContextQuotaService calculates the quota of a root namespace:

    Item Value
    Limit The quota set on the root namespace. When no quota is set, the GitLab.com default from default_security_tracked_context_quota (2 when the setting is empty). On self-managed, nil (no enforcement).
    Usage All tracked contexts of all projects in the root namespace, including default-branch contexts. Untracked contexts do not count.
    Enforcement Behind the security_tracked_context_quota_enforcement feature flag, with the project as the actor.

  3. A new module Security::TrackedContextQuota combines the checks for a project:

    1. It checks the quota of the root namespace of the project.
    2. It then checks the organization quota, but only when the quota is set explicitly on the organization. The organization default no longer applies.

  4. Security::ProjectTrackedContext uses Security::TrackedContextQuota when a context changes to tracked and the enforcement flag is enabled for the project. The follow-up MR !265369 (#621576) uses the same module when it creates a default context.

Before this MR, the model called Security::TrackedContextQuotaService directly, with inline CodeReuse/ServiceClass disables. The model now calls the Security::TrackedContextQuota module, and the module calls the two quota services. The dependency from the model to the quota services does not change.

The organization quota stays at the organization level. This includes Security::TrackedContextQuotaService, the securityTrackedContextQuota field (!239741 (merged)) and the mutation (!239796 (merged)). It becomes the main level when customers move to their own organizations.

The securityTrackedContextQuota field still reports the organization default as the limit when no organization quota is set. That limit is no longer enforced. The root namespace API follow-up changes this.

Behavior

Root namespace quota Organization quota Result
Set Not set The root namespace quota applies.
Not set, GitLab.com Not set The application default applies to the root namespace.
Not set, self-managed Not set No quota.
Any Set explicitly The root namespace quota applies. Then the organization quota applies.

The feature flag security_tracked_context_quota_enforcement is not enabled for customers. Two changes must land before we enable it:

  • #621576 (!265369) creates the default-branch context as untracked when a root namespace is over quota. Without it, a root namespace at its quota cannot create the default-branch context of a new project.
  • #556012 backfills a quota for existing root namespaces. The issue still describes an organization backfill, and we will update it for root namespaces.

Known limits, unchanged from the organization quota:

  • No lock covers the root namespace. Two requests in the same root namespace can pass the check at the same time and go over the quota by a small amount.
  • A project transfer moves tracked contexts to a new root namespace without a quota check.

Database

The migration adds one nullable integer column without a default to namespace_settings. This is a metadata-only change. namespace_settings is not in LargeTables or OverLimitTables in rubocop/rubocop-migrations.yml.

Usage is counted with the existing for_root_namespaces scope. It uses index_security_project_tracked_contexts_on_traversal_ids_id with a range on traversal_ids for one root namespace. The enforcement check limits the count to the quota (LIMIT <quota>). The scan stops after it finds quota tracked rows, plus the untracked rows that the filter removes on the way. The plans below show the unbounded count, which is the worst case.

Query plan for the usage count of a large root namespace on GitLab.com (gitlab-org, 7,883 tracked contexts), from Joe on gitlab-production-sec:

SELECT COUNT(*) FROM security_project_tracked_contexts WHERE security_project_tracked_contexts.state = 2 AND (security_project_tracked_contexts.traversal_ids >= '{9970}' AND security_project_tracked_contexts.traversal_ids < '{9971}')
Plan, warm cache (Joe command 528516) 
 Aggregate  (cost=3.59..3.60 rows=1 width=8) (actual time=11.187..11.188 rows=1 loops=1)
   Buffers: shared hit=7483
   I/O Timings: read=0.000 write=0.000
   ->  Index Scan using index_security_project_tracked_contexts_on_traversal_ids_id on public.security_project_tracked_contexts  (cost=0.56..3.58 rows=1 width=0) (actual time=0.043..10.793 rows=7883 loops=1)
         Index Cond: ((security_project_tracked_contexts.traversal_ids >= '{9970}'::bigint[]) AND (security_project_tracked_contexts.traversal_ids < '{9971}'::bigint[]))
         Index Searches: 1
         Filter: (security_project_tracked_contexts.state = 2)
         Rows Removed by Filter: 6
         Buffers: shared hit=7483
         I/O Timings: read=0.000 write=0.000
Settings: work_mem = '100MB', random_page_cost = '1.5', seq_page_cost = '4', effective_cache_size = '338688MB', jit = 'off'
Query ID: -8632520142418476311

Time 12.406 ms. Buffers: 7,483 hits (~58.46 MiB), 0 reads.

Plan, cold cache (Joe command 528514) 
 Aggregate  (cost=3.59..3.60 rows=1 width=8) (actual time=3385.056..3385.058 rows=1 loops=1)
   Buffers: shared hit=770 read=6719 dirtied=671
   WAL: records=733 fpi=671 bytes=5390641
   I/O Timings: read=3280.321 write=0.000
   ->  Index Scan using index_security_project_tracked_contexts_on_traversal_ids_id on public.security_project_tracked_contexts  (cost=0.56..3.58 rows=1 width=0) (actual time=4.801..3382.969 rows=7883 loops=1)
         Index Cond: ((security_project_tracked_contexts.traversal_ids >= '{9970}'::bigint[]) AND (security_project_tracked_contexts.traversal_ids < '{9971}'::bigint[]))
         Index Searches: 1
         Filter: (security_project_tracked_contexts.state = 2)
         Rows Removed by Filter: 6
         Buffers: shared hit=770 read=6719 dirtied=671
         WAL: records=733 fpi=671 bytes=5390641
         I/O Timings: read=3280.321 write=0.000
Settings: effective_cache_size = '338688MB', jit = 'off', work_mem = '100MB', random_page_cost = '1.5', seq_page_cost = '4'
Query ID: -8632520142418476311

Time 3.386 s. Buffers: 770 hits (~6.02 MiB), 6,719 reads (~52.49 MiB), 671 dirtied (~5.24 MiB).

The Joe CLI does not give a console.postgres.ai link for these runs. The count runs once each time a context changes to tracked, and only when the enforcement flag is enabled.

How to set up and validate locally

  1. Enable the enforcement flag and set a quota in the Rails console:

    group = Group.find_by_full_path('gitlab-org')
project = group.projects.first
Feature.enable(:vulnerabilities_across_contexts, project)
Feature.enable(:security_tracked_context_quota_enforcement, project)
group.namespace_settings.update!(security_tracked_context_quota: 1)

  2. Make sure that the group has at least 1 tracked context. The default-branch context of any project in the group counts.

  3. Track a branch of the project with the securityRefsTrack GraphQL mutation. The response contains the error "The gitlab-org namespace has reached its quota of 1 tracked security context. Contact your administrator to request a quota increase."

    mutation {
  securityRefsTrack(input: { projectPath: "<project full path>", refs: [{ name: "<branch>", refType: BRANCH }] }) {
    trackedRefs { name }
    errors
  }
}

  4. Run the specs:

    bundle exec rspec ee/spec/services/security/root_namespace_tracked_context_quota_service_spec.rb \
  ee/spec/services/security/tracked_context_quota_spec.rb \
  ee/spec/models/security/project_tracked_context_spec.rb \
  ee/spec/models/namespace_setting_spec.rb \
  ee/spec/services/security/project_tracked_contexts/find_or_create_service_spec.rb

MR acceptance checklist

  • Specs pass
  • Query plan for the usage count added
  • Database review

Related issues

Part of &20475 (Tracked Context Quota Management)

Related to #627145, #621576, #556012 and #634802

Edited by Gregory Havenga

Merge request reports