Add excluded namespaces for Orbit auto-indexing

What does this MR do and why?

Adds an exclusion registry for Orbit automatic indexing on GitLab Self-Managed. The worker skips excluded top-level groups, removes existing enrollments within five minutes, and blocks other enrollment paths from re-enabling them. The Orbit info task reports the exclusion count.

UI will be added in !254180 (closed)

References

Related to #606375 (closed)

Screenshots or screen recordings

Before	After

Database

Creates knowledge_graph_excluded_namespaces in gitlab_main_org with root_namespace_id as its primary key, timestamps, and a foreign key to namespaces with ON DELETE CASCADE. The sharding key is root_namespace_id.

The table is operator-managed and has at most one row per top-level group. Reads happen every five minutes and on demand. Writes happen only when an administrator changes the exclusion list or a parent namespace is deleted. This poses no availability risk at the expected scale.

Migration output

main: == 20260908093000 CreateKnowledgeGraphExcludedNamespaces: reverting ===========
main: -- drop_table(:knowledge_graph_excluded_namespaces, {:id=>false})
main:    -> 0.0063s
main: == 20260908093000 CreateKnowledgeGraphExcludedNamespaces: reverted (0.0156s) ==

main: == 20260908093000 CreateKnowledgeGraphExcludedNamespaces: migrating ===========
main: -- create_table(:knowledge_graph_excluded_namespaces, {:id=>false})
main:    -> 0.0076s
main: == 20260908093000 CreateKnowledgeGraphExcludedNamespaces: migrated (0.0147s) ==

Enrollment query

https://postgres.ai/console/gitlab/gitlab-production-main/sessions/57475/commands/161997

SELECT "namespaces"."id"
FROM "namespaces"
LEFT OUTER JOIN "knowledge_graph_enabled_namespaces" "knowledge_graph_enabled_namespace"
  ON "knowledge_graph_enabled_namespace"."root_namespace_id" = "namespaces"."id"
LEFT OUTER JOIN "knowledge_graph_excluded_namespaces" "knowledge_graph_excluded_namespace"
  ON "knowledge_graph_excluded_namespace"."root_namespace_id" = "namespaces"."id"
WHERE "namespaces"."parent_id" IS NULL
  AND "namespaces"."type" = 'Group'
  AND "knowledge_graph_enabled_namespace"."root_namespace_id" IS NULL
  AND "knowledge_graph_excluded_namespace"."root_namespace_id" IS NULL
  AND "namespaces"."id" >= 1
ORDER BY "namespaces"."id" ASC
LIMIT 1000;

Limit (actual time=0.026..0.035 rows=4 loops=1)
  -> Nested Loop Anti Join (actual time=0.026..0.034 rows=4 loops=1)
     -> Nested Loop Anti Join (actual time=0.020..0.028 rows=6 loops=1)
        -> Index Scan using index_namespaces_on_type_and_id (actual time=0.009..0.016 rows=7 loops=1)
        -> Index Only Scan using index_knowledge_graph_enabled_namespaces_on_root_namespace_id
     -> Index Only Scan using knowledge_graph_excluded_namespaces_pkey
Planning Time: 0.224 ms
Execution Time: 0.054 ms

Local GDK plan because the new table is not available in Database Lab before the MR pipeline.

Un-enrollment query

DELETE FROM "knowledge_graph_enabled_namespaces"
WHERE "knowledge_graph_enabled_namespaces"."root_namespace_id" IN (
  SELECT "knowledge_graph_excluded_namespaces"."root_namespace_id"
  FROM "knowledge_graph_excluded_namespaces"
  WHERE "knowledge_graph_excluded_namespaces"."root_namespace_id" >= 1
    AND "knowledge_graph_excluded_namespaces"."root_namespace_id" < 1001
);

Delete on knowledge_graph_enabled_namespaces (actual time=0.014..0.014 rows=0 loops=1)
  -> Nested Loop (actual time=0.009..0.010 rows=1 loops=1)
     -> Bitmap Heap Scan on knowledge_graph_excluded_namespaces (actual time=0.007..0.008 rows=1 loops=1)
        Recheck Cond: ((root_namespace_id >= 1) AND (root_namespace_id < 1001))
        -> Bitmap Index Scan on knowledge_graph_excluded_namespaces_pkey (actual time=0.004..0.004 rows=1 loops=1)
     -> Index Scan using index_knowledge_graph_enabled_namespaces_on_root_namespace_id (actual time=0.001..0.001 rows=1 loops=1)
Planning Time: 0.106 ms
Execution Time: 0.048 ms

The DELETE ran in a rolled-back transaction. The worker batches over exclusion primary keys and runs one delete for each exclusion batch. This is a local GDK plan because the new table is not available in Database Lab before the MR pipeline.

How to set up and validate locally

1. Create an exclusion with Analytics::KnowledgeGraph::ExcludedNamespace.create!(namespace: Group.top_level.first).
2. Run Analytics::KnowledgeGraph::AutoIndexWorker.new.perform.
3. Verify that the excluded group has no knowledge_graph_enabled_namespace.
4. Run bundle exec rake gitlab:orbit:info and check the ExcludedNamespace count line.

Agent context - implementation and verification

• A real foreign key is used because both tables belong to gitlab_main_org. A loose foreign key is not needed.
• Exclusions apply even when automatic enrollment is disabled, so the worker can finish teardown after an administrator turns the setting off.
• Direct insert_all calls remain possible at the database layer for worker race tolerance. Application enrollment paths use model validation.
• The five changed spec files passed with 713 examples and 0 failures. The supporting group and foreign key specs also passed in the combined run.
• RuboCop passed on all 13 changed Ruby files. Markdownlint, Vale, docs link checks, database dictionary generation, migration checksum checks, and pre-commit checks passed.
• bin/rails db:migrate and the namespaced db:migrate:redo tasks completed locally. The migration also records the ON DELETE CASCADE behavior in a model spec.
• bundle exec rake gitlab:orbit:info printed ExcludedNamespace count: 0.
• Database Lab plans were not generated because the table does not exist in the production clone before this MR runs its migration. Local EXPLAIN (ANALYZE, BUFFERS) plans are included above.
• Direct upload to the GitLab project returned HTTP 400, so the screenshots use absolute raw links from dgruzd/droid-workspace for later re-upload.
• MR 2 admin UI, GraphQL operations, REST API, rake add/remove tasks, and bulk import are not included.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

• The migration is reversible and includes a root_namespace_id primary key, sharding key, and cascading foreign key.
• Enrollment and un-enrollment are batched, idempotent, and covered by specs.
• Model validation prevents group-level re-enrollment of excluded roots.
• Product documentation and Orbit diagnostics are updated.
• Run db:gitlabcom-database-testing in the MR pipeline.