ActiveContext Code: set root_namespace_id through preprocessor

Pam Artiagarequested to merge
606107-code-collection-resolve-namespace-preprocessor into master

What does this MR do and why?

Problem and Solution overview

For SaaS (GitLab.com), AIGW requests require the x-gitlab-root-namespace-id header for billing and compliance tracking. However, the semantic search indexing process does not set this header when invoking embeddings requests. This is because indexing is an asynchronous and batched process without a user and is namespace-agnostic.

In order to resolve this, we need to get the root_namespace_id from the projects being indexed, then set that root_namespace_id in the header. The implementation involves several steps:

  • Add a preprocessor to set the root_namespace_id of each reference based on the project_id (Code collection routing) - This MR
  • Update the embeddings preprocessor to group embeddings generation by root_namespace_id - !247480 (closed)
  • Update the Gitlab::Llm::Embeddings::* classes to accept an optional root_namespace_id, to be used for the AIGW request header - This MR - !246851 (merged)

For details, see proposal.

Updates in this MR

  • ActiveContext::Concerns::Preprocessor
    • add a should_run field to the preprocessors list
    • the should_run for each preprocessor returns true by default
  • ActiveContext::Reference: define a project_id and root_namespace_id
  • Ai::ActiveContext::References::Code:
    • set the project_id from routing on init
    • add the new pre-processor to resolve the root namespace
  • New pre-processor (CodeRootNamespaceResolver)
    • determine references' root_namespace_id based on the the project_id
    • only runs for SaaS instances
    • this pre-processor has additional info logging so we can monitor it on production as we roll it out

In the step to update embeddings preprocessor, we would still allow references with root_namespace_id=nil.

Consideration: setting root_namespace_id vs root_namespace

An alternative solution is to set a root_namespace record instead of a root_namespace_id, which will then be passed on to the Gitlab::Llm::Embeddings::Client, allowing it to set the organization_id header. The solution implementation only sets the root_namespace_id because:

  • organization_id is not required information for billing or compliance
  • passing around a root_namespace ActiveRecord object requires more memory; while GitLab may be able to handle this without issues, I would prefer we choose the solution without additional memory overhead since the additional information is not required

References

Issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/606107+

Screenshots or screen recordings

N/A

How to set up and validate locally

  1. Simulate SaaS by setting env variable: GITLAB_SIMULATE_SAAS=1

  2. Enable the Feature Flag on the rails console: Feature.enable(:semantic_search_indexing_set_root_namespace_id)

  3. Add Code collection documents to the processing queue. You may do this by indexing a new project (see instructions) or simply queue an already indexed document to the bulk processing queue:

    # Make sure you are using the correct routing ID, which should be the `project_id`
::Ai::ActiveContext::Collections::Code.track_refs!(
  routing: "1000000",
  hashes: [
    "e4c13de79c8c911cab6913ea9b3d16c16ea69ed2feb9290291a71d53b15b637a",
    "82e2ef14b0203d80ae2e234ac142683a1f1394328b9672e1e5fc0b9b510e1e36"
  ]
)

  4. Run the bulk processor, and verify through the logs that the code_root_namespace_resolver did run:

    # on the rails console
::Ai::ActiveContext::BulkProcessWorker.new.perform("Ai::ActiveContext::Queues::Code", 0)

    Check the logs:

    # on the GitLab directory
tail -f log/active_context.log

{"severity":"INFO","time":"2026-07-29T04:05:35.607Z","class_name":"ActiveContext::BulkProcessQueue","queue":"Ai::ActiveContext::Queues::Code","message":"bulk_indexing_start","meta.indexing.redis_set":"ai_activecontext_queues:{code}:0:zset","meta.indexing.refs_count":2,"meta.indexing.first_score":1.0,"meta.indexing.last_score":2.0}
{"severity":"INFO","time":"2026-07-29T04:05:35.620Z","correlation_id":"49149a470c8d614c84ef653e6b4b1362","message":"ActiveContext client search completed","collection":"Ai::ActiveContext::Collections::Code","duration_s":0.008,"result_count":2}

# log entries for `code_root_namespace_resolver`
{"severity":"INFO","time":"2026-07-29T04:05:35.620Z","correlation_id":"49149a470c8d614c84ef653e6b4b1362","message":"Resolving root_namespace_id for references","class_name":"Ai::ActiveContext::References::Code","preprocessor":"code_root_namespace_resolver","refs_count":2}
{"severity":"INFO","time":"2026-07-29T04:05:35.691Z","correlation_id":"49149a470c8d614c84ef653e6b4b1362","message":"Resolved root_namespace_id for references","class_name":"Ai::ActiveContext::References::Code","preprocessor":"code_root_namespace_resolver","refs_count":2,"refs_with_root_namespaces_count":2,"unique_root_namespace_ids":[1000000]}

# logs indicating successful end-to-end processing
{"severity":"INFO","time":"2026-07-29T04:05:35.703Z","correlation_id":"49149a470c8d614c84ef653e6b4b1362","message":"generate embeddings","model":"gitlab_managed__text_embedding_005_vertex","status":"start","class_name":"ActiveContext::EmbeddingModel"}
{"severity":"INFO","time":"2026-07-29T04:05:37.670Z","correlation_id":"49149a470c8d614c84ef653e6b4b1362","message":"generate embeddings","model":"gitlab_managed__text_embedding_005_vertex","status":"done","class_name":"ActiveContext::EmbeddingModel"}
{"severity":"INFO","time":"2026-07-29T04:05:37.772Z","correlation_id":"49149a470c8d614c84ef653e6b4b1362","message":"bulk_submitted","meta.indexing.bulk_count":2,"meta.indexing.errors_count":0}
{"severity":"INFO","time":"2026-07-29T04:05:37.775Z","correlation_id":"49149a470c8d614c84ef653e6b4b1362","class_name":"ActiveContext::BulkProcessQueue","message":"bulk_indexer_flushed","meta.indexing.flushing_duration_s":0.10210999997798353}
{"severity":"INFO","time":"2026-07-29T04:05:37.776Z","correlation_id":"49149a470c8d614c84ef653e6b4b1362","class_name":"ActiveContext::BulkProcessQueue","message":"bulk_indexing_end","meta.indexing.redis_set":"ai_activecontext_queues:{code}:0:zset","meta.indexing.refs_count":2,"meta.indexing.first_score":1.0,"meta.indexing.last_score":2.0,"meta.indexing.failures_count":0,"meta.indexing.retryable_count":0,"meta.indexing.bulk_execution_duration_s":2.168960999988485}

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #606107

Edited by Pam Artiaga

Merge request reports