<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Collaborate/take over this issue](https://contributors.gitlab.com/manage-issue?action=work&projectId=278964&issueIid=629757) </details> <!--IssueSummary end--> ## Summary We have duplicate `is_default = true` rows per `project_id` in `security_project_tracked_contexts`. Multiple projects show `master`/`main` pairs (and other branch name pairs) created within seconds of each other — consistent with a race condition between the project-create event handler creating the default tracked context and an early default-branch change. ## Current state The existing unique index only covers `(project_id, context_name, context_type)`: ``` "index_security_project_tracked_contexts_on_project_context" UNIQUE, btree (project_id, context_name, context_type) ``` There is **no constraint** preventing multiple rows with `is_default = true` for the same `project_id`. This means the race condition silently produces duplicate default contexts, which causes incorrect vulnerability counts and risk scores (e.g. `securityTrackedRefs.vulnerabilitiesCount` mismatches, findings not associated with the expected tracked context). ## Root cause `CreateOrUpdateDefaultTrackedContextWorker` subscribes to three events: - `Projects::ProjectCreatedEvent` - `Repositories::DefaultBranchChangedEvent` - `Repositories::RepositoryCreatedEvent` Each event triggers `FindOrCreateService.project_default_branch(project).execute`. The service's `update_default_branch_context?` guard calls `existing_default_branch_context` (which uses `.last` — non-deterministic under concurrency) and only renames the existing default context if one is found. If two events are processed concurrently before either has committed, both can pass the "no existing default context" check and each creates a new row with `is_default = true`, producing duplicates. The deduplication strategy (`deduplicate :until_executed, if_deduplicated: :reschedule_once`) reduces but does not eliminate the window, because the two events may arrive close enough together that the first job hasn't committed before the second job reads. ## Proposed fix The fix has three parts, each in its own MR: --- ### Part 1 — Database constraint (post-migration) Add a partial unique index on `(project_id) WHERE is_default = true` using `NOT VALID` so that: - New duplicate rows are rejected at the DB level immediately after the migration runs. - The migration does **not** validate existing rows (which already contain duplicates), so it is safe to deploy before the cleanup BGM finishes. ```ruby # db/post_migrate/YYYYMMDDHHMMSS_add_unique_index_sptc_on_project_id_where_is_default.rb class AddUniqueIndexSptcOnProjectIdWhereIsDefault < Gitlab::Database::Migration[2.3] milestone '19.X' disable_ddl_transaction! INDEX_NAME = 'index_sptc_on_project_id_where_is_default' def up add_concurrent_index( :security_project_tracked_contexts, :project_id, unique: true, where: 'is_default = true', name: INDEX_NAME ) end def down remove_concurrent_index_by_name :security_project_tracked_contexts, INDEX_NAME end end ``` > **Note:** `add_concurrent_index` in GitLab's migration framework uses `CREATE INDEX CONCURRENTLY` under the hood, which in PostgreSQL automatically skips validation of pre-existing rows that violate the constraint — equivalent to `NOT VALID` for check constraints. The index will only enforce uniqueness for rows inserted or updated **after** the index is created. **Files to change:** - `db/post_migrate/YYYYMMDDHHMMSS_add_unique_index_sptc_on_project_id_where_is_default.rb` (new) - `db/schema_migrations/YYYYMMDDHHMMSS` (auto-generated) --- ### Part 2 — Fix the race condition in `FindOrCreateService` Wrap the find-or-create logic in an advisory lock keyed on `project_id` so that concurrent calls for the same project are serialised. This is the correct fix for the root cause. ```ruby # ee/app/services/security/project_tracked_contexts/find_or_create_service.rb def execute return permission_error unless can_manage_contexts? return validation_error unless valid_params? return ref_not_found_error if current_user && !ref_exists_in_repository? # Serialise concurrent calls for the same project to prevent duplicate # default contexts from being created under a race condition. with_project_lock { execute_within_lock } end private def with_project_lock(&block) project.with_lock { block.call } end def execute_within_lock if existing_context.present? return success(existing_context) if existing_context.tracked? || allow_untracked return ServiceResponse.error(message: 'Context is not tracked') end return success(updated_default_branch_context) if update_default_branch_context? return cant_create_non_default_error unless is_default || allow_untracked create_context end ``` > **Alternative:** Use `Gitlab::ExclusiveLease` keyed on `"tracked_context:#{project.id}"` if a row-level lock on `projects` is too broad. Either approach eliminates the race window. **Files to change:** - `ee/app/services/security/project_tracked_contexts/find_or_create_service.rb` - `ee/spec/services/security/project_tracked_contexts/find_or_create_service_spec.rb` (add concurrent-execution spec) --- ### Part 3 — Model-level uniqueness validation Add a Rails uniqueness validation scoped to `is_default = true` so that the application layer also rejects duplicates (defence-in-depth, and gives a clear error message): ```ruby # ee/app/models/security/project_tracked_context.rb validates :is_default, uniqueness: { scope: :project_id }, if: :is_default? ``` **Files to change:** - `ee/app/models/security/project_tracked_context.rb` - `ee/spec/models/security/project_tracked_context_spec.rb` --- ## Sequencing | Step | What | When | |------|------|------| | 1 | Deploy Part 2 (advisory lock) + Part 3 (model validation) | First — stops new duplicates from being created | | 2 | Deploy Part 1 (DB index, `NOT VALID`) | After BGM `RemoveDuplicateDefaultTrackedContexts` has cleared existing duplicates, or immediately if we accept that the index only guards new rows | | 3 | Verify BGM has completed and no duplicates remain | Before removing `NOT VALID` / adding `VALIDATE` | > The existing `RemoveDuplicateDefaultTrackedContexts` BGM (rescheduled in `20260708120854`) already handles cleanup of pre-existing duplicates. Part 1 adds the DB-level guard so that once cleanup is done, the invariant is enforced permanently. ## References - Observed in production DB: 42 rows across 21 projects with `is_default = true` duplicates - `RemoveDuplicateDefaultTrackedContexts` BGM: `lib/gitlab/background_migration/remove_duplicate_default_tracked_contexts.rb` - Reschedule migration: `db/post_migrate/20260708120854_reschedule_remove_duplicate_default_tracked_contexts.rb` - Worker: `ee/app/workers/security/project_tracked_contexts/create_or_update_default_tracked_context_worker.rb` - Service: `ee/app/services/security/project_tracked_contexts/find_or_create_service.rb` - Model: `ee/app/models/security/project_tracked_context.rb`