Skip to content
Snippets Groups Projects

Finalize migration to backfill namespace_id in policy rule schedules

Merged Finalize migration to backfill namespace_id in policy rule schedules
overdue-finalize-background-migration--backfill-security-orchestration-policy-rule-schedules-namespace-id into master
Merged ops.gitlab.net/gitlab-com/database-team/gitlab-com-database-testing:GITLAB_API_TOKEN requested to merge overdue-finalize-background-migration--backfill-security-orchestration-policy-rule-schedules-namespace-id into master

This migration was finished at 2024-08-23 03:43:02 UTC, you can confirm the status using our batched background migration chatops commands. To confirm it is finished you can run:

/chatops run batched_background_migrations status 1000655 --database cell

The last time this background migration was triggered was in db/post_migrate/20240816130544_queue_backfill_security_orchestration_policy_rule_schedules_namespace_id.rb

You can read more about the process for finalizing batched background migrations in https://docs.gitlab.com/ee/development/database/batched_background_migrations.html .

As part of our process we want to ensure all batched background migrations have had at least one required stop to process the migration. Therefore we can finalize any batched background migration that was added before the last required stop.

This change was generated by gitlab-housekeeper using the Keeps::OverdueFinalizeBackgroundMigration keep.

To provide feedback on your experience with gitlab-housekeeper please create an issue with the label GitLab Housekeeper and consider pinging the author of this keep.

Edited by ops.gitlab.net/gitlab-com/database-team/gitlab-com-database-testing:GITLAB_API_TOKEN

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • **** added automation:gitlab-housekeeper-authored groupsecurity policies maintenanceremoval labels

    added automation:gitlab-housekeeper-authored groupsecurity policies maintenanceremoval labels

  • **** requested review from @mcavoj

    requested review from @mcavoj

  • **** added 1 commit

    added 1 commit

    • 33d81ead - Finalize migration BackfillSecurityOrchestrationPolicyRuleSchedulesNamespaceId

    Compare with previous version

  • 🤖 GitLab Bot 🤖 added Category:Security Policy Management devopssecurity risk management sectionsec labels

    added Category:Security Policy Management devopssecurity risk management sectionsec labels

  • 🤖 GitLab Bot 🤖 added typemaintenance label

    added typemaintenance label

  • 🤖 GitLab Bot 🤖 added pipelinetier-1 label

    added pipelinetier-1 label

  • **** added database databasereview pending labels

    added database databasereview pending labels

  • ****
    **** @project_278964_bot_b66b169fda2a3223a645094be35d5515 ·
    Developer
    2 Warnings
    :warning: New migrations added but db/structure.sql wasn't updated

    Usually, when adding new migrations, db/structure.sql should be
    updated too (unless the migration isn't changing the DB schema
    and isn't the most recent one).
    :warning: You've made some app changes, but didn't add any tests.
    That's OK as long as you're refactoring existing code,
    but please consider adding any of the maintenancepipelines, maintenancerefactor, maintenanceworkflow, maintenanceperformance, documentation, QA labels.

    Reviewer roulette

    Category Reviewer Maintainer
    database @dstull profile link current availability (UTC-4) @ahegyi profile link current availability (UTC+1)

    Please refer to documentation page for guidance on how you can benefit from the Reviewer Roulette, or use the GitLab Review Workload Dashboard to find other available reviewers.

    If needed, you can retry the :repeat: danger-review job that generated this comment.

    Generated by :no_entry_sign: Danger

    Edited by ****
  • 🤖 GitLab Bot 🤖 added automation:bot-authored label

    added automation:bot-authored label

  • Martin Čavoj changed title from Finalize migration BackfillSecurityOrchestrationPolicyRuleSchedulesNamespaceId to Finalize migration to backfill namespace_id in policy rule schedules

    changed title from Finalize migration BackfillSecurityOrchestrationPolicyRuleSchedulesNamespaceId to Finalize migration to backfill namespace_id in policy rule schedules

  • Martin Čavoj assigned to @mcavoj

    assigned to @mcavoj

  • Martin Čavoj removed review request for @mcavoj

    removed review request for @mcavoj

  • ****
    **** @project_278964_bot_079be3103ed333565a8fbea3be7252f4 ·
    Developer

    Database migrations (on the main database)

    Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

    Migration Type Total runtime Result DB size change
    20250303231642 - FinalizeHkBackfillSecurityOrchestrationPolicyRuleSchedulesNamespa Post deploy 5.6 s :white_check_mark: +0.00 B
    Runtime Histogram for all migrations
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 0
    0.1 seconds - 1 second 5
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Migration: 20250303231642 - FinalizeHkBackfillSecurityOrchestrationPolicyRuleSchedulesNamespa

     * Type: Post deploy
    * Duration: 5.6 s
    * Database size change: +0.00 B
    Calls Total Time Max Time Mean Time Rows Query
    1 1.0 ms 1.0 ms 1.0 ms 1 
    SELECT "batched_background_migrations".*
      FROM "batched_background_migrations"  WHERE "batched_background_migrations"."job_class_name" = $1 AND "batched_background_migrations"."table_name" = $2 AND "batched_background_migrations"."column_name" = $3 AND (job_arguments = $4) AND (ARRAY[$5] <@ ARRAY[$6, $7, $8, $9, $10, $11, $12]) AND "batched_background_migrations"."gitlab_schema" IN ($13, $14, $15, $16, $17, $18, $19)
      ORDER BY "batched_background_migrations"."id" ASC
      LIMIT $20
    1 0.1 ms 0.1 ms 0.1 ms 1 
    UPDATE "batched_background_migrations"  SET "updated_at" = $1, "status" = $2
      WHERE "batched_background_migrations"."id" = $3
    1 0.0 ms 0.0 ms 0.0 ms 0 
    SELECT $1 AS one
      FROM "batched_background_migrations"  WHERE "batched_background_migrations"."job_arguments" = $2 AND "batched_background_migrations"."id" != $3 AND "batched_background_migrations"."job_class_name" = $4 AND "batched_background_migrations"."table_name" = $5 AND "batched_background_migrations"."column_name" = $6
      LIMIT $7
    2 0.0 ms 0.0 ms 0.0 ms 2 
    SELECT pg_backend_pid()
    Histogram for FinalizeHkBackfillSecurityOrchestrationPolicyRuleSchedulesNamespa
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 0
    0.1 seconds - 1 second 5
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Other information

    Other migrations pending on GitLab.com
    Migration Type Total runtime Result DB size change
    20250304132919 - AddVersionPartsToPackagesTerraformModuleMetadata Regular 8.7 s :warning: +0.00 B
    20250307100237 - AddProjectIdToPackagesDebianProjectComponentFiles Regular 5.2 s :white_check_mark: +0.00 B
    20250211220147 - ScheduleIndexOnMergeRequestDiffsIdAndProjectId Post deploy 6.9 s :white_check_mark: +0.00 B
    20250224063503 - DeleteRemoteDevelopmentNamespaceClusterAgentMappingView Post deploy 5.1 s :white_check_mark: +0.00 B
    20250225043739 - AddEvidencesProjectIdNotNull Post deploy 15.6 s :warning: +0.00 B
    20250225043936 - AddPackagesDebianProjectComponentsProjectIdNotNull Post deploy 6.5 s :white_check_mark: +0.00 B
    20250225044336 - AddPackagesMavenMetadataProjectIdNotNull Post deploy 9.8 s :warning: +0.00 B
    20250227091623 - IndexRedirectRoutesOnNamespaceId Post deploy 8.4 s :white_check_mark: +34.88 MiB
    20250227092023 - AddRedirectRoutesNamespaceIdFk Post deploy 6.3 s :white_check_mark: +0.00 B
    20250227231728 - FinalizeHkBackfillVulnerabilityOccurrenceIdentifiersProjectId Post deploy 5.9 s :white_check_mark: +0.00 B
    20250228142820 - IndexIncidentManagementPendingIssueEscalationsOnNamespaceId Post deploy 14.7 s :white_check_mark: +72.00 KiB
    20250228142821 - AddIncidentManagementPendingIssueEscalationsNamespaceIdFk Post deploy 18.3 s :white_check_mark: +16.00 KiB
    20250228142822 - AddIncidentManagementPendingIssueEscalationsNamespaceIdTrigger Post deploy 5.4 s :white_check_mark: +8.00 KiB [note]
    20250228142823 - QueueBackfillIncidentManagementPendingIssueEscalationsNamespaceId Post deploy 5.6 s :white_check_mark: +0.00 B
    20250228143727 - RequeueSyncUnlinkedSecurityPolicyProjectLinks Post deploy 6.4 s :white_check_mark: +0.00 B
    20250228155146 - QueueBackfillOnboardingStatusSetupForCompany Post deploy 5.8 s :white_check_mark: +0.00 B
    20250303122513 - RemoveBrokenFkOnCiBuildsAndCiPipelines Post deploy 5.8 s :white_check_mark: +0.00 B
    20250303231437 - FinalizeHkBackfillProtectedEnvironmentApprovalRulesProtectedEnvir47878 Post deploy 5.6 s :white_check_mark: +0.00 B
    20250304084537 - AddDoraDailyMetricsProjectIdDateIndex Post deploy 82.4 s :white_check_mark: +580.66 MiB
    20250304204818 - DeleteIndexTodosOnAuthorIdAndCreatedAtIndex Post deploy 7.2 s :white_check_mark: -3.36 GiB
    20250304231245 - IncreaseTokenEncryptedConstraint Post deploy 6.8 s :white_check_mark: +0.00 B
    20250304231710 - FinalizeHkBackfillSecurityOrchestrationPolicyRuleSchedulesProject Post deploy 5.6 s :white_check_mark: +0.00 B
    20250305084646 - IndexScanResultPoliciesOnApprovalPolicyRuleId Post deploy 6.6 s :white_check_mark: +3.80 MiB
    20250305232245 - FinalizeHkBackfillCiBuildTraceMetadataProjectId Post deploy 4.8 s :white_check_mark: +0.00 B
    20250307100238 - IndexPackagesDebianProjectComponentFilesOnProjectId Post deploy 5.9 s :white_check_mark: +8.00 KiB [note]
    20250307100239 - AddPackagesDebianProjectComponentFilesProjectIdFk Post deploy 6.0 s :white_check_mark: +8.00 KiB [note]
    20250307100240 - AddPackagesDebianProjectComponentFilesProjectIdTrigger Post deploy 5.4 s :white_check_mark: +0.00 B
    20250307100241 - QueueBackfillPackagesDebianProjectComponentFilesProjectId Post deploy 5.9 s :white_check_mark: +0.00 B
    Clone details
    Clone ID Clone Created At Clone Data Timestamp Expected Removal Time
    database-testing-4267870-17540303-main 2025-03-08T12:11:23Z 2025-03-08T08:09:58Z 2025-03-09 00:32:49 +0000
    database-testing-4267870-17540303-ci 2025-03-08T12:11:24Z 2025-03-08T08:45:04Z 2025-03-09 00:32:49 +0000

    Job artifacts

    Database migrations (on the ci database)

    Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

    Migration Type Total runtime Result DB size change
    20250303231642 - FinalizeHkBackfillSecurityOrchestrationPolicyRuleSchedulesNamespa Post deploy 6.9 s :white_check_mark: +0.00 B
    Runtime Histogram for all migrations
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 0
    0.1 seconds - 1 second 2
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Migration: 20250303231642 - FinalizeHkBackfillSecurityOrchestrationPolicyRuleSchedulesNamespa

     * Type: Post deploy
    * Duration: 6.9 s
    * Database size change: +0.00 B
    Calls Total Time Max Time Mean Time Rows Query
    2 0.0 ms 0.0 ms 0.0 ms 2 
    SELECT pg_backend_pid()
    Histogram for FinalizeHkBackfillSecurityOrchestrationPolicyRuleSchedulesNamespa
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 0
    0.1 seconds - 1 second 2
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Other information

    Other migrations pending on GitLab.com
    Migration Type Total runtime Result DB size change
    20250304132919 - AddVersionPartsToPackagesTerraformModuleMetadata Regular 9.8 s :white_check_mark: +0.00 B
    20250307100237 - AddProjectIdToPackagesDebianProjectComponentFiles Regular 7.1 s :white_check_mark: +0.00 B
    20250211220147 - ScheduleIndexOnMergeRequestDiffsIdAndProjectId Post deploy 9.4 s :white_check_mark: +0.00 B
    20250224063503 - DeleteRemoteDevelopmentNamespaceClusterAgentMappingView Post deploy 7.1 s :white_check_mark: +0.00 B
    20250225043739 - AddEvidencesProjectIdNotNull Post deploy 8.3 s :white_check_mark: +0.00 B
    20250225043936 - AddPackagesDebianProjectComponentsProjectIdNotNull Post deploy 8.2 s :white_check_mark: +0.00 B
    20250225044336 - AddPackagesMavenMetadataProjectIdNotNull Post deploy 8.2 s :white_check_mark: +8.00 KiB [note]
    20250227091623 - IndexRedirectRoutesOnNamespaceId Post deploy 7.9 s :white_check_mark: +8.00 KiB [note]
    20250227092023 - AddRedirectRoutesNamespaceIdFk Post deploy 8.2 s :white_check_mark: +0.00 B
    20250227231728 - FinalizeHkBackfillVulnerabilityOccurrenceIdentifiersProjectId Post deploy 6.8 s :white_check_mark: +0.00 B
    20250228142820 - IndexIncidentManagementPendingIssueEscalationsOnNamespaceId Post deploy 15.5 s :white_check_mark: +72.00 KiB
    20250228142821 - AddIncidentManagementPendingIssueEscalationsNamespaceIdFk Post deploy 19.3 s :white_check_mark: +24.00 KiB
    20250228142822 - AddIncidentManagementPendingIssueEscalationsNamespaceIdTrigger Post deploy 7.3 s :white_check_mark: +0.00 B
    20250228142823 - QueueBackfillIncidentManagementPendingIssueEscalationsNamespaceId Post deploy 7.0 s :white_check_mark: +0.00 B
    20250228143727 - RequeueSyncUnlinkedSecurityPolicyProjectLinks Post deploy 7.1 s :white_check_mark: +0.00 B
    20250228155146 - QueueBackfillOnboardingStatusSetupForCompany Post deploy 7.0 s :white_check_mark: +0.00 B
    20250303122513 - RemoveBrokenFkOnCiBuildsAndCiPipelines Post deploy 7.8 s :white_check_mark: +0.00 B
    20250303231437 - FinalizeHkBackfillProtectedEnvironmentApprovalRulesProtectedEnvir47878 Post deploy 6.8 s :white_check_mark: +0.00 B
    20250304084537 - AddDoraDailyMetricsProjectIdDateIndex Post deploy 9.0 s :white_check_mark: +0.00 B
    20250304204818 - DeleteIndexTodosOnAuthorIdAndCreatedAtIndex Post deploy 8.8 s :white_check_mark: -8.00 KiB
    20250304231245 - IncreaseTokenEncryptedConstraint Post deploy 10.1 s :warning: +0.00 B
    20250304231710 - FinalizeHkBackfillSecurityOrchestrationPolicyRuleSchedulesProject Post deploy 7.0 s :white_check_mark: +0.00 B
    20250305084646 - IndexScanResultPoliciesOnApprovalPolicyRuleId Post deploy 8.1 s :white_check_mark: +8.00 KiB [note]
    20250305232245 - FinalizeHkBackfillCiBuildTraceMetadataProjectId Post deploy 7.8 s :white_check_mark: +0.00 B
    20250307100238 - IndexPackagesDebianProjectComponentFilesOnProjectId Post deploy 7.9 s :white_check_mark: +8.00 KiB [note]
    20250307100239 - AddPackagesDebianProjectComponentFilesProjectIdFk Post deploy 8.0 s :white_check_mark: +0.00 B
    20250307100240 - AddPackagesDebianProjectComponentFilesProjectIdTrigger Post deploy 7.4 s :white_check_mark: +0.00 B
    20250307100241 - QueueBackfillPackagesDebianProjectComponentFilesProjectId Post deploy 7.0 s :white_check_mark: +0.00 B
    Clone details
    Clone ID Clone Created At Clone Data Timestamp Expected Removal Time
    database-testing-4267870-17540303-main 2025-03-08T12:11:23Z 2025-03-08T08:09:58Z 2025-03-09 00:32:49 +0000
    database-testing-4267870-17540303-ci 2025-03-08T12:11:24Z 2025-03-08T08:45:04Z 2025-03-09 00:32:49 +0000

    Job artifacts

    Brought to you by gitlab-org/database-team/gitlab-com-database-testing. Epic

    Edited by ****
  • **** added database-testing-automation label

    added database-testing-automation label

  • Martin Čavoj changed milestone to %17.10

    changed milestone to %17.10

  • Martin Čavoj added 1 commit

    added 1 commit

    • 835d43da - Finalize migration to backfill namespace_id in policy rule schedules

    Compare with previous version

  • Martin Čavoj added 1 commit

    added 1 commit

    • 0027993a - Finalize migration to backfill namespace_id in policy rule schedules

    Compare with previous version

  • Martin Čavoj added 90 commits

    added 90 commits

    Compare with previous version

  • Martin Čavoj added 39 commits

    added 39 commits

    Compare with previous version

  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
Please register or sign in to reply