Skip to content

Adding migrations for gcp config name backfilling and non null column

Hitesh Raghuvanshi requested to merge 416572-backfill-name-gcp into master

What does this MR do and why?

As part of issue #416572 (closed), we need to add support for name attribute for GCP logging configuration which streams audit events to GCP. This MR is backfilling data in recently added name column in MR !127620 (merged) for existing entries in audit_events_google_cloud_logging_configurations table. This MR is quite similar to !124296 (merged) which was merged few days back for backfilling name column in audit_events_external_audit_event_destinations table.

Notes around this migration

  1. There are currently only 7 entries on gitlab.com for audit_events_google_cloud_logging_configurations as of 4th August 2023. This feature was recently introduced. Because of less no. of entries the migration will not timeout.
  2. Currently there is no way the name field in audit_events_google_cloud_logging_configurations table can be updated, so the values being backfilled will not impact any current functionality.
  3. The default destination name being given by this backfill migration will be like Destination <id>, here id is the record id.

Migration logs


# up for backfilling name data

RAILS_ENV=development bin/rails db:migrate:up:main VERSION=20230804064817
main: == [advisory_lock_connection] object_id: 224600, pg_backend_pid: 88053
main: == 20230804064817 BackfillGoogleCloudLoggingName: migrating ===================
main: == 20230804064817 BackfillGoogleCloudLoggingName: migrated (0.0246s) ==========

main: == [advisory_lock_connection] object_id: 224600, pg_backend_pid: 88053


# down for backfilling name data

RAILS_ENV=development bin/rails db:migrate:down:main VERSION=20230804064817
main: == [advisory_lock_connection] object_id: 224600, pg_backend_pid: 87225
main: == 20230804064817 BackfillGoogleCloudLoggingName: reverting ===================
main: == 20230804064817 BackfillGoogleCloudLoggingName: reverted (0.0038s) ==========

main: == [advisory_lock_connection] object_id: 224600, pg_backend_pid: 87225


# up for setting name as non null

RAILS_ENV=development bin/rails db:migrate:up:main VERSION=20230804065052
main: == [advisory_lock_connection] object_id: 224600, pg_backend_pid: 89065
main: == 20230804065052 AddNotNullToGcpConfigName: migrating ========================
main: -- change_column_null(:audit_events_google_cloud_logging_configurations, :name, false)
main:    -> 0.0024s
main: == 20230804065052 AddNotNullToGcpConfigName: migrated (0.0067s) ===============


# down for setting name as non null

RAILS_ENV=development bin/rails db:migrate:down:main VERSION=20230804065052
main: == [advisory_lock_connection] object_id: 224660, pg_backend_pid: 90094
main: == 20230804065052 AddNotNullToGcpConfigName: reverting ========================
main: -- change_column_null(:audit_events_google_cloud_logging_configurations, :name, true)
main:    -> 0.0016s
main: == 20230804065052 AddNotNullToGcpConfigName: reverted (0.0060s) ===============

main: == [advisory_lock_connection] object_id: 224660, pg_backend_pid: 90094

Query plan

Query:

UPDATE "audit_events_google_cloud_logging_configurations" SET name = 'Destination ' || id

https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/21086/commands/68889

How to set up and validate locally

  1. Run migration rake db:migrate
  2. Verify every row in table audit_events_google_cloud_logging_configurations has a name value that is unique and follows pattern Destination id

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #416572 (closed)

Edited by Hitesh Raghuvanshi

Merge request reports