Name column in audit event destination
What does this MR do and why?
Name column in audit event destination tables
This commit adds a name column in the audit event external destination table.
We need to add a name column to identify a destination uniquely. Adding unique column for this change is a multi-deployment process because we need to backfill data for existing records.
So the changes will go in the below order to avoid data failures.
- Add a name column with the default name value and a uniqueness check. (This MR pushes this change)
- Backfill data for nil values and change column null for the name column to false.
- Add name field in existing create update APIs.
Changelog: added
Migration Logs
❯ rake db:migrate:down:main VERSION=20230612091910
WARNING: This version of GitLab depends on gitlab-shell 14.23.0, but you're running 14.22.0. Please update gitlab-shell.
main: == [advisory_lock_connection] object_id: 227880, pg_backend_pid: 28391
main: == 20230612091910 AddTextLimitToInstanceAuditEventDestinationName: reverting ==
main: -- transaction_open?()
main: -> 0.0002s
main: -- transaction_open?()
main: -> 0.0001s
main: -- execute(" ALTER TABLE audit_events_instance_external_audit_event_destinations\n DROP CONSTRAINT IF EXISTS check_433fbb3305\n")
main: -> 0.0031s
main: == 20230612091910 AddTextLimitToInstanceAuditEventDestinationName: reverted (0.0591s)
main: == [advisory_lock_connection] object_id: 227880, pg_backend_pid: 28391
❯
❯ rake db:migrate:up:main VERSION=20230612074428
WARNING: This version of GitLab depends on gitlab-shell 14.23.0, but you're running 14.22.0. Please update gitlab-shell.
main: == [advisory_lock_connection] object_id: 227900, pg_backend_pid: 29474
main: == 20230612074428 AddNameToExternalAuditEventDestination: migrating ===========
main: -- add_column(:audit_events_external_audit_event_destinations, :name, :text)
main: -> 0.0051s
main: == 20230612074428 AddNameToExternalAuditEventDestination: migrated (0.0157s) ==
main: == [advisory_lock_connection] object_id: 227900, pg_backend_pid: 29474
❯ rake db:migrate:up:main VERSION=20230612091526
WARNING: This version of GitLab depends on gitlab-shell 14.23.0, but you're running 14.22.0. Please update gitlab-shell.
main: == [advisory_lock_connection] object_id: 227900, pg_backend_pid: 30441
main: == 20230612091526 AddTextLimitToExternalAuditEventDestinationName: migrating ==
main: -- transaction_open?()
main: -> 0.0000s
main: -- transaction_open?()
main: -> 0.0000s
main: -- execute("ALTER TABLE audit_events_external_audit_event_destinations\nADD CONSTRAINT check_c52ff8e90e\nCHECK ( char_length(name) <= 72 )\nNOT VALID;\n")
main: -> 0.0026s
main: -- execute("SET statement_timeout TO 0")
main: -> 0.0003s
main: -- execute("ALTER TABLE audit_events_external_audit_event_destinations VALIDATE CONSTRAINT check_c52ff8e90e;")
main: -> 0.0014s
main: -- execute("RESET statement_timeout")
main: -> 0.0006s
main: == 20230612091526 AddTextLimitToExternalAuditEventDestinationName: migrated (0.2781s)
main: == [advisory_lock_connection] object_id: 227900, pg_backend_pid: 30441
❯ rake db:migrate:up:main VERSION=20230612091747
WARNING: This version of GitLab depends on gitlab-shell 14.23.0, but you're running 14.22.0. Please update gitlab-shell.
main: == [advisory_lock_connection] object_id: 227880, pg_backend_pid: 31088
main: == 20230612091747 AddNameToInstanceAuditEventDestination: migrating ===========
main: -- add_column(:audit_events_instance_external_audit_event_destinations, :name, :text)
main: -> 0.0020s
main: == 20230612091747 AddNameToInstanceAuditEventDestination: migrated (0.0073s) ==
main: == [advisory_lock_connection] object_id: 227880, pg_backend_pid: 31088
❯ rake db:migrate:up:main VERSION=20230612091910
WARNING: This version of GitLab depends on gitlab-shell 14.23.0, but you're running 14.22.0. Please update gitlab-shell.
main: == [advisory_lock_connection] object_id: 227880, pg_backend_pid: 31636
main: == 20230612091910 AddTextLimitToInstanceAuditEventDestinationName: migrating ==
main: -- transaction_open?()
main: -> 0.0000s
main: -- transaction_open?()
main: -> 0.0000s
main: -- execute("ALTER TABLE audit_events_instance_external_audit_event_destinations\nADD CONSTRAINT check_433fbb3305\nCHECK ( char_length(name) <= 72 )\nNOT VALID;\n")
main: -> 0.0028s
main: -- execute("SET statement_timeout TO 0")
main: -> 0.0014s
main: -- execute("ALTER TABLE audit_events_instance_external_audit_event_destinations VALIDATE CONSTRAINT check_433fbb3305;")
main: -> 0.0016s
main: -- execute("RESET statement_timeout")
main: -> 0.0016s
main: == 20230612091910 AddTextLimitToInstanceAuditEventDestinationName: migrated (0.2666s)
main: == [advisory_lock_connection] object_id: 227880, pg_backend_pid: 31636
❯ rake db:migrate:down:main VERSION=20230612074428
WARNING: This version of GitLab depends on gitlab-shell 14.23.0, but you're running 14.22.0. Please update gitlab-shell.
main: == [advisory_lock_connection] object_id: 227600, pg_backend_pid: 24467
main: == 20230612074428 AddNameToExternalAuditEventDestination: reverting ===========
main: -- remove_column(:audit_events_external_audit_event_destinations, :name, :text)
main: -> 0.0038s
main: == 20230612074428 AddNameToExternalAuditEventDestination: reverted (0.0084s) ==
main: == [advisory_lock_connection] object_id: 227600, pg_backend_pid: 24467
❯ rake db:migrate:down:main VERSION=20230612091526
WARNING: This version of GitLab depends on gitlab-shell 14.23.0, but you're running 14.22.0. Please update gitlab-shell.
main: == [advisory_lock_connection] object_id: 227880, pg_backend_pid: 25092
main: == 20230612091526 AddTextLimitToExternalAuditEventDestinationName: reverting ==
main: -- transaction_open?()
main: -> 0.0000s
main: -- transaction_open?()
main: -> 0.0000s
main: -- execute(" ALTER TABLE audit_events_external_audit_event_destinations\n DROP CONSTRAINT IF EXISTS check_c52ff8e90e\n")
main: -> 0.0029s
main: == 20230612091526 AddTextLimitToExternalAuditEventDestinationName: reverted (0.0671s)
main: == [advisory_lock_connection] object_id: 227880, pg_backend_pid: 25092
❯
❯ rake db:migrate:down:main VERSION=20230612091747
WARNING: This version of GitLab depends on gitlab-shell 14.23.0, but you're running 14.22.0. Please update gitlab-shell.
main: == [advisory_lock_connection] object_id: 227880, pg_backend_pid: 25740
main: == 20230612091747 AddNameToInstanceAuditEventDestination: reverting ===========
main: -- remove_column(:audit_events_instance_external_audit_event_destinations, :name, :text)
main: -> 0.0271s
main: == 20230612091747 AddNameToInstanceAuditEventDestination: reverted (0.1306s) ==
main: == [advisory_lock_connection] object_id: 227880, pg_backend_pid: 25740
❯
❯ rake db:migrate:down:main VERSION=20230612091910
WARNING: This version of GitLab depends on gitlab-shell 14.23.0, but you're running 14.22.0. Please update gitlab-shell.
main: == [advisory_lock_connection] object_id: 227880, pg_backend_pid: 28391
main: == 20230612091910 AddTextLimitToInstanceAuditEventDestinationName: reverting ==
main: -- transaction_open?()
main: -> 0.0002s
main: -- transaction_open?()
main: -> 0.0001s
main: -- execute(" ALTER TABLE audit_events_instance_external_audit_event_destinations\n DROP CONSTRAINT IF EXISTS check_433fbb3305\n")
main: -> 0.0031s
main: == 20230612091910 AddTextLimitToInstanceAuditEventDestinationName: reverted (0.0591s)
main: == [advisory_lock_connection] object_id: 227880, pg_backend_pid: 28391
==== INDEX MIGRATIONS ======
❯ rake db:migrate:up:main VERSION=20230615074515
WARNING: This version of GitLab depends on gitlab-shell 14.23.0, but you're running 14.22.0. Please update gitlab-shell.
main: == [advisory_lock_connection] object_id: 227900, pg_backend_pid: 98285
main: == 20230615074515 AddIndexToAuditEventExternalDestination: migrating ==========
main: -- transaction_open?()
main: -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main: -> 0.4307s
main: -- index_exists?(:audit_events_external_audit_event_destinations, [:namespace_id, :name], {:unique=>true, :name=>"unique_external_audit_event_destination_namespace_id_and_name", :algorithm=>:concurrently})
main: -> 0.0346s
main: -- execute("SET statement_timeout TO 0")
main: -> 0.0038s
main: -- add_index(:audit_events_external_audit_event_destinations, [:namespace_id, :name], {:unique=>true, :name=>"unique_external_audit_event_destination_namespace_id_and_name", :algorithm=>:concurrently})
main: -> 0.0249s
main: -- execute("RESET statement_timeout")
main: -> 0.0066s
main: == 20230615074515 AddIndexToAuditEventExternalDestination: migrated (0.6192s) =
main: == [advisory_lock_connection] object_id: 227900, pg_backend_pid: 98285
❯ rake db:migrate:up:main VERSION=20230615074544
WARNING: This version of GitLab depends on gitlab-shell 14.23.0, but you're running 14.22.0. Please update gitlab-shell.
main: == [advisory_lock_connection] object_id: 264360, pg_backend_pid: 98920
main: == 20230615074544 AddIndexToInstanceAuditEventDestination: migrating ==========
main: -- transaction_open?()
main: -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main: -> 0.2549s
main: -- index_exists?(:audit_events_instance_external_audit_event_destinations, :name, {:unique=>true, :name=>"unique_instance_audit_event_destination_namespace_id_and_name", :algorithm=>:concurrently})
main: -> 0.0023s
main: -- execute("SET statement_timeout TO 0")
main: -> 0.0019s
main: -- add_index(:audit_events_instance_external_audit_event_destinations, :name, {:unique=>true, :name=>"unique_instance_audit_event_destination_namespace_id_and_name", :algorithm=>:concurrently})
main: -> 0.0038s
main: -- execute("RESET statement_timeout")
main: -> 0.0027s
main: == 20230615074544 AddIndexToInstanceAuditEventDestination: migrated (0.3008s) =
main: == [advisory_lock_connection] object_id: 264360, pg_backend_pid: 98920
❯ rake db:migrate:down:main VERSION=20230615074515
WARNING: This version of GitLab depends on gitlab-shell 14.23.0, but you're running 14.22.0. Please update gitlab-shell.
main: == [advisory_lock_connection] object_id: 227960, pg_backend_pid: 96967
main: == 20230615074515 AddIndexToAuditEventExternalDestination: reverting ==========
main: -- transaction_open?()
main: -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main: -> 0.1003s
main: -- indexes(:audit_events_external_audit_event_destinations)
main: -> 0.0037s
main: -- execute("SET statement_timeout TO 0")
main: -> 0.0003s
main: -- remove_index(:audit_events_external_audit_event_destinations, {:algorithm=>:concurrently, :name=>"unique_external_audit_event_destination_namespace_id_and_name"})
main: -> 0.0026s
main: -- execute("RESET statement_timeout")
main: -> 0.0003s
main: == 20230615074515 AddIndexToAuditEventExternalDestination: reverted (0.1200s) =
main: == [advisory_lock_connection] object_id: 227960, pg_backend_pid: 96967
❯ rake db:migrate:down:main VERSION=20230615074544
WARNING: This version of GitLab depends on gitlab-shell 14.23.0, but you're running 14.22.0. Please update gitlab-shell.
main: == [advisory_lock_connection] object_id: 227880, pg_backend_pid: 97643
main: == 20230615074544 AddIndexToInstanceAuditEventDestination: reverting ==========
main: -- transaction_open?()
main: -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main: -> 0.2073s
main: -- indexes(:audit_events_instance_external_audit_event_destinations)
main: -> 0.0042s
main: -- execute("SET statement_timeout TO 0")
main: -> 0.0011s
main: -- remove_index(:audit_events_instance_external_audit_event_destinations, {:algorithm=>:concurrently, :name=>"unique_instance_audit_event_destination_namespace_id_and_name"})
main: -> 0.0050s
main: -- execute("RESET statement_timeout")
main: -> 0.0025s
main: == 20230615074544 AddIndexToInstanceAuditEventDestination: reverted (0.2653s) =How to set up and validate locally
To test this change we can try to create or update a already existing destination in our database for this you can use graphql explorer, refer to: https://docs.gitlab.com/ee/administration/audit_event_streaming.html#use-the-api for mutations.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #413894 (closed)
Edited by Harsimar Sandhu