As part of a the update streaming audit event UI styles (Update streaming audit event UI styles (#411005 - closed)) there is a new field to add a name to a destination. This field to help users better distinguish between destinations.
Proposal
Add a field to the streaming audit events destination API called Name. This field is required to add a destination
Update to allow a name field to be provided in the following qraphql queries:
First we can send database MR for both external audit event destination and instance external audit event destination and then we can send another MR to add name field in APIs.
For migration MR we can follow below implementation plan
Introduce a new column name as a string, initially allowing for null values.
Backfill the name field for existing records with the format "Destination_#{record_id}".
Alter the name column to disallow null values.
Add a unique index on the combination of namespace_id and name.
In order to implement a new required name field in our streaming tables, we'll need to add a unique index that is scoped to both the namespace_id and the name column, with the constraint that the name cannot be null.
For existing records, we can backfill the name field with a default value like "Destination_#{record_id}".
The steps of the migration would then be as follows:
Introduce a new column name as a string, initially allowing for null values.
Backfill the name field for existing records with the format "Destination_#{record_id}".
Alter the name column to disallow null values.
Add a unique index on the combination of namespace_id and name.
This will ensure that the name field is always present and unique within each namespace_id, providing the desired constraints at both the application level and the database level.
I tested the migration queries on postgres.ai and observed that the execution time for each query was below 35 milliseconds, significantly less than our allowable time limit of 3 minutes. Therefore, I will proceed with creating a Merge Request (MR) to implement these changes using a regular migration.
Should the migration pipeline succeed and remain within the time limits, we can push these changes to production. If, however, we encounter issues with execution times during the migration pipeline, we can consider using a post-deploy migration as an alternative solution
Nope, @nrosandich This needs to be multi deployment step.
In the first migration:
A column named name will be added, allowing null values with the default value set as Destination_#{id}.
In the second migration:
Data will be backfilled, null values will be disallowed, and a unique index will be added.
These two migrations are performed in separate steps to ensure that no null values can be inserted into the database. If both steps were combined into a single migration, there could be an extremely rare edge case where a new destination is created while the backfill process is ongoing but before the disallow null step has started. This could result in a migration failure.
In this milestone, I will send the first MR. For the second MR, I have two options: either create a separate issue or update the milestone of the current one to 16.2 once the first MR is merged.
This feature issue does not have the documentation label.
Please add it if appropriate, because documentation is one of the aspects of our MR acceptance checklist.
Update: First MR !123354 (merged) is in review and should merge soon.
After the first MR !123354 (merged) is merged I will create another MR to backfill the data and continue working on this same issue. So I am updating the milestone to %16.2
@harsimarsandhu This issue looks like it may slip this current milestone. Can you leave a or to signify if you are on track to deliver this issue?
Please also consider updating the issue's Health Status or Milestone to reflect its current state,
and communicate with your Product Manager as appropriate.
mutationexternalAuditEventDestinationCreate{externalAuditEventDestinationCreate(input:{name:"New Name",destinationUrl:"https://mydomain.io/endpoint/",groupPath:"complianceframework"}){errorsexternalAuditEventDestination{idnamedestinationUrlverificationTokengroup{name}}}}mutationexternalAuditEventDestinationUpdate{externalAuditEventDestinationUpdate(input:{name:"New Name 2",id:"gid://gitlab/AuditEvents::ExternalAuditEventDestination/7"}){errorsexternalAuditEventDestination{idnamedestinationUrlverificationTokengroup{name}}}}queryListExternalAuditEventDestinations{group(fullPath:"complianceframework"){idexternalAuditEventDestinations{nodes{idname}}}}mutationinstanceExternalAuditEventDestinationCreate{instanceExternalAuditEventDestinationCreate(input:{destinationUrl:"https://mydomain.io/endpoint/ingest",name:"New Name"}){errorsinstanceExternalAuditEventDestination{destinationUrlidverificationToken}}}mutationinstanceExternalAuditEventDestinationUpdate{instanceExternalAuditEventDestinationUpdate(input:{id:"gid://gitlab/AuditEvents::InstanceExternalAuditEventDestination/10",name:"New Name 2"}){errorsinstanceExternalAuditEventDestination{destinationUrlidverificationToken}}}queryListInstanceExternalAuditEventDestinations{instanceExternalAuditEventDestinations{nodes{idname}}}