Skip to content
Snippets Groups Projects
New look: Off

Migration of data from existing tables to new tables

    • Open Issue created by Hitesh Raghuvanshi @hraghuvanshi

    Implementation plan

    1. Consistency between old and new data
      1. Create: If any new destinations are being added for non-consolidated destinations then create the same data in new tables too.
      2. Update: Check if the non-consolidated destination being updated also exist in new table then update data for that destination too.
      3. Delete: Check if the non-consolidated destination being deleted also exist in new table then delete data for that destination too.
      4. How will we find out the link between non-consolidated and consolidated destination?
        1. name field is unique for destinations belonging to same namespace id and for an instance, so we can treat name as the bridge between old and new data tables.
      5. Where should we add these checks for create, update and delete?
        1. We can do it by creating hooks in ExternallyCommonDestinationable for after_create, after_update and after_destroy .
        2. Or We can do it in the mutations for the non-consolidated destinations where we are creating updating and deleting the records.
        3. We need to do this with a feature flag.
    2. Migration for moving all the data from non-consolidated tables to new tables and let step 1 make sure the data is consistent.
    3. Once the consistency of the data is ensured.
    4. We can start using the new tables for streaming.
    5. Start using new apis on frontend with feature flag.
    6. Start deprecating old apis and table.
    7. Cleanup of old tables.

    Table information:

    Legacy Tables (12):

    • HTTP
      • audit_events_external_audit_event_destinations
      • audit_events_instance_external_audit_event_destinations
      • headers:
        • audit_events_streaming_headers
        • instance_audit_events_streaming_headers
      • event type filters:
        • audit_events_streaming_event_type_filters
        • audit_events_streaming_instance_event_type_filters
      • namespace filters:
        • audit_events_streaming_http_group_namespace_filters
        • audit_events_streaming_http_instance_namespace_filters
    • AWS
      • audit_events_amazon_s3_configurations
      • audit_events_instance_amazon_s3_configurations
    • GCP
      • audit_events_google_cloud_logging_configurations
      • audit_events_instance_google_cloud_logging_configurations

    Streaming Tables (6):

    • audit_events_group_external_streaming_destinations
      • audit_events_group_streaming_event_type_filters [http only]
      • audit_events_streaming_group_namespace_filters [http only]
    • audit_events_instance_external_streaming_destinations
      • audit_events_instance_streaming_event_type_filters [http only]
      • audit_events_streaming_instance_namespace_filters [http only]
    Edited

    Designs

      Child items ...

      2. Activity

      Please register or sign in to reply