Skip to content
Snippets Groups Projects

Update BulkImports::Failure to display subrelation that failed to import

Merged Update BulkImports::Failure to display subrelation that failed to import
georgekoltsov/update-bulk-imports-failures-relations into master
Merged George Koltsov requested to merge georgekoltsov/update-bulk-imports-failures-relations into master

What does this MR do and why?

Direct Transfer API (and UI) shows failures for the things that we could not import. If a subrelation of a top level relation failed to import (e.g. MR diff note) - we still show top level relation as the one that failed to import. In reality, the top level relation is imported, but a subrelation did not. Which is confusing.

Update relation to include the failed to import subrelation, so it's more obvious what exactly failed to import.

Changelog: added

Migration Output

Click to expand 
main: == [advisory_lock_connection] object_id: 117600, pg_backend_pid: 75580
main: == 20240126095808 AddSubrelationColumnToBulkImportFailures: migrating =========
main: -- add_column(:bulk_import_failures, :subrelation, :text)
main:    -> 0.0029s
main: == 20240126095808 AddSubrelationColumnToBulkImportFailures: migrated (0.0068s) 

main: == [advisory_lock_connection] object_id: 117600, pg_backend_pid: 75580
ci: == [advisory_lock_connection] object_id: 117860, pg_backend_pid: 75585
ci: == 20240126095808 AddSubrelationColumnToBulkImportFailures: migrating =========
ci: -- add_column(:bulk_import_failures, :subrelation, :text)
ci:    -> 0.0019s
ci: == 20240126095808 AddSubrelationColumnToBulkImportFailures: migrated (0.0105s) 

ci: == [advisory_lock_connection] object_id: 117860, pg_backend_pid: 75585
main: == [advisory_lock_connection] object_id: 118020, pg_backend_pid: 75589
main: == 20240126100509 AddTextLimitToBulkImportFailuresSubrelationColumn: migrating 
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- execute("ALTER TABLE bulk_import_failures\nADD CONSTRAINT check_f99665a440\nCHECK ( char_length(subrelation) <= 255 )\nNOT VALID;\n")
main:    -> 0.0019s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0002s
main: -- execute("ALTER TABLE bulk_import_failures VALIDATE CONSTRAINT check_f99665a440;")
main:    -> 0.0008s
main: -- execute("RESET statement_timeout")
main:    -> 0.0002s
main: == 20240126100509 AddTextLimitToBulkImportFailuresSubrelationColumn: migrated (0.0289s) 

main: == [advisory_lock_connection] object_id: 118020, pg_backend_pid: 75589
ci: == [advisory_lock_connection] object_id: 118160, pg_backend_pid: 75591
ci: == 20240126100509 AddTextLimitToBulkImportFailuresSubrelationColumn: migrating 
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- execute("ALTER TABLE bulk_import_failures\nADD CONSTRAINT check_f99665a440\nCHECK ( char_length(subrelation) <= 255 )\nNOT VALID;\n")
ci:    -> 0.0013s
ci: -- execute("SET statement_timeout TO 0")
ci:    -> 0.0004s
ci: -- execute("ALTER TABLE bulk_import_failures VALIDATE CONSTRAINT check_f99665a440;")
ci:    -> 0.0008s
ci: -- execute("RESET statement_timeout")
ci:    -> 0.0003s
ci: == 20240126100509 AddTextLimitToBulkImportFailuresSubrelationColumn: migrated (0.0144s) 

ci: == [advisory_lock_connection] object_id: 118160, pg_backend_pid: 75591

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Edited by George Koltsov

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • George Koltsov requested review from @SamWord

    requested review from @SamWord

  • Ghost User
    Ghost User @ghost1
    Contributor

    E2E Test Result Summary

    allure-report-publisher generated test report!

    e2e-test-on-gdk: :white_check_mark: test report for 8d44a19f

    expand test summary 
    +------------------------------------------------------------------+
|                          suites summary                          |
+-------------+--------+--------+---------+-------+-------+--------+
|             | passed | failed | skipped | flaky | total | result |
+-------------+--------+--------+---------+-------+-------+--------+
| Manage      | 0      | 0      | 1       | 0     | 1     | ➖     |
| Create      | 8      | 0      | 3       | 0     | 11    | ✅     |
| Monitor     | 4      | 0      | 0       | 0     | 4     | ✅     |
| Plan        | 4      | 0      | 0       | 0     | 4     | ✅     |
| Package     | 0      | 0      | 1       | 0     | 1     | ➖     |
| Data Stores | 0      | 0      | 2       | 0     | 2     | ➖     |
| Govern      | 3      | 0      | 0       | 0     | 3     | ✅     |
+-------------+--------+--------+---------+-------+-------+--------+
| Total       | 19     | 0      | 7       | 0     | 26    | ✅     |
+-------------+--------+--------+---------+-------+-------+--------+

    e2e-package-and-test: :white_check_mark: test report for 8d44a19f

    expand test summary 
    +------------------------------------------------------------------+
|                          suites summary                          |
+-------------+--------+--------+---------+-------+-------+--------+
|             | passed | failed | skipped | flaky | total | result |
+-------------+--------+--------+---------+-------+-------+--------+
| Manage      | 39     | 0      | 11      | 3     | 50    | ✅     |
| Create      | 148    | 0      | 39      | 4     | 187   | ✅     |
| Monitor     | 8      | 0      | 0       | 0     | 8     | ✅     |
| Data Stores | 0      | 0      | 4       | 0     | 4     | ➖     |
| Govern      | 6      | 0      | 0       | 0     | 6     | ✅     |
| Package     | 0      | 0      | 2       | 0     | 2     | ➖     |
| Plan        | 8      | 0      | 0       | 0     | 8     | ✅     |
+-------------+--------+--------+---------+-------+-------+--------+
| Total       | 209    | 0      | 56      | 7     | 265   | ✅     |
+-------------+--------+--------+---------+-------+-------+--------+
    Edited by Ghost User
  • Sam Word
  • Sam Word
    Sam Word @SamWord
    Maintainer

    I left a comment on moving away from constantize-able pipeline_names on BulkImport::Failures. This is a good change though, I think our users would want to know exactly what subrelation is failing

  • 🤖 GitLab Bot 🤖 added groupimport and integrate label

    added groupimport and integrate label

  • George Koltsov changed the description

    changed the description

  • 🤖 GitLab Bot 🤖 added devopsmanage sectioncore platform labels

    added devopsmanage sectioncore platform labels

  • George Koltsov added 1 commit

    added 1 commit

    • 1f7b4df2 - Update BulkImports::Failure to display subrelation that failed to import

    Compare with previous version

  • A deleted user added database databasereview pending labels

    added database databasereview pending labels

  • A deleted user added Data WarehouseImpact Check label

    added Data WarehouseImpact Check label

  • Ghost User
    Ghost User @ghost1
    Contributor

    Database migrations (on the main database)

    1 Warnings
    :warning: 20240126100509 - AddTextLimitToBulkImportFailuresSubrelationColumn had a query that exceeded
    timing guidelines    . Run time should not exceed 100ms, but it was 737.92ms. Please consider possible
    options to improve the query performance. 
    ALTER TABLE bulk_import_failures VALIDATE
     CONSTRAINT check_f99665a440

    Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

    Migration Type Total runtime Result DB size change
    20240126095808 - AddSubrelationColumnToBulkImportFailures Regular 1.9 s :white_check_mark: +0.00 B
    20240126100509 - AddTextLimitToBulkImportFailuresSubrelationColumn Regular 2.7 s :warning: +0.00 B
    Runtime Histogram for all migrations
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 7
    0.1 seconds - 1 second 1
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Migration: 20240126095808 - AddSubrelationColumnToBulkImportFailures

    • Type: Regular
    • Duration: 1.9 s
    • Database size change: +0.00 B
    Calls Total Time Max Time Mean Time Rows Query
    1 9.0 ms 9.0 ms 9.0 ms 0 
    ALTER TABLE "bulk_import_failures" ADD "subrelation" text
    1 1.1 ms 1.1 ms 1.1 ms 1 
    SELECT "feature_gates"."key", "feature_gates"."value"  FROM "feature_gates"  WHERE "feature_gates"."feature_key" = $1
    2 0.0 ms 0.0 ms 0.0 ms 2 
    SELECT pg_backend_pid()
    Histogram for AddSubrelationColumnToBulkImportFailures
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 4
    0.1 seconds - 1 second 0
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    :warning: Migration: 20240126100509 - AddTextLimitToBulkImportFailuresSubrelationColumn

    • Type: Regular
    • Duration: 2.7 s
    • Database size change: +0.00 B
    Calls Total Time Max Time Mean Time Rows Query
    1 737.9 ms 737.9 ms 737.9 ms 0 
    ALTER TABLE bulk_import_failures VALIDATE CONSTRAINT check_f99665a440
    1 6.7 ms 6.7 ms 6.7 ms 0 
    ALTER TABLE bulk_import_failures ADD CONSTRAINT check_f99665a440 CHECK ( char_length(subrelation) <= 255 ) NOT VALID
    2 0.0 ms 0.0 ms 0.0 ms 2 
    SELECT pg_backend_pid()
    Histogram for AddTextLimitToBulkImportFailuresSubrelationColumn
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 3
    0.1 seconds - 1 second 1
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Background Migration: BackfillCatalogResourceVersionsReleasedAt

    Sampled 778 batches. Estimated Time to complete: 4 minutes
    • Interval: 120s
    • Total tuple count: 1009
    • Max batch size: 0
    • Estimated seconds to complete: 240s
    • Estimated number of batches: 2
    • Average batch time: 1.63s
    • Batch size: 500
    • N. of batches sampled: 778
    • N. of failed batches: 0

    Time estimation is conservative and based on sampling production data in a test environment. It represents the max time that migration could take. The actual time may differ from this estimation.

    _Consider changing max_batch_size and interval if this estimate is unacceptable._

    Calls Total Time Max Time Mean Time Rows Query
    3112 5132.3 ms 536.2 ms 1.6 ms 311200 
    UPDATE catalog_resource_versions
      SET released_at = releases.released_at
      FROM releases
      WHERE catalog_resource_versions.id BETWEEN $1 AND $2 AND catalog_resource_versions.id >= $3 AND catalog_resource_versions.id < $4 AND release_id = releases.id
    778 1135.3 ms 277.1 ms 1.5 ms 77737 
    UPDATE catalog_resource_versions
      SET released_at = releases.released_at
      FROM releases
      WHERE catalog_resource_versions.id BETWEEN $1 AND $2 AND catalog_resource_versions.id >= $3 AND release_id = releases.id
    1556 393.4 ms 87.3 ms 0.3 ms 1556 
    INSERT INTO batched_background_migration_job_transition_logs (batched_background_migration_job_id, created_at, updated_at, previous_status, next_status) VALUES ($1, $2, $3, $4, $5) RETURNING id
    778 288.8 ms 74.4 ms 0.4 ms 778 
    SELECT sum(batched_background_migration_jobs.batch_size)
      FROM batched_background_migration_jobs
      WHERE batched_background_migration_jobs.batched_background_migration_id = $1 AND batched_background_migration_jobs.status IN ($2)
    1556 97.4 ms 10.0 ms 0.1 ms 1556 
    SELECT batched_background_migration_jobs.*
      FROM batched_background_migration_jobs
      WHERE batched_background_migration_jobs.id = $1
      LIMIT $2
    3890 392.3 ms 6.4 ms 0.1 ms 3112 
    SELECT catalog_resource_versions.id
      FROM catalog_resource_versions
      WHERE catalog_resource_versions.id BETWEEN $1 AND $2 AND catalog_resource_versions.id >= $3
      ORDER BY catalog_resource_versions.id ASC
      LIMIT $4
      OFFSET $5
    778 120.7 ms 6.0 ms 0.2 ms 778 
    UPDATE batched_background_migration_jobs
      SET updated_at = $1, started_at = $2, status = $3, attempts = $4
      WHERE batched_background_migration_jobs.id = $5
    778 144.7 ms 3.4 ms 0.2 ms 778 
    UPDATE batched_background_migration_jobs
      SET updated_at = $1, finished_at = $2, status = $3, metrics = $4
      WHERE batched_background_migration_jobs.id = $5
    778 21.4 ms 0.7 ms 0.0 ms 778 
    SELECT catalog_resource_versions.id
      FROM catalog_resource_versions
      WHERE catalog_resource_versions.id BETWEEN $1 AND $2
      ORDER BY catalog_resource_versions.id ASC
      LIMIT $3
    778 34.1 ms 0.4 ms 0.0 ms 778 
    SELECT batched_background_migrations.*
      FROM batched_background_migrations
      WHERE batched_background_migrations.id = $1
      LIMIT $2
    Histogram of batch runtimes for BackfillCatalogResourceVersionsReleasedAt
    Batch Runtime Count
    0 seconds - 10 seconds 778
    10 seconds - 1 minute 0
    1 minute - 2 minutes 0
    2 minutes - 3 minutes 0
    3 minutes - 5 minutes 0
    5 minutes + 0
    Histogram across all sampled batches of BackfillCatalogResourceVersionsReleasedAt
    Query Runtime Count
    0 seconds - 0.1 seconds 14701
    0.1 seconds - 0.5 seconds 79
    0.5 seconds - 1 second 1
    1 second - 2 seconds 0
    2 seconds - 5 seconds 1
    5 seconds + 0

    Other information

    Other migrations pending on GitLab.com
    Migration Type Total runtime Result DB size change
    20240118225727 - QueueBackfillCatalogResourceVersionsReleasedAt Post deploy 2.1 s :white_check_mark: +0.00 B
    20240122071840 - EnsureBackfillForCiBuildsIntegerColumnsIsFinished Post deploy 1.4 s :white_check_mark: +0.00 B
    20240123071840 - PrepareAsyncIndexesForPCiBuildsAutoCanceledById Post deploy 4.0 s :white_check_mark: +8.00 KiB [note]
    20240123101716 - EnsureIdUniquenessForPCiPipelineVariables Post deploy 2.1 s :warning: +0.00 B
    20240124073815 - SwapPrimaryKeyCiJobArtifacts Post deploy 1.7 s :white_check_mark: -8.00 KiB
    20240124081840 - PrepareAsyncIndexesForPCiBuildsCommitIdPart1 Post deploy 6.0 s :white_check_mark: +0.00 B
    20240124171424 - CreateAsyncIndexMembersOnSourceAndTypeAndId Post deploy 2.2 s :white_check_mark: +0.00 B
    Clone details
    Clone ID Clone Created At Clone Data Timestamp Expected Removal Time
    database-testing-2791924-12639497-main 2024-01-26T11:30:43Z 2024-01-26T05:08:48Z 2024-01-27 00:06:35 +0000
    database-testing-2791924-12639497-ci 2024-01-26T11:30:43Z 2024-01-26T08:45:46Z 2024-01-27 00:06:35 +0000

    Job artifacts

    Database migrations (on the ci database)

    1 Warnings
    :warning: 20240126100509 - AddTextLimitToBulkImportFailuresSubrelationColumn had a query that exceeded
    timing guidelines    . Run time should not exceed 100ms, but it was 110.93ms. Please consider possible
    options to improve the query performance. 
    ALTER TABLE bulk_import_failures ADD CONSTRAINT
     check_f99665a440 CHECK ( char_length(subrelation) <= 255 ) NOT VALID

    Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

    Migration Type Total runtime Result DB size change
    20240126095808 - AddSubrelationColumnToBulkImportFailures Regular 2.7 s :white_check_mark: +0.00 B
    20240126100509 - AddTextLimitToBulkImportFailuresSubrelationColumn Regular 3.0 s :warning: +0.00 B
    Runtime Histogram for all migrations
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 6
    0.1 seconds - 1 second 1
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Migration: 20240126095808 - AddSubrelationColumnToBulkImportFailures

    • Type: Regular
    • Duration: 2.7 s
    • Database size change: +0.00 B
    Calls Total Time Max Time Mean Time Rows Query
    1 13.6 ms 13.6 ms 13.6 ms 0 
    ALTER TABLE "bulk_import_failures" ADD "subrelation" text
    2 0.0 ms 0.0 ms 0.0 ms 2 
    SELECT pg_backend_pid()
    Histogram for AddSubrelationColumnToBulkImportFailures
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 3
    0.1 seconds - 1 second 0
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    :warning: Migration: 20240126100509 - AddTextLimitToBulkImportFailuresSubrelationColumn

    • Type: Regular
    • Duration: 3.0 s
    • Database size change: +0.00 B
    Calls Total Time Max Time Mean Time Rows Query
    1 110.9 ms 110.9 ms 110.9 ms 0 
    ALTER TABLE bulk_import_failures ADD CONSTRAINT check_f99665a440 CHECK ( char_length(subrelation) <= 255 ) NOT VALID
    1 18.6 ms 18.6 ms 18.6 ms 0 
    ALTER TABLE bulk_import_failures VALIDATE CONSTRAINT check_f99665a440
    2 0.0 ms 0.0 ms 0.0 ms 2 
    SELECT pg_backend_pid()
    Histogram for AddTextLimitToBulkImportFailuresSubrelationColumn
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 3
    0.1 seconds - 1 second 1
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Other information

    Other migrations pending on GitLab.com
    Migration Type Total runtime Result DB size change
    20240118225727 - QueueBackfillCatalogResourceVersionsReleasedAt Post deploy 2.3 s :white_check_mark: +0.00 B
    20240122071840 - EnsureBackfillForCiBuildsIntegerColumnsIsFinished Post deploy 2.7 s :white_check_mark: +0.00 B
    20240123071840 - PrepareAsyncIndexesForPCiBuildsAutoCanceledById Post deploy 5.1 s :white_check_mark: +8.00 KiB [note]
    20240123101716 - EnsureIdUniquenessForPCiPipelineVariables Post deploy 3.7 s :warning: +0.00 B
    20240124073815 - SwapPrimaryKeyCiJobArtifacts Post deploy 19.9 s :white_check_mark: -79.83 GiB
    20240124081840 - PrepareAsyncIndexesForPCiBuildsCommitIdPart1 Post deploy 7.3 s :white_check_mark: +0.00 B
    20240124171424 - CreateAsyncIndexMembersOnSourceAndTypeAndId Post deploy 3.3 s :white_check_mark: +0.00 B
    Clone details
    Clone ID Clone Created At Clone Data Timestamp Expected Removal Time
    database-testing-2791924-12639497-main 2024-01-26T11:30:43Z 2024-01-26T05:08:48Z 2024-01-27 00:06:35 +0000
    database-testing-2791924-12639497-ci 2024-01-26T11:30:43Z 2024-01-26T08:45:46Z 2024-01-27 00:06:35 +0000

    Job artifacts

    Brought to you by gitlab-org/database-team/gitlab-com-database-testing. Epic

    Edited by Ghost User
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
    • Please register or sign in to reply