Skip to content
Snippets Groups Projects

Persist gists import failures

Merged Tetiana Zavediuk requested to merge tz-384173-track-gists-import-failure into master
All threads resolved!

What does this MR do and why?

If a gist fails to be imported, we will save the failure in the database to be able to present any failed imports to the user in the future.

This implements Track gists import failure.

Screenshots or screen recordings

gists-import-failure

How to set up and validate locally

  1. Use GitLab API to try to import one or more GitHub gists with more than 10 files.
  2. In rails console make sure that related failures have been saved to the database:
ImportFailure.where(user: user) # 'user' - GitLab user who initiated the gists import

MR acceptance checklist

Edited by Tetiana Zavediuk

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Tetiana Zavediuk changed milestone to %16.0

    changed milestone to %16.0

  • Tetiana Zavediuk added 1 deleted label

    added 1 deleted label

  • Contributor
    1 Warning
    :warning: a9fe0816: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines.

    Reviewer roulette

    Changes that require review have been detected!

    Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:

    Category Reviewer Maintainer
    backend Ian Baum current availability (@ibaum) (UTC-5) Mark Chao current availability (@lulalala) (UTC+8)
    database Vitali Tatarintev current availability (@ck3g) (UTC+2) Adam Hegyi current availability (@ahegyi) (UTC+2)
    ~"migration" No reviewer available No maintainer available

    To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.

    To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.

    Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.

    If needed, you can retry the :repeat: danger-review job that generated this comment.

    Generated by :no_entry_sign: Danger

  • A deleted user added Data WarehouseImpact Check label
  • Tetiana Zavediuk requested review from @ghavenga

    requested review from @ghavenga

  • requested review from @tianwenchen

  • Tetiana Zavediuk changed the description

    changed the description

  • added workflowin review label and removed workflowin dev label

    • Contributor
      Resolved by Ahmed Hemdan

      Database migrations (on the main database)

      1 Warnings
      :warning: 20230510142931 - AddForeignKeyForUserIdToImportFailures had a query that exceeded timing
      guidelines
      . Run time should not exceed 100ms, but it was 120.09ms. Please consider possible options
      to improve the query performance.
      ALTER TABLE import_failures VALIDATE CONSTRAINT
      fk_9a9b9ba21c

      Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

      Migration Type Total runtime Result DB size change
      20230510141454 - AddUserIdToImportFailures Regular 2.1 s :white_check_mark: +0.00 B
      20230510142316 - AddIndexOnUserIdToImportFailures Regular 81.4 s :white_check_mark: +8.00 KiB
      20230510142931 - AddForeignKeyForUserIdToImportFailures Regular 2.2 s :warning: +0.00 B
      Runtime Histogram for all migrations
      Query Runtime Count
      0 seconds - 0.01 seconds 0
      0.01 seconds - 0.1 seconds 12
      0.1 seconds - 1 second 1
      1 second - 5 seconds 0
      5 seconds - 15 seconds 0
      15 seconds - 5 minutes 1
      5 minutes + 0

      Migration: 20230510141454 - AddUserIdToImportFailures

      • Type: Regular
      • Duration: 2.1 s
      • Database size change: +0.00 B
      Calls Total Time Max Time Mean Time Rows Query
      1 5.4 ms 5.4 ms 5.4 ms 825
      SELECT "features".*
      FROM "features"
      1 4.0 ms 4.0 ms 4.0 ms 0
      ALTER TABLE "import_failures" ADD "user_id" bigint
      2 0.0 ms 0.0 ms 0.0 ms 2
      SELECT pg_backend_pid()
      Histogram for AddUserIdToImportFailures
      Query Runtime Count
      0 seconds - 0.01 seconds 0
      0.01 seconds - 0.1 seconds 4
      0.1 seconds - 1 second 0
      1 second - 5 seconds 0
      5 seconds - 15 seconds 0
      15 seconds - 5 minutes 0
      5 minutes + 0

      Migration: 20230510142316 - AddIndexOnUserIdToImportFailures

      • Type: Regular
      • Duration: 81.4 s
      • Database size change: +8.00 KiB
      Calls Total Time Max Time Mean Time Rows Query
      1 78925.7 ms 78925.7 ms 78925.7 ms 0
      CREATE INDEX CONCURRENTLY "index_import_failures_on_user_id_not_null" ON "import_failures" ("user_id")
      WHERE user_id IS NOT NULL
      2 0.0 ms 0.0 ms 0.0 ms 2
      SELECT pg_backend_pid()
      Histogram for AddIndexOnUserIdToImportFailures
      Query Runtime Count
      0 seconds - 0.01 seconds 0
      0.01 seconds - 0.1 seconds 2
      0.1 seconds - 1 second 0
      1 second - 5 seconds 0
      5 seconds - 15 seconds 0
      15 seconds - 5 minutes 1
      5 minutes + 0

      :warning: Migration: 20230510142931 - AddForeignKeyForUserIdToImportFailures

      • Type: Regular
      • Duration: 2.2 s
      • Database size change: +0.00 B
      Calls Total Time Max Time Mean Time Rows Query
      1 120.1 ms 120.1 ms 120.1 ms 0
      ALTER TABLE import_failures VALIDATE CONSTRAINT fk_9a9b9ba21c
      1 42.8 ms 42.8 ms 42.8 ms 0
      ALTER TABLE import_failures ADD CONSTRAINT fk_9a9b9ba21c FOREIGN KEY (user_id) REFERENCES users (id) ON DELETE CASCADE NOT VALID
      2 0.4 ms 0.2 ms 0.2 ms 0
      SELECT "postgres_partitioned_tables".*
      FROM "postgres_partitioned_tables" WHERE (identifier = concat(current_schema(), $1, $2))
      LIMIT $3
      1 0.1 ms 0.1 ms 0.1 ms 0
      SELECT $1 AS one
      FROM "postgres_foreign_keys" WHERE "postgres_foreign_keys"."constrained_table_name" = $2 AND "postgres_foreign_keys"."referenced_table_name" = $3 AND "postgres_foreign_keys"."name" = $4 AND "postgres_foreign_keys"."constrained_columns" = $5 AND "postgres_foreign_keys"."referenced_columns" = $6 AND "postgres_foreign_keys"."on_delete_action" = $7
      LIMIT $8
      2 0.0 ms 0.0 ms 0.0 ms 2
      SELECT pg_backend_pid()
      Histogram for AddForeignKeyForUserIdToImportFailures
      Query Runtime Count
      0 seconds - 0.01 seconds 0
      0.01 seconds - 0.1 seconds 6
      0.1 seconds - 1 second 1
      1 second - 5 seconds 0
      5 seconds - 15 seconds 0
      15 seconds - 5 minutes 0
      5 minutes + 0

      Other information

      No other migrations pending on GitLab.com

      Clone details
      Clone ID Clone Created At Clone Data Timestamp Expected Removal Time
      database-testing-1928995-10016763-main 2023-05-11T05:28:02Z 2023-05-11T03:05:00Z 2023-05-11 17:35:12 +0000
      database-testing-1928995-10016763-ci 2023-05-11T05:28:02Z 2023-05-11T04:45:16Z 2023-05-11 17:35:12 +0000

      Job artifacts

      Database migrations (on the ci database)

      2 Warnings
      :warning: 20230510142931 - AddForeignKeyForUserIdToImportFailures had a query that exceeded timing
      guidelines
      . Run time should not exceed 100ms, but it was 604.26ms. Please consider possible options
      to improve the query performance.
      ALTER TABLE import_failures VALIDATE CONSTRAINT
      fk_9a9b9ba21c
      :warning: 20230510142931 - AddForeignKeyForUserIdToImportFailures had a query that exceeded timing
      guidelines
      . Run time should not exceed 100ms, but it was 197.88ms. Please consider possible options
      to improve the query performance.
      ALTER TABLE import_failures ADD CONSTRAINT fk_9a9b9ba21c
      FOREIGN KEY (user_id) REFERENCES users (id) ON DELETE CASCADE NOT VALID

      Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

      Migration Type Total runtime Result DB size change
      20230510141454 - AddUserIdToImportFailures Regular 2.3 s :white_check_mark: +0.00 B
      20230510142316 - AddIndexOnUserIdToImportFailures Regular 3.3 s :white_check_mark: +8.00 KiB
      20230510142931 - AddForeignKeyForUserIdToImportFailures Regular 3.6 s :warning: +0.00 B
      Runtime Histogram for all migrations
      Query Runtime Count
      0 seconds - 0.01 seconds 0
      0.01 seconds - 0.1 seconds 10
      0.1 seconds - 1 second 3
      1 second - 5 seconds 0
      5 seconds - 15 seconds 0
      15 seconds - 5 minutes 0
      5 minutes + 0

      Migration: 20230510141454 - AddUserIdToImportFailures

      • Type: Regular
      • Duration: 2.3 s
      • Database size change: +0.00 B
      Calls Total Time Max Time Mean Time Rows Query
      1 16.4 ms 16.4 ms 16.4 ms 0
      ALTER TABLE "import_failures" ADD "user_id" bigint
      2 0.0 ms 0.0 ms 0.0 ms 2
      SELECT pg_backend_pid()
      Histogram for AddUserIdToImportFailures
      Query Runtime Count
      0 seconds - 0.01 seconds 0
      0.01 seconds - 0.1 seconds 3
      0.1 seconds - 1 second 0
      1 second - 5 seconds 0
      5 seconds - 15 seconds 0
      15 seconds - 5 minutes 0
      5 minutes + 0

      Migration: 20230510142316 - AddIndexOnUserIdToImportFailures

      • Type: Regular
      • Duration: 3.3 s
      • Database size change: +8.00 KiB
      Calls Total Time Max Time Mean Time Rows Query
      1 33.4 ms 33.4 ms 33.4 ms 0
      CREATE INDEX CONCURRENTLY "index_import_failures_on_user_id_not_null" ON "import_failures" ("user_id")
      WHERE user_id IS NOT NULL
      2 0.0 ms 0.0 ms 0.0 ms 2
      SELECT pg_backend_pid()
      Histogram for AddIndexOnUserIdToImportFailures
      Query Runtime Count
      0 seconds - 0.01 seconds 0
      0.01 seconds - 0.1 seconds 3
      0.1 seconds - 1 second 0
      1 second - 5 seconds 0
      5 seconds - 15 seconds 0
      15 seconds - 5 minutes 0
      5 minutes + 0

      :warning: Migration: 20230510142931 - AddForeignKeyForUserIdToImportFailures

      • Type: Regular
      • Duration: 3.6 s
      • Database size change: +0.00 B
      Calls Total Time Max Time Mean Time Rows Query
      1 604.3 ms 604.3 ms 604.3 ms 0
      ALTER TABLE import_failures VALIDATE CONSTRAINT fk_9a9b9ba21c
      1 197.9 ms 197.9 ms 197.9 ms 0
      ALTER TABLE import_failures ADD CONSTRAINT fk_9a9b9ba21c FOREIGN KEY (user_id) REFERENCES users (id) ON DELETE CASCADE NOT VALID
      1 10.3 ms 10.3 ms 10.3 ms 0
      SELECT $1 AS one
      FROM "postgres_foreign_keys" WHERE "postgres_foreign_keys"."constrained_table_name" = $2 AND "postgres_foreign_keys"."referenced_table_name" = $3 AND "postgres_foreign_keys"."name" = $4 AND "postgres_foreign_keys"."constrained_columns" = $5 AND "postgres_foreign_keys"."referenced_columns" = $6 AND "postgres_foreign_keys"."on_delete_action" = $7
      LIMIT $8
      2 0.5 ms 0.3 ms 0.3 ms 0
      SELECT "postgres_partitioned_tables".*
      FROM "postgres_partitioned_tables" WHERE (identifier = concat(current_schema(), $1, $2))
      LIMIT $3
      2 0.0 ms 0.0 ms 0.0 ms 2
      SELECT pg_backend_pid()
      Histogram for AddForeignKeyForUserIdToImportFailures
      Query Runtime Count
      0 seconds - 0.01 seconds 0
      0.01 seconds - 0.1 seconds 4
      0.1 seconds - 1 second 3
      1 second - 5 seconds 0
      5 seconds - 15 seconds 0
      15 seconds - 5 minutes 0
      5 minutes + 0

      Other information

      No other migrations pending on GitLab.com

      Clone details
      Clone ID Clone Created At Clone Data Timestamp Expected Removal Time
      database-testing-1928995-10016763-main 2023-05-11T05:28:02Z 2023-05-11T03:05:00Z 2023-05-11 17:35:12 +0000
      database-testing-1928995-10016763-ci 2023-05-11T05:28:02Z 2023-05-11T04:45:16Z 2023-05-11 17:35:12 +0000

      Job artifacts


      Brought to you by gitlab-org/database-team/gitlab-com-database-testing. Epic

  • Tianwen Chen approved this merge request

    approved this merge request

  • Tianwen Chen removed review request for @tianwenchen

    removed review request for @tianwenchen

  • Gregory Havenga approved this merge request

    approved this merge request

  • Gregory Havenga requested review from @minac and removed review request for @ghavenga

    requested review from @minac and removed review request for @ghavenga

  • Tetiana Zavediuk requested review from @ahegyi

    requested review from @ahegyi

  • Mehmet Emin INAC removed review request for @minac

    removed review request for @minac

  • Adam Hegyi
  • added databaseapproved label and removed databasereview pending label

  • Adam Hegyi approved this merge request

    approved this merge request

  • added 1 commit

    • 3b92f82b - Apply reviewer's suggestions

    Compare with previous version

  • Tetiana Zavediuk added 1686 commits

    added 1686 commits

    Compare with previous version

  • requested review from @ahmed.hemdan

  • 🤖 GitLab Bot 🤖 changed milestone to %16.1

    changed milestone to %16.1

  • Ahmed Hemdan
  • Donald Cook added typemaintenance label and removed typefeature label

    added typemaintenance label and removed typefeature label

  • 🤖 GitLab Bot 🤖 removed 1 deleted label

    removed 1 deleted label

  • Tetiana Zavediuk added 444 commits

    added 444 commits

    Compare with previous version

  • Tetiana Zavediuk changed the description

    changed the description

  • Ahmed Hemdan approved this merge request

    approved this merge request

  • Ahmed Hemdan resolved all threads

    resolved all threads

  • Ahmed Hemdan enabled an automatic merge when the pipeline for b30e344a succeeds

    enabled an automatic merge when the pipeline for b30e344a succeeds

  • merged

  • Hello @cervols-ext :wave:

    The database team is looking for ways to improve the database review process and we would love your help!

    If you'd be open to someone on the database team reaching out to you for a chat, or if you'd like to leave some feedback asynchronously, just post a reply to this comment mentioning:

    @gitlab-org/database-team

    And someone will be by shortly!

    Thanks for your help! :heart:

    This message was generated automatically. You're welcome to improve it.

  • Ahmed Hemdan mentioned in commit bc8c22f1

    mentioned in commit bc8c22f1

  • added workflowstaging label and removed workflowcanary label

  • Please register or sign in to reply
    Loading