Skip to content
Snippets Groups Projects

Enforce new Markdown upload URL format

Merged Enforce new Markdown upload URL format
hly-enforce-markdown-upload-urls into master
Merged Heinrich Lee Yu requested to merge hly-enforce-markdown-upload-urls into master

What does this MR do and why?

In 17.1, we started generating ID-based URLs for Markdown uploads. But we maintained support for the old URL format to keep backward compatibility with existing upload URLs.

This MR enforces the use of the new URL format for uploads created starting 17.2.

Related to https://gitlab.com/gitlab-sirt/shared-incidents/incident_5281/-/issues/14 (limited access)

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

  1. Upload a file to an issue comment or description and save.
  2. Check the generated HTML and you should see that the URL generated is in the format: http://localhost:3000/-/project/<project-id>/uploads/<secret>/<filename>
  3. Manually visit: http://localhost:3000/<project-full-path>/uploads/<secret>/<filename>. This should 404.
  4. You can also try an upload that was created before switching to this branch and the legacy URL should work.
Edited by Heinrich Lee Yu

Merge request reports

Merge train pipeline #1346394564 passed

Stage: sync
Merge train pipeline jobs can not be retried
Stage: pre-merge
Merge train pipeline jobs can not be retried

Merge train pipeline passed for 0e49f7b8

Merged by Heinrich Lee YuHeinrich Lee Yu 9 months ago (Jun 25, 2024 2:00am UTC)

Loading

Pipeline #1346396224 passed with warnings

Stage: sync
Stage: preflight
Stage: prepare
Stage: build-images
Stage: fixtures
Stage: lint
Stage: test
Stage: post-test
Stage: benchmark

Pipeline passed with warnings for 454f4701 on master

Test coverage 64.08% from 0 jobs
10 environments impacted.

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Heinrich Lee Yu assigned to @engwan

    assigned to @engwan

  • 🤖 GitLab Bot 🤖 added pipelinetier-1 label

    added pipelinetier-1 label

  • Heinrich Lee Yu added devopsplan groupproject management labels

    added devopsplan groupproject management labels

  • 🤖 GitLab Bot 🤖 added Category:Team Planning sectiondev labels

    added Category:Team Planning sectiondev labels

  • Heinrich Lee Yu added featureenhancement typefeature labels

    added featureenhancement typefeature labels

  • Heinrich Lee Yu changed milestone to %17.2

    changed milestone to %17.2

  • A deleted user added backend database databasereview pending labels

    added backend database databasereview pending labels

  • Ghost User
    Ghost User @ghost1
    Contributor
    1 Warning
    ⚠

    featureaddition and featureenhancement merge requests normally have a documentation change. Consider adding a documentation update or confirming the documentation plan with the Technical Writer counterpart.

    For more information, see:
    1 Message
    📖 CHANGELOG missing:

    If this merge request needs a changelog entry, add the Changelog trailer to the commit message you want to add to the changelog.

    If this merge request doesn't need a CHANGELOG entry, feel free to ignore this message.

    Reviewer roulette

    Category Reviewer Maintainer
    backend @SamWord profile link current availability (UTC-4, 12 hours behind author) @pshutsin profile link current availability (UTC+2, 6 hours behind author)
    database @bauerdominic profile link current availability (UTC+2, 6 hours behind author) @pshutsin profile link current availability (UTC+2, 6 hours behind author)

    Please check reviewer's status!

    • available Reviewer is available!
    • unavailable Reviewer is unavailable!

    Please refer to documentation page for guidance on how you can benefit from the Reviewer Roulette, or use the GitLab Review Workload Dashboard to find other available reviewers.

    If needed, you can retry the 🔁 danger-review job that generated this comment.

    Generated by 🚫 Danger

    Edited by Ghost User
  • Ghost User
    Ghost User @ghost1
    Contributor

    Database migrations (on the main database)

    Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

    Migration Type Total runtime Result DB size change
    20240620135128 - AddVersionToUploads Regular 4.2 s ✅ +0.00 B
    Runtime Histogram for all migrations
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 3
    0.1 seconds - 1 second 1
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Migration: 20240620135128 - AddVersionToUploads

    • Type: Regular
    • Duration: 4.2 s
    • Database size change: +0.00 B
    Calls Total Time Max Time Mean Time Rows Query
    1 4.8 ms 4.8 ms 4.8 ms 0 
    ALTER TABLE "uploads" ADD "version" integer DEFAULT 1 NOT NULL
    2 0.0 ms 0.0 ms 0.0 ms 2 
    SELECT pg_backend_pid()
    1 0.0 ms 0.0 ms 0.0 ms 1 
    SELECT $1::regtype::oid
    Histogram for AddVersionToUploads
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 3
    0.1 seconds - 1 second 1
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Background Migration: BackfillProtectedTagCreateAccessLevelsProjectId

    Sampled 21 batches. Estimated Time to complete: 11 hours
    • Interval: 120s
    • Total tuple count: 330902
    • Max batch size: 0
    • Estimated seconds to complete: 39600s
    • Estimated number of batches: 330
    • Average batch time: 6.1s
    • Batch size: 1000
    • N. of batches sampled: 21
    • N. of failed batches: 0

    Time estimation is conservative and based on sampling production data in a test environment. It represents the max time that migration could take. The actual time may differ from this estimation.

    _Consider changing max_batch_size and interval if this estimate is unacceptable._

    Calls Total Time Max Time Mean Time Rows Query
    21 2664.5 ms 434.1 ms 126.9 ms 2098 
    UPDATE protected_tag_create_access_levels
      SET project_id = protected_tags.project_id
      FROM protected_tags
      WHERE protected_tags.id = protected_tag_create_access_levels.protected_tag_id AND protected_tag_create_access_levels.id IN (
        SELECT protected_tag_create_access_levels.id
        FROM protected_tag_create_access_levels
        WHERE protected_tag_create_access_levels.id BETWEEN $1 AND $2 AND protected_tag_create_access_levels.project_id IS NULL AND protected_tag_create_access_levels.id >= $3
      )
    186 23972.7 ms 320.6 ms 128.9 ms 18600 
    UPDATE protected_tag_create_access_levels
      SET project_id = protected_tags.project_id
      FROM protected_tags
      WHERE protected_tags.id = protected_tag_create_access_levels.protected_tag_id AND protected_tag_create_access_levels.id IN (
        SELECT protected_tag_create_access_levels.id
        FROM protected_tag_create_access_levels
        WHERE protected_tag_create_access_levels.id BETWEEN $1 AND $2 AND protected_tag_create_access_levels.project_id IS NULL AND protected_tag_create_access_levels.id >= $3 AND protected_tag_create_access_levels.id < $4
      )
    207 118.6 ms 4.0 ms 0.6 ms 186 
    SELECT protected_tag_create_access_levels.id
      FROM protected_tag_create_access_levels
      WHERE protected_tag_create_access_levels.id BETWEEN $1 AND $2 AND protected_tag_create_access_levels.project_id IS NULL AND protected_tag_create_access_levels.id >= $3
      ORDER BY protected_tag_create_access_levels.id ASC
      LIMIT $4
      OFFSET $5
    21 1.4 ms 0.4 ms 0.1 ms 21 
    SELECT batched_background_migrations.*
      FROM batched_background_migrations
      WHERE batched_background_migrations.id = $1
      LIMIT $2
    42 7.8 ms 0.3 ms 0.2 ms 42 
    INSERT INTO batched_background_migration_job_transition_logs (batched_background_migration_job_id, created_at, updated_at, previous_status, next_status) VALUES ($1, $2, $3, $4, $5) RETURNING id
    21 2.9 ms 0.3 ms 0.1 ms 21 
    UPDATE batched_background_migration_jobs
      SET updated_at = $1, finished_at = $2, status = $3, metrics = $4
      WHERE batched_background_migration_jobs.id = $5
    21 2.7 ms 0.2 ms 0.1 ms 21 
    UPDATE batched_background_migration_jobs
      SET updated_at = $1, started_at = $2, status = $3, attempts = $4
      WHERE batched_background_migration_jobs.id = $5
    21 0.9 ms 0.1 ms 0.0 ms 21 
    SELECT protected_tag_create_access_levels.id
      FROM protected_tag_create_access_levels
      WHERE protected_tag_create_access_levels.id BETWEEN $1 AND $2 AND protected_tag_create_access_levels.project_id IS NULL
      ORDER BY protected_tag_create_access_levels.id ASC
      LIMIT $3
    42 2.2 ms 0.1 ms 0.1 ms 42 
    SELECT batched_background_migration_jobs.*
      FROM batched_background_migration_jobs
      WHERE batched_background_migration_jobs.id = $1
      LIMIT $2
    21 1.2 ms 0.1 ms 0.1 ms 21 
    SELECT sum(batched_background_migration_jobs.batch_size)
      FROM batched_background_migration_jobs
      WHERE batched_background_migration_jobs.batched_background_migration_id = $1 AND batched_background_migration_jobs.status IN ($2)
    Histogram of batch runtimes for BackfillProtectedTagCreateAccessLevelsProjectId
    Batch Runtime Count
    0 seconds - 10 seconds 21
    10 seconds - 1 minute 0
    1 minute - 2 minutes 0
    2 minutes - 3 minutes 0
    3 minutes - 5 minutes 0
    5 minutes + 0
    Histogram across all sampled batches of BackfillProtectedTagCreateAccessLevelsProjectId
    Query Runtime Count
    0 seconds - 0.1 seconds 74
    0.1 seconds - 0.5 seconds 528
    0.5 seconds - 1 second 1
    1 second - 2 seconds 0
    2 seconds - 5 seconds 0
    5 seconds + 0

    Background Migration: BackfillMergeRequestContextCommitsProjectId

    Sampled 17 batches. Estimated Time to complete: 1 hour and 16 minutes
    • Interval: 120s
    • Total tuple count: 38020
    • Max batch size: 0
    • Estimated seconds to complete: 4560s
    • Estimated number of batches: 38
    • Average batch time: 6.48s
    • Batch size: 1000
    • N. of batches sampled: 17
    • N. of failed batches: 0

    Time estimation is conservative and based on sampling production data in a test environment. It represents the max time that migration could take. The actual time may differ from this estimation.

    _Consider changing max_batch_size and interval if this estimate is unacceptable._

    Calls Total Time Max Time Mean Time Rows Query
    152 26178.6 ms 372.0 ms 172.2 ms 15200 
    UPDATE merge_request_context_commits
      SET project_id = merge_requests.target_project_id
      FROM merge_requests
      WHERE merge_requests.id = merge_request_context_commits.merge_request_id AND merge_request_context_commits.id IN (
        SELECT merge_request_context_commits.id
        FROM merge_request_context_commits
        WHERE merge_request_context_commits.id BETWEEN $1 AND $2 AND merge_request_context_commits.project_id IS NULL AND merge_request_context_commits.id >= $3 AND merge_request_context_commits.id < $4
      )
    17 2181.5 ms 222.2 ms 128.3 ms 1633 
    UPDATE merge_request_context_commits
      SET project_id = merge_requests.target_project_id
      FROM merge_requests
      WHERE merge_requests.id = merge_request_context_commits.merge_request_id AND merge_request_context_commits.id IN (
        SELECT merge_request_context_commits.id
        FROM merge_request_context_commits
        WHERE merge_request_context_commits.id BETWEEN $1 AND $2 AND merge_request_context_commits.project_id IS NULL AND merge_request_context_commits.id >= $3
      )
    169 98.5 ms 3.2 ms 0.6 ms 152 
    SELECT merge_request_context_commits.id
      FROM merge_request_context_commits
      WHERE merge_request_context_commits.id BETWEEN $1 AND $2 AND merge_request_context_commits.project_id IS NULL AND merge_request_context_commits.id >= $3
      ORDER BY merge_request_context_commits.id ASC
      LIMIT $4
      OFFSET $5
    17 5.3 ms 2.7 ms 0.3 ms 17 
    UPDATE batched_background_migration_jobs
      SET updated_at = $1, started_at = $2, status = $3, attempts = $4
      WHERE batched_background_migration_jobs.id = $5
    17 11.2 ms 1.9 ms 0.7 ms 17 
    SELECT merge_request_context_commits.id
      FROM merge_request_context_commits
      WHERE merge_request_context_commits.id BETWEEN $1 AND $2 AND merge_request_context_commits.project_id IS NULL
      ORDER BY merge_request_context_commits.id ASC
      LIMIT $3
    17 3.6 ms 1.6 ms 0.2 ms 17 
    UPDATE batched_background_migration_jobs
      SET updated_at = $1, finished_at = $2, status = $3, metrics = $4
      WHERE batched_background_migration_jobs.id = $5
    34 7.0 ms 1.0 ms 0.2 ms 34 
    INSERT INTO batched_background_migration_job_transition_logs (batched_background_migration_job_id, created_at, updated_at, previous_status, next_status) VALUES ($1, $2, $3, $4, $5) RETURNING id
    34 1.7 ms 0.1 ms 0.0 ms 34 
    SELECT batched_background_migration_jobs.*
      FROM batched_background_migration_jobs
      WHERE batched_background_migration_jobs.id = $1
      LIMIT $2
    17 0.8 ms 0.1 ms 0.0 ms 17 
    SELECT sum(batched_background_migration_jobs.batch_size)
      FROM batched_background_migration_jobs
      WHERE batched_background_migration_jobs.batched_background_migration_id = $1 AND batched_background_migration_jobs.status IN ($2)
    17 0.7 ms 0.0 ms 0.0 ms 17 
    SELECT batched_background_migrations.*
      FROM batched_background_migrations
      WHERE batched_background_migrations.id = $1
      LIMIT $2
    Histogram of batch runtimes for BackfillMergeRequestContextCommitsProjectId
    Batch Runtime Count
    0 seconds - 10 seconds 17
    10 seconds - 1 minute 0
    1 minute - 2 minutes 0
    2 minutes - 3 minutes 0
    3 minutes - 5 minutes 0
    5 minutes + 0
    Histogram across all sampled batches of BackfillMergeRequestContextCommitsProjectId
    Query Runtime Count
    0 seconds - 0.1 seconds 135
    0.1 seconds - 0.5 seconds 355
    0.5 seconds - 1 second 1
    1 second - 2 seconds 0
    2 seconds - 5 seconds 0
    5 seconds + 0

    Background Migration: BackfillIssueLinksNamespaceId

    Sampled 42 batches. Estimated Time to complete: 4 days, 14 hours, and 30 minutes
    • Interval: 120s
    • Total tuple count: 3315845
    • Max batch size: 0
    • Estimated seconds to complete: 397800s
    • Estimated number of batches: 3315
    • Average batch time: 6.97s
    • Batch size: 1000
    • N. of batches sampled: 42
    • N. of failed batches: 0

    Time estimation is conservative and based on sampling production data in a test environment. It represents the max time that migration could take. The actual time may differ from this estimation.

    _Consider changing max_batch_size and interval if this estimate is unacceptable._

    Calls Total Time Max Time Mean Time Rows Query
    378 72934.9 ms 596.8 ms 192.9 ms 37800 
    UPDATE issue_links
      SET namespace_id = issues.namespace_id
      FROM issues
      WHERE issues.id = issue_links.source_id AND issue_links.id IN (
        SELECT issue_links.id
        FROM issue_links
        WHERE issue_links.id BETWEEN $1 AND $2 AND issue_links.namespace_id IS NULL AND issue_links.id >= $3 AND issue_links.id < $4
      )
    42 7118.4 ms 305.1 ms 169.5 ms 4169 
    UPDATE issue_links
      SET namespace_id = issues.namespace_id
      FROM issues
      WHERE issues.id = issue_links.source_id AND issue_links.id IN (
        SELECT issue_links.id
        FROM issue_links
        WHERE issue_links.id BETWEEN $1 AND $2 AND issue_links.namespace_id IS NULL AND issue_links.id >= $3
      )
    420 8908.9 ms 186.3 ms 21.2 ms 378 
    SELECT issue_links.id
      FROM issue_links
      WHERE issue_links.id BETWEEN $1 AND $2 AND issue_links.namespace_id IS NULL AND issue_links.id >= $3
      ORDER BY issue_links.id ASC
      LIMIT $4
      OFFSET $5
    84 19.5 ms 4.2 ms 0.2 ms 84 
    INSERT INTO batched_background_migration_job_transition_logs (batched_background_migration_job_id, created_at, updated_at, previous_status, next_status) VALUES ($1, $2, $3, $4, $5) RETURNING id
    42 11.7 ms 2.1 ms 0.3 ms 42 
    UPDATE batched_background_migration_jobs
      SET updated_at = $1, finished_at = $2, status = $3, metrics = $4
      WHERE batched_background_migration_jobs.id = $5
    42 8.4 ms 1.7 ms 0.2 ms 42 
    UPDATE batched_background_migration_jobs
      SET updated_at = $1, started_at = $2, status = $3, attempts = $4
      WHERE batched_background_migration_jobs.id = $5
    84 4.3 ms 0.1 ms 0.1 ms 84 
    SELECT batched_background_migration_jobs.*
      FROM batched_background_migration_jobs
      WHERE batched_background_migration_jobs.id = $1
      LIMIT $2
    42 2.9 ms 0.1 ms 0.1 ms 42 
    SELECT sum(batched_background_migration_jobs.batch_size)
      FROM batched_background_migration_jobs
      WHERE batched_background_migration_jobs.batched_background_migration_id = $1 AND batched_background_migration_jobs.status IN ($2)
    42 1.6 ms 0.1 ms 0.0 ms 42 
    SELECT issue_links.id
      FROM issue_links
      WHERE issue_links.id BETWEEN $1 AND $2 AND issue_links.namespace_id IS NULL
      ORDER BY issue_links.id ASC
      LIMIT $3
    42 1.8 ms 0.1 ms 0.0 ms 42 
    SELECT batched_background_migrations.*
      FROM batched_background_migrations
      WHERE batched_background_migrations.id = $1
      LIMIT $2
    Histogram of batch runtimes for BackfillIssueLinksNamespaceId
    Batch Runtime Count
    0 seconds - 10 seconds 42
    10 seconds - 1 minute 0
    1 minute - 2 minutes 0
    2 minutes - 3 minutes 0
    3 minutes - 5 minutes 0
    5 minutes + 0
    Histogram across all sampled batches of BackfillIssueLinksNamespaceId
    Query Runtime Count
    0 seconds - 0.1 seconds 256
    0.1 seconds - 0.5 seconds 953
    0.5 seconds - 1 second 9
    1 second - 2 seconds 0
    2 seconds - 5 seconds 0
    5 seconds + 0

    Background Migration: BackfillVulnerabilityFindingsRemediationsProjectId

    Sampled 40 batches. Estimated Time to complete: 2 days, 14 hours, and 52 minutes
    • Interval: 120s
    • Total tuple count: 1886460
    • Max batch size: 0
    • Estimated seconds to complete: 226320s
    • Estimated number of batches: 1886
    • Average batch time: 5.34s
    • Batch size: 1000
    • N. of batches sampled: 40
    • N. of failed batches: 0

    Time estimation is conservative and based on sampling production data in a test environment. It represents the max time that migration could take. The actual time may differ from this estimation.

    _Consider changing max_batch_size and interval if this estimate is unacceptable._

    Calls Total Time Max Time Mean Time Rows Query
    353 18212.9 ms 320.2 ms 51.6 ms 35300 
    UPDATE vulnerability_findings_remediations
      SET project_id = vulnerability_occurrences.project_id
      FROM vulnerability_occurrences
      WHERE vulnerability_occurrences.id = vulnerability_findings_remediations.vulnerability_occurrence_id AND vulnerability_findings_remediations.id IN (
        SELECT vulnerability_findings_remediations.id
        FROM vulnerability_findings_remediations
        WHERE vulnerability_findings_remediations.id BETWEEN $1 AND $2 AND vulnerability_findings_remediations.project_id IS NULL AND vulnerability_findings_remediations.id >= $3 AND vulnerability_findings_remediations.id < $4
      )
    40 1707.4 ms 141.4 ms 42.7 ms 3972 
    UPDATE vulnerability_findings_remediations
      SET project_id = vulnerability_occurrences.project_id
      FROM vulnerability_occurrences
      WHERE vulnerability_occurrences.id = vulnerability_findings_remediations.vulnerability_occurrence_id AND vulnerability_findings_remediations.id IN (
        SELECT vulnerability_findings_remediations.id
        FROM vulnerability_findings_remediations
        WHERE vulnerability_findings_remediations.id BETWEEN $1 AND $2 AND vulnerability_findings_remediations.project_id IS NULL AND vulnerability_findings_remediations.id >= $3
      )
    40 12.9 ms 11.3 ms 0.3 ms 40 
    SELECT vulnerability_findings_remediations.id
      FROM vulnerability_findings_remediations
      WHERE vulnerability_findings_remediations.id BETWEEN $1 AND $2 AND vulnerability_findings_remediations.project_id IS NULL
      ORDER BY vulnerability_findings_remediations.id ASC
      LIMIT $3
    393 75.5 ms 6.0 ms 0.2 ms 353 
    SELECT vulnerability_findings_remediations.id
      FROM vulnerability_findings_remediations
      WHERE vulnerability_findings_remediations.id BETWEEN $1 AND $2 AND vulnerability_findings_remediations.project_id IS NULL AND vulnerability_findings_remediations.id >= $3
      ORDER BY vulnerability_findings_remediations.id ASC
      LIMIT $4
      OFFSET $5
    40 13.5 ms 3.4 ms 0.3 ms 40 
    UPDATE batched_background_migration_jobs
      SET updated_at = $1, finished_at = $2, status = $3, metrics = $4
      WHERE batched_background_migration_jobs.id = $5
    80 15.2 ms 1.0 ms 0.2 ms 80 
    INSERT INTO batched_background_migration_job_transition_logs (batched_background_migration_job_id, created_at, updated_at, previous_status, next_status) VALUES ($1, $2, $3, $4, $5) RETURNING id
    40 6.2 ms 0.8 ms 0.2 ms 40 
    UPDATE batched_background_migration_jobs
      SET updated_at = $1, started_at = $2, status = $3, attempts = $4
      WHERE batched_background_migration_jobs.id = $5
    40 2.6 ms 0.1 ms 0.1 ms 40 
    SELECT sum(batched_background_migration_jobs.batch_size)
      FROM batched_background_migration_jobs
      WHERE batched_background_migration_jobs.batched_background_migration_id = $1 AND batched_background_migration_jobs.status IN ($2)
    80 4.1 ms 0.1 ms 0.1 ms 80 
    SELECT batched_background_migration_jobs.*
      FROM batched_background_migration_jobs
      WHERE batched_background_migration_jobs.id = $1
      LIMIT $2
    40 1.8 ms 0.1 ms 0.0 ms 40 
    SELECT batched_background_migrations.*
      FROM batched_background_migrations
      WHERE batched_background_migrations.id = $1
      LIMIT $2
    Histogram of batch runtimes for BackfillVulnerabilityFindingsRemediationsProjectId
    Batch Runtime Count
    0 seconds - 10 seconds 40
    10 seconds - 1 minute 0
    1 minute - 2 minutes 0
    2 minutes - 3 minutes 0
    3 minutes - 5 minutes 0
    5 minutes + 0
    Histogram across all sampled batches of BackfillVulnerabilityFindingsRemediationsProjectId
    Query Runtime Count
    0 seconds - 0.1 seconds 269
    0.1 seconds - 0.5 seconds 877
    0.5 seconds - 1 second 0
    1 second - 2 seconds 0
    2 seconds - 5 seconds 0
    5 seconds + 0

    Background Migration: BackfillRelatedEpicLinksGroupId

    Sampled 6 batches. Estimated Time to complete: 14 minutes
    • Interval: 120s
    • Total tuple count: 7860
    • Max batch size: 0
    • Estimated seconds to complete: 840s
    • Estimated number of batches: 7
    • Average batch time: 4.82s
    • Batch size: 1000
    • N. of batches sampled: 6
    • N. of failed batches: 0

    Time estimation is conservative and based on sampling production data in a test environment. It represents the max time that migration could take. The actual time may differ from this estimation.

    _Consider changing max_batch_size and interval if this estimate is unacceptable._

    Calls Total Time Max Time Mean Time Rows Query
    37 4580.2 ms 351.5 ms 123.8 ms 3700 
    UPDATE related_epic_links
      SET group_id = epics.group_id
      FROM epics
      WHERE epics.id = related_epic_links.source_id AND related_epic_links.id IN (
        SELECT related_epic_links.id
        FROM related_epic_links
        WHERE related_epic_links.id BETWEEN $1 AND $2 AND related_epic_links.group_id IS NULL AND related_epic_links.id >= $3 AND related_epic_links.id < $4
      )
    6 430.9 ms 148.6 ms 71.8 ms 460 
    UPDATE related_epic_links
      SET group_id = epics.group_id
      FROM epics
      WHERE epics.id = related_epic_links.source_id AND related_epic_links.id IN (
        SELECT related_epic_links.id
        FROM related_epic_links
        WHERE related_epic_links.id BETWEEN $1 AND $2 AND related_epic_links.group_id IS NULL AND related_epic_links.id >= $3
      )
    12 11.3 ms 8.4 ms 0.9 ms 12 
    INSERT INTO batched_background_migration_job_transition_logs (batched_background_migration_job_id, created_at, updated_at, previous_status, next_status) VALUES ($1, $2, $3, $4, $5) RETURNING id
    6 3.8 ms 3.3 ms 0.6 ms 6 
    UPDATE batched_background_migration_jobs
      SET updated_at = $1, finished_at = $2, status = $3, metrics = $4
      WHERE batched_background_migration_jobs.id = $5
    6 5.6 ms 2.3 ms 0.9 ms 6 
    UPDATE batched_background_migration_jobs
      SET updated_at = $1, started_at = $2, status = $3, attempts = $4
      WHERE batched_background_migration_jobs.id = $5
    43 22.1 ms 2.2 ms 0.5 ms 37 
    SELECT related_epic_links.id
      FROM related_epic_links
      WHERE related_epic_links.id BETWEEN $1 AND $2 AND related_epic_links.group_id IS NULL AND related_epic_links.id >= $3
      ORDER BY related_epic_links.id ASC
      LIMIT $4
      OFFSET $5
    6 0.9 ms 0.4 ms 0.1 ms 6 
    SELECT related_epic_links.id
      FROM related_epic_links
      WHERE related_epic_links.id BETWEEN $1 AND $2 AND related_epic_links.group_id IS NULL
      ORDER BY related_epic_links.id ASC
      LIMIT $3
    6 0.3 ms 0.1 ms 0.0 ms 6 
    SELECT sum(batched_background_migration_jobs.batch_size)
      FROM batched_background_migration_jobs
      WHERE batched_background_migration_jobs.batched_background_migration_id = $1 AND batched_background_migration_jobs.status IN ($2)
    12 0.6 ms 0.1 ms 0.0 ms 12 
    SELECT batched_background_migration_jobs.*
      FROM batched_background_migration_jobs
      WHERE batched_background_migration_jobs.id = $1
      LIMIT $2
    6 0.3 ms 0.1 ms 0.0 ms 6 
    SELECT batched_background_migrations.*
      FROM batched_background_migrations
      WHERE batched_background_migrations.id = $1
      LIMIT $2
    Histogram of batch runtimes for BackfillRelatedEpicLinksGroupId
    Batch Runtime Count
    0 seconds - 10 seconds 6
    10 seconds - 1 minute 0
    1 minute - 2 minutes 0
    2 minutes - 3 minutes 0
    3 minutes - 5 minutes 0
    5 minutes + 0
    Histogram across all sampled batches of BackfillRelatedEpicLinksGroupId
    Query Runtime Count
    0 seconds - 0.1 seconds 38
    0.1 seconds - 0.5 seconds 102
    0.5 seconds - 1 second 0
    1 second - 2 seconds 0
    2 seconds - 5 seconds 0
    5 seconds + 0

    Other information

    Other migrations pending on GitLab.com
    Migration Type Total runtime Result DB size change
    20240527072607 - AddOrganizationGroupsProjectsSort Regular 14.7 s ⚠ +0.00 B
    20240527073021 - AddOrganizationGroupsProjectsDisplay Regular 4.5 s ✅ +0.00 B
    20240607035355 - AddMemberRoleIdToLdapGroupLinks Regular 5.0 s ✅ +0.00 B
    20240607041020 - AddFkToMemberRoleOnLdapGroupLinks Regular 5.3 s ✅ +8.00 KiB [note]
    20240611091436 - AddIndexToLdapGroupLinksOnMemberRoleId Regular 5.0 s ✅ +16.00 KiB
    20240604111157 - AddApprovalPolicyRulesFkOnApprovalGroupRules Post deploy 5.9 s ✅ +0.00 B
    20240604111324 - AddApprovalPolicyRulesFkOnApprovalProjectRules Post deploy 6.5 s ✅ +8.00 KiB [note]
    20240604111411 - AddApprovalPolicyRulesFkOnApprovalMergeRequestRules Post deploy 6.8 s ✅ +0.00 B
    20240604111512 - AddApprovalPolicyRulesFkOnSoftwareLicensePolicies Post deploy 6.0 s ✅ +0.00 B
    20240604111555 - AddApprovalPolicyRulesFkOnScanResultPolicyViolations Post deploy 6.0 s ✅ +0.00 B
    20240605080835 - RemovePartitionIdDefaultValueForCiPipelineMessage Post deploy 4.3 s ✅ +0.00 B
    20240610125054 - CreateIdxVulnerabilityOccurencesOnPrimIdenId Post deploy 5.5 s ✅ +0.00 B
    20240610134828 - DropIndexCiStageOnPipelineId Post deploy 5.3 s ✅ -16.00 KiB
    20240611103909 - RemoveTmpIndexProjectStatisticsUpdatedAtSync Post deploy 5.2 s ✅ +0.00 B
    20240613065417 - IndexRelatedEpicLinksOnGroupId Post deploy 5.0 s ✅ +72.00 KiB
    20240613065418 - AddRelatedEpicLinksGroupIdFk Post deploy 5.0 s ✅ +0.00 B
    20240613065419 - AddRelatedEpicLinksGroupIdTrigger Post deploy 4.4 s ✅ +8.00 KiB [note]
    20240613065420 - QueueBackfillRelatedEpicLinksGroupId Post deploy 5.3 s ✅ +0.00 B
    20240613072539 - IndexMergeRequestContextCommitsOnProjectId Post deploy 5.1 s ✅ +272.00 KiB
    20240613072540 - AddMergeRequestContextCommitsProjectIdFk Post deploy 5.2 s ⚠ +0.00 B
    20240613072541 - AddMergeRequestContextCommitsProjectIdTrigger Post deploy 4.4 s ✅ +0.00 B
    20240613072542 - QueueBackfillMergeRequestContextCommitsProjectId Post deploy 4.8 s ✅ +0.00 B
    20240613073928 - IndexIssueLinksOnNamespaceId Post deploy 11.5 s ✅ +21.94 MiB
    20240613073929 - AddIssueLinksNamespaceIdFk Post deploy 5.1 s ✅ +0.00 B
    20240613073930 - AddIssueLinksNamespaceIdTrigger Post deploy 4.4 s ✅ +0.00 B
    20240613073931 - QueueBackfillIssueLinksNamespaceId Post deploy 4.8 s ✅ +0.00 B
    20240613214355 - AddIndexForOrganizationIdOnSnippets Post deploy 18.6 s ✅ +3.69 MiB
    20240617003021 - CleanupBigintConversionsForPCiBuilds Post deploy 6.2 s ✅ +0.00 B
    20240617114649 - RemoveSubscriptionHistoryWithNullNamespaceId Post deploy 39.9 s ⚠ +0.00 B
    20240617210449 - ChangeCatalogResourcesLast30DayUsageCountUpdatedAtDefault Post deploy 4.3 s ✅ +0.00 B
    20240618121518 - IndexVulnerabilityFindingsRemediationsOnProjectId Post deploy 10.4 s ✅ +12.48 MiB
    20240618121519 - AddVulnerabilityFindingsRemediationsProjectIdFk Post deploy 5.1 s ✅ +0.00 B
    20240618121520 - AddVulnerabilityFindingsRemediationsProjectIdTrigger Post deploy 4.3 s ✅ +0.00 B
    20240618121521 - QueueBackfillVulnerabilityFindingsRemediationsProjectId Post deploy 4.8 s ✅ +0.00 B
    20240618122508 - IndexProtectedTagCreateAccessLevelsOnProjectId Post deploy 5.4 s ✅ +2.21 MiB
    20240618122509 - AddProtectedTagCreateAccessLevelsProjectIdFk Post deploy 5.0 s ✅ +0.00 B
    20240618122510 - AddProtectedTagCreateAccessLevelsProjectIdTrigger Post deploy 4.3 s ✅ +0.00 B
    20240618122511 - QueueBackfillProtectedTagCreateAccessLevelsProjectId Post deploy 4.8 s ✅ +0.00 B
    Clone details
    Clone ID Clone Created At Clone Data Timestamp Expected Removal Time
    database-testing-3418581-14377045-main 2024-06-20T15:48:19Z 2024-06-20T12:09:13Z 2024-06-21 04:23:40 +0000
    database-testing-3418581-14377045-ci 2024-06-20T15:48:19Z 2024-06-20T12:45:31Z 2024-06-21 04:23:40 +0000

    Job artifacts

    Database migrations (on the ci database)

    Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

    Migration Type Total runtime Result DB size change
    20240620135128 - AddVersionToUploads Regular 6.3 s ✅ +0.00 B
    Runtime Histogram for all migrations
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 3
    0.1 seconds - 1 second 1
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Migration: 20240620135128 - AddVersionToUploads

    • Type: Regular
    • Duration: 6.3 s
    • Database size change: +0.00 B
    Calls Total Time Max Time Mean Time Rows Query
    1 1.6 ms 1.6 ms 1.6 ms 0 
    ALTER TABLE "uploads" ADD "version" integer DEFAULT 1 NOT NULL
    1 0.0 ms 0.0 ms 0.0 ms 1 
    SELECT $1::regtype::oid
    2 0.0 ms 0.0 ms 0.0 ms 2 
    SELECT pg_backend_pid()
    Histogram for AddVersionToUploads
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 3
    0.1 seconds - 1 second 1
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Other information

    Other migrations pending on GitLab.com
    Migration Type Total runtime Result DB size change
    20240527072607 - AddOrganizationGroupsProjectsSort Regular 7.6 s ✅ +0.00 B
    20240527073021 - AddOrganizationGroupsProjectsDisplay Regular 6.3 s ✅ +0.00 B
    20240607035355 - AddMemberRoleIdToLdapGroupLinks Regular 6.8 s ✅ +0.00 B
    20240607041020 - AddFkToMemberRoleOnLdapGroupLinks Regular 7.4 s ✅ +0.00 B
    20240611091436 - AddIndexToLdapGroupLinksOnMemberRoleId Regular 7.0 s ✅ +8.00 KiB [note]
    20240604111157 - AddApprovalPolicyRulesFkOnApprovalGroupRules Post deploy 7.9 s ✅ +0.00 B
    20240604111324 - AddApprovalPolicyRulesFkOnApprovalProjectRules Post deploy 8.5 s ✅ +8.00 KiB [note]
    20240604111411 - AddApprovalPolicyRulesFkOnApprovalMergeRequestRules Post deploy 8.8 s ✅ +0.00 B
    20240604111512 - AddApprovalPolicyRulesFkOnSoftwareLicensePolicies Post deploy 8.0 s ✅ +0.00 B
    20240604111555 - AddApprovalPolicyRulesFkOnScanResultPolicyViolations Post deploy 7.9 s ✅ +0.00 B
    20240605080835 - RemovePartitionIdDefaultValueForCiPipelineMessage Post deploy 6.3 s ✅ +0.00 B
    20240610125054 - CreateIdxVulnerabilityOccurencesOnPrimIdenId Post deploy 7.5 s ✅ +0.00 B
    20240610134828 - DropIndexCiStageOnPipelineId Post deploy 7.7 s ✅ -56.24 GiB
    20240611103909 - RemoveTmpIndexProjectStatisticsUpdatedAtSync Post deploy 7.2 s ✅ +0.00 B
    20240613065417 - IndexRelatedEpicLinksOnGroupId Post deploy 7.1 s ✅ +8.00 KiB [note]
    20240613065418 - AddRelatedEpicLinksGroupIdFk Post deploy 7.0 s ✅ +0.00 B
    20240613065419 - AddRelatedEpicLinksGroupIdTrigger Post deploy 6.4 s ✅ +8.00 KiB [note]
    20240613065420 - QueueBackfillRelatedEpicLinksGroupId Post deploy 6.1 s ✅ +0.00 B
    20240613072539 - IndexMergeRequestContextCommitsOnProjectId Post deploy 6.8 s ✅ +8.00 KiB [note]
    20240613072540 - AddMergeRequestContextCommitsProjectIdFk Post deploy 7.0 s ✅ +8.00 KiB [note]
    20240613072541 - AddMergeRequestContextCommitsProjectIdTrigger Post deploy 6.4 s ✅ +0.00 B
    20240613072542 - QueueBackfillMergeRequestContextCommitsProjectId Post deploy 6.1 s ✅ +0.00 B
    20240613073928 - IndexIssueLinksOnNamespaceId Post deploy 7.0 s ✅ +8.00 KiB [note]
    20240613073929 - AddIssueLinksNamespaceIdFk Post deploy 7.2 s ✅ +0.00 B
    20240613073930 - AddIssueLinksNamespaceIdTrigger Post deploy 6.4 s ✅ +8.00 KiB [note]
    20240613073931 - QueueBackfillIssueLinksNamespaceId Post deploy 6.2 s ✅ +0.00 B
    20240613214355 - AddIndexForOrganizationIdOnSnippets Post deploy 8.0 s ✅ +8.00 KiB [note]
    20240617003021 - CleanupBigintConversionsForPCiBuilds Post deploy 8.0 s ✅ +0.00 B
    20240617114649 - RemoveSubscriptionHistoryWithNullNamespaceId Post deploy 5.9 s ✅ +0.00 B
    20240617210449 - ChangeCatalogResourcesLast30DayUsageCountUpdatedAtDefault Post deploy 6.3 s ✅ +0.00 B
    20240618121518 - IndexVulnerabilityFindingsRemediationsOnProjectId Post deploy 6.9 s ✅ +8.00 KiB [note]
    20240618121519 - AddVulnerabilityFindingsRemediationsProjectIdFk Post deploy 7.0 s ✅ +0.00 B
    20240618121520 - AddVulnerabilityFindingsRemediationsProjectIdTrigger Post deploy 6.4 s ✅ +0.00 B
    20240618121521 - QueueBackfillVulnerabilityFindingsRemediationsProjectId Post deploy 6.3 s ✅ +0.00 B
    20240618122508 - IndexProtectedTagCreateAccessLevelsOnProjectId Post deploy 7.1 s ✅ +8.00 KiB [note]
    20240618122509 - AddProtectedTagCreateAccessLevelsProjectIdFk Post deploy 7.0 s ✅ +0.00 B
    20240618122510 - AddProtectedTagCreateAccessLevelsProjectIdTrigger Post deploy 6.4 s ✅ +0.00 B
    20240618122511 - QueueBackfillProtectedTagCreateAccessLevelsProjectId Post deploy 6.1 s ✅ +0.00 B
    Clone details
    Clone ID Clone Created At Clone Data Timestamp Expected Removal Time
    database-testing-3418581-14377045-main 2024-06-20T15:48:19Z 2024-06-20T12:09:13Z 2024-06-21 04:23:40 +0000
    database-testing-3418581-14377045-ci 2024-06-20T15:48:19Z 2024-06-20T12:45:31Z 2024-06-21 04:23:40 +0000

    Job artifacts

    Brought to you by gitlab-org/database-team/gitlab-com-database-testing. Epic

    Edited by ****
  • Heinrich Lee Yu changed the description

    changed the description

  • **** added database-testing-automation label

    added database-testing-automation label

  • Heinrich Lee Yu added 1 commit

    added 1 commit

    • d2ba5e80 - Enforce new Markdown upload URL format

    Compare with previous version

  • Heinrich Lee Yu
  • Heinrich Lee Yu changed the description

    changed the description

  • Heinrich Lee Yu requested review from @hkhanna2 and @bauerdominic

    requested review from @hkhanna2 and @bauerdominic

  • Dominic Bauer approved this merge request

    approved this merge request

  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
    • Please register or sign in to reply