Skip to content
Snippets Groups Projects

Add a constraint to ensure application_settings.rate_limits is a hash

Merged Abdul Wadood requested to merge 420321-add-rate-limits-hash-constraint into master
All threads resolved!

What does this MR do and why?

Add a constraint to ensure application_settings.rate_limits is a hash

In f89cc164, we introduced a JSONB column rate_limits to store the application rate limits. We want to ensure that we always store a hash in this column otherwise, the rails application will break. This can happen if a bad update query is executed like in the following case if we redefine the application_settings class:

class MyMigration

  class TmpSetting < ApplicationRecord
    self.table_name = :application_settings
  end

  def up
    TmpSetting.update(rate_limits: [1])
  end
end

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Migration output

up
bin/rails db:migrate
main: == [advisory_lock_connection] object_id: 183100, pg_backend_pid: 45389
main: == 20240115115029 AddRateLimitsHashConstraintToApplicationSettings: migrating =
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- execute("ALTER TABLE application_settings\nADD CONSTRAINT check_application_settings_rate_limits_is_hash\nCHECK ( (jsonb_typeof(rate_limits) = 'object') )\nNOT VALID;\n")
main:    -> 0.0036s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0002s
main: -- execute("ALTER TABLE application_settings VALIDATE CONSTRAINT check_application_settings_rate_limits_is_hash;")
main:    -> 0.0006s
main: -- execute("RESET statement_timeout")
main:    -> 0.0002s
main: == 20240115115029 AddRateLimitsHashConstraintToApplicationSettings: migrated (0.0264s)

main: == [advisory_lock_connection] object_id: 183100, pg_backend_pid: 45389
ci: == [advisory_lock_connection] object_id: 183340, pg_backend_pid: 45391
ci: == 20240115115029 AddRateLimitsHashConstraintToApplicationSettings: migrating =
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- execute("ALTER TABLE application_settings\nADD CONSTRAINT check_application_settings_rate_limits_is_hash\nCHECK ( (jsonb_typeof(rate_limits) = 'object') )\nNOT VALID;\n")
ci:    -> 0.0036s
ci: -- execute("SET statement_timeout TO 0")
ci:    -> 0.0002s
ci: -- execute("ALTER TABLE application_settings VALIDATE CONSTRAINT check_application_settings_rate_limits_is_hash;")
ci:    -> 0.0011s
ci: -- execute("RESET statement_timeout")
ci:    -> 0.0002s
ci: == 20240115115029 AddRateLimitsHashConstraintToApplicationSettings: migrated (0.0213s)

ci: == [advisory_lock_connection] object_id: 183340, pg_backend_pid: 45391
down
VERSION=20240115115029 bin/rails db:rollback:main && bin/rails db:rollback:ci
main: == [advisory_lock_connection] object_id: 182700, pg_backend_pid: 52189
main: == 20240115115029 AddRateLimitsHashConstraintToApplicationSettings: reverting =
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- execute("            ALTER TABLE application_settings\n            DROP CONSTRAINT IF EXISTS check_application_settings_rate_limits_is_hash\n")
main:    -> 0.0017s
main: == 20240115115029 AddRateLimitsHashConstraintToApplicationSettings: reverted (0.0140s)

main: == [advisory_lock_connection] object_id: 182700, pg_backend_pid: 52189
ci: == [advisory_lock_connection] object_id: 182640, pg_backend_pid: 52597
ci: == 20240115115029 AddRateLimitsHashConstraintToApplicationSettings: reverting =
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- execute("            ALTER TABLE application_settings\n            DROP CONSTRAINT IF EXISTS check_application_settings_rate_limits_is_hash\n")
ci:    -> 0.0015s
ci: == 20240115115029 AddRateLimitsHashConstraintToApplicationSettings: reverted (0.0237s)

ci: == [advisory_lock_connection] object_id: 182640, pg_backend_pid: 52597

How to set up and validate locally

Update the rate_limits column using the below queries on the master branch and all of them will succeed. On this branch, only {} will be allowed.

update application_settings set rate_limits = '"a"';
update application_settings set rate_limits = '1';
update application_settings set rate_limits = '[]';
update application_settings set rate_limits = '{}';

select jsonb_typeof(rate_limits) from application_settings;

Related to #420321 (closed)

Edited by Abdul Wadood

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Abdul Wadood changed milestone to %16.8

    changed milestone to %16.8

  • assigned to @abdwdd

  • Contributor
    2 Warnings
    :warning:

    featureaddition and featureenhancement merge requests normally have a documentation change. Consider adding a documentation update or confirming the documentation plan with the Technical Writer counterpart.

    For more information, see:

    :warning: You've made some app changes, but didn't add any tests.
    That's OK as long as you're refactoring existing code,
    but please consider adding any of the maintenancepipelines, maintenancerefactor, maintenanceworkflow, documentation, QA labels.

    Reviewer roulette

    Changes that require review have been detected!

    Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:

    Category Reviewer Maintainer
    database @ck3g profile link current availability (UTC+1, 4.5 hours behind author) @dfrazao-gitlab profile link current availability (UTC+1, 4.5 hours behind author)

    Please check reviewer's status!

    • available Reviewer is available!
    • unavailable Reviewer is unavailable!

    Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.

    To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.

    Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.

    If needed, you can retry the :repeat: danger-review job that generated this comment.

    Generated by :no_entry_sign: Danger

  • Abdul Wadood changed the description

    changed the description

  • Abdul Wadood changed milestone to %16.9

    changed milestone to %16.9

  • Abdul Wadood requested review from @ahegyi

    requested review from @ahegyi

  • Contributor

    E2E Test Result Summary

    allure-report-publisher generated test report!

    e2e-test-on-gdk: :white_check_mark: test report for a02d2e55

    expand test summary
    +------------------------------------------------------------------+
    |                          suites summary                          |
    +-------------+--------+--------+---------+-------+-------+--------+
    |             | passed | failed | skipped | flaky | total | result |
    +-------------+--------+--------+---------+-------+-------+--------+
    | Verify      | 31     | 0      | 0       | 0     | 31    | ✅     |
    | Create      | 51     | 0      | 11      | 0     | 62    | ✅     |
    | Plan        | 53     | 0      | 1       | 0     | 54    | ✅     |
    | Govern      | 67     | 0      | 0       | 0     | 67    | ✅     |
    | Data Stores | 23     | 0      | 0       | 0     | 23    | ✅     |
    | Package     | 15     | 0      | 1       | 0     | 16    | ✅     |
    | Monitor     | 7      | 0      | 0       | 0     | 7     | ✅     |
    | Manage      | 0      | 0      | 1       | 0     | 1     | ➖     |
    | Release     | 5      | 0      | 0       | 0     | 5     | ✅     |
    | Analytics   | 2      | 0      | 0       | 0     | 2     | ✅     |
    +-------------+--------+--------+---------+-------+-------+--------+
    | Total       | 254    | 0      | 14      | 0     | 268   | ✅     |
    +-------------+--------+--------+---------+-------+-------+--------+
  • Adam Hegyi approved this merge request

    approved this merge request

  • added databaseapproved label and removed databasereview pending label

  • Contributor

    Database migrations (on the main database)

    Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

    Migration Type Total runtime Result DB size change
    20240115115029 - AddRateLimitsHashConstraintToApplicationSettings Regular 2.0 s :white_check_mark: +0.00 B
    Runtime Histogram for all migrations
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 4
    0.1 seconds - 1 second 0
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Migration: 20240115115029 - AddRateLimitsHashConstraintToApplicationSettings

    • Type: Regular
    • Duration: 2.0 s
    • Database size change: +0.00 B
    Calls Total Time Max Time Mean Time Rows Query
    1 22.8 ms 22.8 ms 22.8 ms 0
    ALTER TABLE application_settings VALIDATE CONSTRAINT check_application_settings_rate_limits_is_hash
    1 12.3 ms 12.3 ms 12.3 ms 0
    ALTER TABLE application_settings ADD CONSTRAINT check_application_settings_rate_limits_is_hash CHECK ( (jsonb_typeof(rate_limits) = 'object') ) NOT VALID
    2 0.0 ms 0.0 ms 0.0 ms 2
    SELECT pg_backend_pid()
    Histogram for AddRateLimitsHashConstraintToApplicationSettings
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 4
    0.1 seconds - 1 second 0
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Other information

    Other migrations pending on GitLab.com
    Migration Type Total runtime Result DB size change
    20240108123023 - CreateAiAgentsTable Regular 2.2 s :warning: +40.00 KiB
    20240108123115 - CreateAiAgentVersionsTable Regular 1.6 s :white_check_mark: +40.00 KiB
    20240108123509 - AddFkOnAiAgentVersionsToAiAgents Regular 2.0 s :white_check_mark: +0.00 B
    20240110085226 - AddRateLimitsToApplicationSettings Regular 1.6 s :white_check_mark: +0.00 B
    20231222072237 - AddTextLimitToCiInstanceVariablesDescription Post deploy 1.8 s :white_check_mark: +0.00 B
    20240104155616 - FinalizeBackfillDefaultBranchProtectionNamespaceSettings Post deploy 1.6 s :white_check_mark: +0.00 B
    20240107154747 - SentNotificationsSelfInstallFinalizeBbm Post deploy 1.5 s :white_check_mark: +0.00 B
    20240107154805 - SentNotificationsSelfInstallIdSwap Post deploy 1.4 s :white_check_mark: +0.00 B
    20240108082419 - QueueBackfillPartitionIdCiPipelineMetadata Post deploy 1.5 s :white_check_mark: +0.00 B
    20240108125135 - RemoveInternalIdsTriggers Post deploy 1.6 s :white_check_mark: +0.00 B
    20240108125335 - AddTemporaryIndexInternalIdsOnIdAndUsage Post deploy 52.1 s :white_check_mark: +656.00 KiB
    20240108185335 - BackfillInternalIdsWithIssuesUsageForEpics Post deploy 149.4 s :warning: +2.52 MiB
    20240108215335 - RemoveInternalIdsTmpIndex Post deploy 1.9 s :white_check_mark: -656.00 KiB
    20240109090354 - QueueBackfillPartitionIdCiPipelineArtifact Post deploy 1.5 s :white_check_mark: +0.00 B
    20240110090352 - QueueBackfillPartitionIdCiPipelineConfig Post deploy 1.5 s :white_check_mark: +0.00 B
    20240110093654 - ScheduleIndexRemovalCiBuildTraceMetadata Post deploy 1.8 s :white_check_mark: +8.00 KiB [note]
    20240110094002 - DropIndexFromCiJobArtifactState Post deploy 2.0 s :white_check_mark: -8.00 KiB
    20240110094510 - RemoveFkFromCiJobArtifactState Post deploy 2.3 s :warning: +0.00 B
    20240110132029 - ReRemoveRequirementsIgnoredColumns Post deploy 2.4 s :white_check_mark: -3.47 MiB
    Clone details
    Clone ID Clone Created At Clone Data Timestamp Expected Removal Time
    database-testing-2747447-12500151-main 2024-01-15T13:18:25Z 2024-01-15T10:53:13Z 2024-01-16 01:58:17 +0000
    database-testing-2747447-12500151-ci 2024-01-15T13:18:25Z 2024-01-15T12:46:10Z 2024-01-16 01:58:17 +0000

    Job artifacts

    Database migrations (on the ci database)

    Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

    Migration Type Total runtime Result DB size change
    20240115115029 - AddRateLimitsHashConstraintToApplicationSettings Regular 2.9 s :white_check_mark: +0.00 B
    Runtime Histogram for all migrations
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 4
    0.1 seconds - 1 second 0
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Migration: 20240115115029 - AddRateLimitsHashConstraintToApplicationSettings

    • Type: Regular
    • Duration: 2.9 s
    • Database size change: +0.00 B
    Calls Total Time Max Time Mean Time Rows Query
    1 10.2 ms 10.2 ms 10.2 ms 0
    ALTER TABLE application_settings ADD CONSTRAINT check_application_settings_rate_limits_is_hash CHECK ( (jsonb_typeof(rate_limits) = 'object') ) NOT VALID
    1 1.6 ms 1.6 ms 1.6 ms 0
    ALTER TABLE application_settings VALIDATE CONSTRAINT check_application_settings_rate_limits_is_hash
    2 0.0 ms 0.0 ms 0.0 ms 2
    SELECT pg_backend_pid()
    Histogram for AddRateLimitsHashConstraintToApplicationSettings
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 4
    0.1 seconds - 1 second 0
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Background Migration: BackfillPartitionIdCiPipelineArtifact

    Sampled 83 batches. Estimated Time to complete: 2 days, 10 hours, and 32 minutes
    • Interval: 120s
    • Total tuple count: 1756404
    • Max batch size: 0
    • Estimated seconds to complete: 210720s
    • Estimated number of batches: 1756
    • Average batch time: 6.46s
    • Batch size: 1000
    • N. of batches sampled: 83
    • N. of failed batches: 0

    Time estimation is conservative and based on sampling production data in a test environment. It represents the max time that migration could take. The actual time may differ from this estimation.

    _Consider changing max_batch_size and interval if this estimate is unacceptable._

    Calls Total Time Max Time Mean Time Rows Query
    747 286658.7 ms 2330.1 ms 383.7 ms 74700
    UPDATE ci_pipeline_artifacts
    SET partition_id = ci_pipelines.partition_id
    FROM ci_pipelines
    WHERE ci_pipeline_artifacts.id BETWEEN $1 AND $2 AND ci_pipeline_artifacts.id >= $3 AND ci_pipeline_artifacts.id < $4 AND ci_pipeline_artifacts.pipeline_id = ci_pipelines.id
    83 28216.7 ms 988.5 ms 340.0 ms 8220
    UPDATE ci_pipeline_artifacts
    SET partition_id = ci_pipelines.partition_id
    FROM ci_pipelines
    WHERE ci_pipeline_artifacts.id BETWEEN $1 AND $2 AND ci_pipeline_artifacts.id >= $3 AND ci_pipeline_artifacts.pipeline_id = ci_pipelines.id
    830 210.5 ms 3.6 ms 0.3 ms 747
    SELECT ci_pipeline_artifacts.id
    FROM ci_pipeline_artifacts
    WHERE ci_pipeline_artifacts.id BETWEEN $1 AND $2 AND ci_pipeline_artifacts.id >= $3
    ORDER BY ci_pipeline_artifacts.id ASC
    LIMIT $4
    OFFSET $5
    166 30.4 ms 1.7 ms 0.2 ms 166
    INSERT INTO batched_background_migration_job_transition_logs (batched_background_migration_job_id, created_at, updated_at, previous_status, next_status) VALUES ($1, $2, $3, $4, $5) RETURNING id
    83 10.9 ms 1.5 ms 0.1 ms 83
    UPDATE batched_background_migration_jobs
    SET updated_at = $1, started_at = $2, status = $3, attempts = $4
    WHERE batched_background_migration_jobs.id = $5
    166 10.1 ms 1.4 ms 0.1 ms 166
    SELECT batched_background_migration_jobs.*
    FROM batched_background_migration_jobs
    WHERE batched_background_migration_jobs.id = $1
    LIMIT $2
    83 12.3 ms 0.6 ms 0.1 ms 83
    UPDATE batched_background_migration_jobs
    SET updated_at = $1, finished_at = $2, status = $3, metrics = $4
    WHERE batched_background_migration_jobs.id = $5
    83 6.0 ms 0.2 ms 0.1 ms 83
    SELECT sum(batched_background_migration_jobs.batch_size)
    FROM batched_background_migration_jobs
    WHERE batched_background_migration_jobs.batched_background_migration_id = $1 AND batched_background_migration_jobs.status IN ($2)
    83 3.4 ms 0.1 ms 0.0 ms 83
    SELECT ci_pipeline_artifacts.id
    FROM ci_pipeline_artifacts
    WHERE ci_pipeline_artifacts.id BETWEEN $1 AND $2
    ORDER BY ci_pipeline_artifacts.id ASC
    LIMIT $3
    83 3.6 ms 0.1 ms 0.0 ms 83
    SELECT batched_background_migrations.*
    FROM batched_background_migrations
    WHERE batched_background_migrations.id = $1
    LIMIT $2
    83 3.1 ms 0.1 ms 0.0 ms 83
    SELECT $1
    FROM p_ci_builds
    WHERE partition_id = $2
    LIMIT $3
    Histogram of batch runtimes for BackfillPartitionIdCiPipelineArtifact
    Batch Runtime Count
    0 seconds - 10 seconds 81
    10 seconds - 1 minute 2
    1 minute - 2 minutes 0
    2 minutes - 3 minutes 0
    3 minutes - 5 minutes 0
    5 minutes + 0
    Histogram across all sampled batches of BackfillPartitionIdCiPipelineArtifact
    Query Runtime Count
    0 seconds - 0.1 seconds 1700
    0.1 seconds - 0.5 seconds 547
    0.5 seconds - 1 second 214
    1 second - 2 seconds 28
    2 seconds - 5 seconds 1
    5 seconds + 0

    Background Migration: BackfillPartitionIdCiPipelineMetadata

    Sampled 207 batches. Estimated Time to complete: 6 days, 17 hours, and 6 minutes
    • Interval: 120s
    • Total tuple count: 4833824
    • Max batch size: 0
    • Estimated seconds to complete: 579960s
    • Estimated number of batches: 4833
    • Average batch time: 2.18s
    • Batch size: 1000
    • N. of batches sampled: 207
    • N. of failed batches: 0

    Time estimation is conservative and based on sampling production data in a test environment. It represents the max time that migration could take. The actual time may differ from this estimation.

    _Consider changing max_batch_size and interval if this estimate is unacceptable._

    Calls Total Time Max Time Mean Time Rows Query
    618 99125.7 ms 2443.6 ms 160.4 ms 154500
    UPDATE ci_pipeline_metadata
    SET partition_id = ci_pipelines.partition_id
    FROM ci_pipelines
    WHERE ci_pipeline_metadata.pipeline_id BETWEEN $1 AND $2 AND ci_pipeline_metadata.pipeline_id >= $3 AND ci_pipeline_metadata.pipeline_id < $4 AND ci_pipeline_metadata.pipeline_id = ci_pipelines.id
    207 28340.6 ms 1499.7 ms 136.9 ms 51508
    UPDATE ci_pipeline_metadata
    SET partition_id = ci_pipelines.partition_id
    FROM ci_pipelines
    WHERE ci_pipeline_metadata.pipeline_id BETWEEN $1 AND $2 AND ci_pipeline_metadata.pipeline_id >= $3 AND ci_pipeline_metadata.pipeline_id = ci_pipelines.id
    414 95.6 ms 19.0 ms 0.2 ms 414
    INSERT INTO batched_background_migration_job_transition_logs (batched_background_migration_job_id, created_at, updated_at, previous_status, next_status) VALUES ($1, $2, $3, $4, $5) RETURNING id
    207 68.7 ms 11.1 ms 0.3 ms 207
    UPDATE batched_background_migration_jobs
    SET updated_at = $1, finished_at = $2, status = $3, metrics = $4
    WHERE batched_background_migration_jobs.id = $5
    825 236.5 ms 7.6 ms 0.3 ms 618
    SELECT ci_pipeline_metadata.pipeline_id
    FROM ci_pipeline_metadata
    WHERE ci_pipeline_metadata.pipeline_id BETWEEN $1 AND $2 AND ci_pipeline_metadata.pipeline_id >= $3
    ORDER BY ci_pipeline_metadata.pipeline_id ASC
    LIMIT $4
    OFFSET $5
    207 11.9 ms 4.4 ms 0.1 ms 207
    SELECT $1
    FROM p_ci_builds
    WHERE partition_id = $2
    LIMIT $3
    207 33.6 ms 4.0 ms 0.2 ms 207
    UPDATE batched_background_migration_jobs
    SET updated_at = $1, started_at = $2, status = $3, attempts = $4
    WHERE batched_background_migration_jobs.id = $5
    207 12.4 ms 2.6 ms 0.1 ms 207
    SELECT batched_background_migrations.*
    FROM batched_background_migrations
    WHERE batched_background_migrations.id = $1
    LIMIT $2
    414 23.2 ms 1.8 ms 0.1 ms 414
    SELECT batched_background_migration_jobs.*
    FROM batched_background_migration_jobs
    WHERE batched_background_migration_jobs.id = $1
    LIMIT $2
    207 21.0 ms 1.2 ms 0.1 ms 207
    SELECT sum(batched_background_migration_jobs.batch_size)
    FROM batched_background_migration_jobs
    WHERE batched_background_migration_jobs.batched_background_migration_id = $1 AND batched_background_migration_jobs.status IN ($2)
    207 7.2 ms 0.9 ms 0.0 ms 207
    SELECT ci_pipeline_metadata.pipeline_id
    FROM ci_pipeline_metadata
    WHERE ci_pipeline_metadata.pipeline_id BETWEEN $1 AND $2
    ORDER BY ci_pipeline_metadata.pipeline_id ASC
    LIMIT $3
    Histogram of batch runtimes for BackfillPartitionIdCiPipelineMetadata
    Batch Runtime Count
    0 seconds - 10 seconds 207
    10 seconds - 1 minute 0
    1 minute - 2 minutes 0
    2 minutes - 3 minutes 0
    3 minutes - 5 minutes 0
    5 minutes + 0
    Histogram across all sampled batches of BackfillPartitionIdCiPipelineMetadata
    Query Runtime Count
    0 seconds - 0.1 seconds 3510
    0.1 seconds - 0.5 seconds 91
    0.5 seconds - 1 second 86
    1 second - 2 seconds 31
    2 seconds - 5 seconds 2
    5 seconds + 0

    Background Migration: BackfillPartitionIdCiPipelineConfig

    Sampled 173 batches. Estimated Time to complete: 3 years, 2 months, 6 hours, 44 minutes, and 12 seconds
    • Interval: 120s
    • Total tuple count: 832955100
    • Max batch size: 0
    • Estimated seconds to complete: 99954600s
    • Estimated number of batches: 832955
    • Average batch time: 2.71s
    • Batch size: 1000
    • N. of batches sampled: 173
    • N. of failed batches: 0

    Time estimation is conservative and based on sampling production data in a test environment. It represents the max time that migration could take. The actual time may differ from this estimation.

    _Consider changing max_batch_size and interval if this estimate is unacceptable._

    Calls Total Time Max Time Mean Time Rows Query
    1554 7373.8 ms 554.5 ms 4.7 ms 155400
    UPDATE ci_pipelines_config
    SET partition_id = ci_pipelines.partition_id
    FROM ci_pipelines
    WHERE ci_pipelines_config.pipeline_id BETWEEN $1 AND $2 AND ci_pipelines_config.pipeline_id >= $3 AND ci_pipelines_config.pipeline_id >= $4 AND ci_pipelines_config.pipeline_id < $5 AND ci_pipelines_config.pipeline_id = ci_pipelines.id
    173 823.9 ms 306.6 ms 4.8 ms 17287
    UPDATE ci_pipelines_config
    SET partition_id = ci_pipelines.partition_id
    FROM ci_pipelines
    WHERE ci_pipelines_config.pipeline_id BETWEEN $1 AND $2 AND ci_pipelines_config.pipeline_id >= $3 AND ci_pipelines_config.pipeline_id >= $4 AND ci_pipelines_config.pipeline_id = ci_pipelines.id
    173 35.8 ms 27.0 ms 0.2 ms 173
    SELECT batched_background_migrations.*
    FROM batched_background_migrations
    WHERE batched_background_migrations.id = $1
    LIMIT $2
    1900 600.3 ms 26.1 ms 0.3 ms 1727
    SELECT ci_pipelines_config.pipeline_id
    FROM ci_pipelines_config
    WHERE ci_pipelines_config.pipeline_id BETWEEN $1 AND $2 AND ci_pipelines_config.pipeline_id >= $3 AND ci_pipelines_config.pipeline_id >= $4
    ORDER BY ci_pipelines_config.pipeline_id ASC
    LIMIT $5
    OFFSET $6
    346 64.7 ms 8.1 ms 0.2 ms 346
    INSERT INTO batched_background_migration_job_transition_logs (batched_background_migration_job_id, created_at, updated_at, previous_status, next_status) VALUES ($1, $2, $3, $4, $5) RETURNING id
    173 22.3 ms 5.2 ms 0.1 ms 173
    SELECT sum(batched_background_migration_jobs.batch_size)
    FROM batched_background_migration_jobs
    WHERE batched_background_migration_jobs.batched_background_migration_id = $1 AND batched_background_migration_jobs.status IN ($2)
    346 27.3 ms 2.6 ms 0.1 ms 346
    SELECT batched_background_migration_jobs.*
    FROM batched_background_migration_jobs
    WHERE batched_background_migration_jobs.id = $1
    LIMIT $2
    173 22.6 ms 2.4 ms 0.1 ms 173
    UPDATE batched_background_migration_jobs
    SET updated_at = $1, started_at = $2, status = $3, attempts = $4
    WHERE batched_background_migration_jobs.id = $5
    173 28.3 ms 2.0 ms 0.2 ms 173
    UPDATE batched_background_migration_jobs
    SET updated_at = $1, finished_at = $2, status = $3, metrics = $4
    WHERE batched_background_migration_jobs.id = $5
    173 7.4 ms 0.1 ms 0.0 ms 173
    SELECT min(commit_id)
    FROM p_ci_builds
    WHERE partition_id = $1
    Histogram of batch runtimes for BackfillPartitionIdCiPipelineConfig
    Batch Runtime Count
    0 seconds - 10 seconds 173
    10 seconds - 1 minute 0
    1 minute - 2 minutes 0
    2 minutes - 3 minutes 0
    3 minutes - 5 minutes 0
    5 minutes + 0
    Histogram across all sampled batches of BackfillPartitionIdCiPipelineConfig
    Query Runtime Count
    0 seconds - 0.1 seconds 5131
    0.1 seconds - 0.5 seconds 52
    0.5 seconds - 1 second 1
    1 second - 2 seconds 0
    2 seconds - 5 seconds 0
    5 seconds + 0

    Other information

    Other migrations pending on GitLab.com
    Migration Type Total runtime Result DB size change
    20240108123023 - CreateAiAgentsTable Regular 3.3 s :warning: +32.00 KiB
    20240108123115 - CreateAiAgentVersionsTable Regular 3.2 s :white_check_mark: +48.00 KiB
    20240108123509 - AddFkOnAiAgentVersionsToAiAgents Regular 2.9 s :white_check_mark: +0.00 B
    20240110085226 - AddRateLimitsToApplicationSettings Regular 2.5 s :white_check_mark: +0.00 B
    20231222072237 - AddTextLimitToCiInstanceVariablesDescription Post deploy 2.7 s :white_check_mark: +0.00 B
    20240104155616 - FinalizeBackfillDefaultBranchProtectionNamespaceSettings Post deploy 2.5 s :white_check_mark: +0.00 B
    20240107154747 - SentNotificationsSelfInstallFinalizeBbm Post deploy 2.3 s :white_check_mark: +0.00 B
    20240107154805 - SentNotificationsSelfInstallIdSwap Post deploy 2.3 s :white_check_mark: +0.00 B
    20240108082419 - QueueBackfillPartitionIdCiPipelineMetadata Post deploy 2.9 s :white_check_mark: +0.00 B
    20240108125135 - RemoveInternalIdsTriggers Post deploy 2.6 s :white_check_mark: +0.00 B
    20240108125335 - AddTemporaryIndexInternalIdsOnIdAndUsage Post deploy 2.9 s :white_check_mark: +8.00 KiB [note]
    20240108185335 - BackfillInternalIdsWithIssuesUsageForEpics Post deploy 2.4 s :white_check_mark: +0.00 B
    20240108215335 - RemoveInternalIdsTmpIndex Post deploy 2.8 s :white_check_mark: -8.00 KiB
    20240109090354 - QueueBackfillPartitionIdCiPipelineArtifact Post deploy 2.6 s :white_check_mark: +0.00 B
    20240110090352 - QueueBackfillPartitionIdCiPipelineConfig Post deploy 2.9 s :white_check_mark: +0.00 B
    20240110093654 - ScheduleIndexRemovalCiBuildTraceMetadata Post deploy 2.7 s :white_check_mark: +8.00 KiB [note]
    20240110094002 - DropIndexFromCiJobArtifactState Post deploy 3.0 s :white_check_mark: -5.93 GiB
    20240110094510 - RemoveFkFromCiJobArtifactState Post deploy 3.0 s :warning: +0.00 B
    20240110132029 - ReRemoveRequirementsIgnoredColumns Post deploy 3.4 s :warning: -48.00 KiB
    Clone details
    Clone ID Clone Created At Clone Data Timestamp Expected Removal Time
    database-testing-2747447-12500151-main 2024-01-15T13:18:25Z 2024-01-15T10:53:13Z 2024-01-16 01:58:17 +0000
    database-testing-2747447-12500151-ci 2024-01-15T13:18:25Z 2024-01-15T12:46:10Z 2024-01-16 01:58:17 +0000

    Job artifacts


    Brought to you by gitlab-org/database-team/gitlab-com-database-testing. Epic

  • Adam Hegyi resolved all threads

    resolved all threads

  • Adam Hegyi enabled an automatic merge when all merge checks for a9ffca21 pass

    enabled an automatic merge when all merge checks for a9ffca21 pass

  • merged

  • Adam Hegyi mentioned in commit 090e84ce

    mentioned in commit 090e84ce

  • Hello @abdwdd :wave:

    The database team is looking for ways to improve the database review process and we would love your help!

    If you'd be open to someone on the database team reaching out to you for a chat, or if you'd like to leave some feedback asynchronously, just post a reply to this comment mentioning:

    @gitlab-org/database-team

    And someone will be by shortly!

    Thanks for your help! :heart:

    This message was generated automatically. You're welcome to improve it.

  • added workflowstaging label and removed workflowcanary label

  • Abdul Wadood changed milestone to %16.8

    changed milestone to %16.8

  • Dominic Bauer mentioned in merge request !182484

    mentioned in merge request !182484

Please register or sign in to reply
Loading