Remove triggers and old columns for bigint p_ci_builds
What does this MR do and why?
Remove triggers and old columns for bigint p_ci_builds
Changelog: removed
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
| Before | After |
|---|---|
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
Related to #422838 (closed)
Merge request reports
Activity
changed milestone to %Backlog
assigned to @tianwenchen
Database migrations (on the main database)
Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).
Migration Type Total runtime Result DB size change 20240617003021 - CleanupBigintConversionsForPCiBuilds Post deploy 6.2 s 
+0.00 B Runtime Histogram for all migrations
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 2 0.1 seconds - 1 second 10 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Migration: 20240617003021 - CleanupBigintConversionsForPCiBuilds
- Type: Post deploy
- Duration: 6.2 s
- Database size change: +0.00 B
Calls Total Time Max Time Mean Time Rows Query 1 25.1 ms 25.1 ms 25.1 ms 0 ALTER TABLE "p_ci_builds" DROP COLUMN "upstream_pipeline_id_convert_to_bigint"
1 17.2 ms 17.2 ms 17.2 ms 0 ALTER TABLE "p_ci_builds" DROP COLUMN "auto_canceled_by_id_convert_to_bigint"
1 14.7 ms 14.7 ms 14.7 ms 0 DROP TRIGGER IF EXISTS trigger_10ee1357e825 ON "p_ci_builds"
1 3.1 ms 3.1 ms 3.1 ms 0 ALTER TABLE "p_ci_builds" DROP COLUMN "commit_id_convert_to_bigint"
1 2.7 ms 2.7 ms 2.7 ms 0 DROP FUNCTION IF EXISTS trigger_10ee1357e825()
1 1.5 ms 1.5 ms 1.5 ms 0 ALTER TABLE "p_ci_builds" DROP COLUMN "trigger_request_id_convert_to_bigint"
1 1.5 ms 1.5 ms 1.5 ms 0 ALTER TABLE "p_ci_builds" DROP COLUMN "runner_id_convert_to_bigint"
1 1.4 ms 1.4 ms 1.4 ms 0 ALTER TABLE "p_ci_builds" DROP COLUMN "user_id_convert_to_bigint"
1 1.1 ms 1.1 ms 1.1 ms 0 ALTER TABLE "p_ci_builds" DROP COLUMN "erased_by_id_convert_to_bigint"
1 0.9 ms 0.9 ms 0.9 ms 0 ALTER TABLE "p_ci_builds" DROP COLUMN "project_id_convert_to_bigint"
2 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for CleanupBigintConversionsForPCiBuilds
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 2 0.1 seconds - 1 second 10 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Background Migration: BackfillVulnerabilityIssueLinksProjectId
Sampled 12 batches. Estimated Time to complete: 22 hours and 54 minutes
- Interval: 120s
- Total tuple count: 687037
- Max batch size: 0
- Estimated seconds to complete: 82440s
- Estimated number of batches: 687
- Average batch time: 5.71s
- Batch size: 1000
- N. of batches sampled: 12
- N. of failed batches: 0
Time estimation is conservative and based on sampling production data in a test environment. It represents the max time that migration could take. The actual time may differ from this estimation.
_Consider changing max_batch_size and interval if this estimate is unacceptable._
Calls Total Time Max Time Mean Time Rows Query 99 12182.2 ms 535.4 ms 123.1 ms 9900 UPDATE vulnerability_issue_links
SET project_id = vulnerabilities.project_id
FROM vulnerabilities
WHERE vulnerabilities.id = vulnerability_issue_links.vulnerability_id AND vulnerability_issue_links.id IN (
SELECT vulnerability_issue_links.id
FROM vulnerability_issue_links
WHERE vulnerability_issue_links.id BETWEEN $1 AND $2 AND vulnerability_issue_links.project_id IS NULL AND vulnerability_issue_links.id >= $3 AND vulnerability_issue_links.id < $4
)12 1063.2 ms 203.8 ms 88.6 ms 1102 UPDATE vulnerability_issue_links
SET project_id = vulnerabilities.project_id
FROM vulnerabilities
WHERE vulnerabilities.id = vulnerability_issue_links.vulnerability_id AND vulnerability_issue_links.id IN (
SELECT vulnerability_issue_links.id
FROM vulnerability_issue_links
WHERE vulnerability_issue_links.id BETWEEN $1 AND $2 AND vulnerability_issue_links.project_id IS NULL AND vulnerability_issue_links.id >= $3
)111 41.7 ms 1.2 ms 0.4 ms 99 SELECT vulnerability_issue_links.id
FROM vulnerability_issue_links
WHERE vulnerability_issue_links.id BETWEEN $1 AND $2 AND vulnerability_issue_links.project_id IS NULL AND vulnerability_issue_links.id >= $3
ORDER BY vulnerability_issue_links.id ASC
LIMIT $4
OFFSET $512 2.7 ms 1.2 ms 0.2 ms 12 UPDATE batched_background_migration_jobs
SET updated_at = $1, finished_at = $2, status = $3, metrics = $4
WHERE batched_background_migration_jobs.id = $524 3.9 ms 0.3 ms 0.2 ms 24 INSERT INTO batched_background_migration_job_transition_logs (batched_background_migration_job_id, created_at, updated_at, previous_status, next_status) VALUES ($1, $2, $3, $4, $5) RETURNING id
12 1.6 ms 0.3 ms 0.1 ms 12 UPDATE batched_background_migration_jobs
SET updated_at = $1, started_at = $2, status = $3, attempts = $4
WHERE batched_background_migration_jobs.id = $524 1.1 ms 0.1 ms 0.0 ms 24 SELECT batched_background_migration_jobs.*
FROM batched_background_migration_jobs
WHERE batched_background_migration_jobs.id = $1
LIMIT $212 0.5 ms 0.1 ms 0.0 ms 12 SELECT sum(batched_background_migration_jobs.batch_size)
FROM batched_background_migration_jobs
WHERE batched_background_migration_jobs.batched_background_migration_id = $1 AND batched_background_migration_jobs.status IN ($2)12 0.4 ms 0.0 ms 0.0 ms 12 SELECT vulnerability_issue_links.id
FROM vulnerability_issue_links
WHERE vulnerability_issue_links.id BETWEEN $1 AND $2 AND vulnerability_issue_links.project_id IS NULL
ORDER BY vulnerability_issue_links.id ASC
LIMIT $312 0.4 ms 0.0 ms 0.0 ms 12 SELECT batched_background_migrations.*
FROM batched_background_migrations
WHERE batched_background_migrations.id = $1
LIMIT $2Histogram of batch runtimes for BackfillVulnerabilityIssueLinksProjectId
Batch Runtime Count 0 seconds - 10 seconds 12 10 seconds - 1 minute 0 1 minute - 2 minutes 0 2 minutes - 3 minutes 0 3 minutes - 5 minutes 0 5 minutes + 0 Histogram across all sampled batches of BackfillVulnerabilityIssueLinksProjectId
Query Runtime Count 0 seconds - 0.1 seconds 37 0.1 seconds - 0.5 seconds 291 0.5 seconds - 1 second 2 1 second - 2 seconds 0 2 seconds - 5 seconds 0 5 seconds + 0 Background Migration: BackfillSbomOccurrencesVulnerabilitiesProjectId
Sampled 130 batches. Estimated Time to complete: 5 days, 21 hours, and 14 minutes
- Interval: 120s
- Total tuple count: 4237221
- Max batch size: 0
- Estimated seconds to complete: 508440s
- Estimated number of batches: 4237
- Average batch time: 5.35s
- Batch size: 1000
- N. of batches sampled: 130
- N. of failed batches: 0
Time estimation is conservative and based on sampling production data in a test environment. It represents the max time that migration could take. The actual time may differ from this estimation.
_Consider changing max_batch_size and interval if this estimate is unacceptable._
Calls Total Time Max Time Mean Time Rows Query 1161 60978.6 ms 334.2 ms 52.5 ms 116100 UPDATE sbom_occurrences_vulnerabilities
SET project_id = sbom_occurrences.project_id
FROM sbom_occurrences
WHERE sbom_occurrences.id = sbom_occurrences_vulnerabilities.sbom_occurrence_id AND sbom_occurrences_vulnerabilities.id IN (
SELECT sbom_occurrences_vulnerabilities.id
FROM sbom_occurrences_vulnerabilities
WHERE sbom_occurrences_vulnerabilities.id BETWEEN $1 AND $2 AND sbom_occurrences_vulnerabilities.project_id IS NULL AND sbom_occurrences_vulnerabilities.id >= $3 AND sbom_occurrences_vulnerabilities.id < $4
)130 5608.9 ms 229.9 ms 43.1 ms 12908 UPDATE sbom_occurrences_vulnerabilities
SET project_id = sbom_occurrences.project_id
FROM sbom_occurrences
WHERE sbom_occurrences.id = sbom_occurrences_vulnerabilities.sbom_occurrence_id AND sbom_occurrences_vulnerabilities.id IN (
SELECT sbom_occurrences_vulnerabilities.id
FROM sbom_occurrences_vulnerabilities
WHERE sbom_occurrences_vulnerabilities.id BETWEEN $1 AND $2 AND sbom_occurrences_vulnerabilities.project_id IS NULL AND sbom_occurrences_vulnerabilities.id >= $3
)260 73.8 ms 14.2 ms 0.3 ms 260 INSERT INTO batched_background_migration_job_transition_logs (batched_background_migration_job_id, created_at, updated_at, previous_status, next_status) VALUES ($1, $2, $3, $4, $5) RETURNING id
130 25.2 ms 7.3 ms 0.2 ms 130 UPDATE batched_background_migration_jobs
SET updated_at = $1, started_at = $2, status = $3, attempts = $4
WHERE batched_background_migration_jobs.id = $51291 274.1 ms 5.9 ms 0.2 ms 1161 SELECT sbom_occurrences_vulnerabilities.id
FROM sbom_occurrences_vulnerabilities
WHERE sbom_occurrences_vulnerabilities.id BETWEEN $1 AND $2 AND sbom_occurrences_vulnerabilities.project_id IS NULL AND sbom_occurrences_vulnerabilities.id >= $3
ORDER BY sbom_occurrences_vulnerabilities.id ASC
LIMIT $4
OFFSET $5130 25.3 ms 2.4 ms 0.2 ms 130 UPDATE batched_background_migration_jobs
SET updated_at = $1, finished_at = $2, status = $3, metrics = $4
WHERE batched_background_migration_jobs.id = $5260 12.8 ms 0.8 ms 0.0 ms 260 SELECT batched_background_migration_jobs.*
FROM batched_background_migration_jobs
WHERE batched_background_migration_jobs.id = $1
LIMIT $2130 10.8 ms 0.4 ms 0.1 ms 130 SELECT sum(batched_background_migration_jobs.batch_size)
FROM batched_background_migration_jobs
WHERE batched_background_migration_jobs.batched_background_migration_id = $1 AND batched_background_migration_jobs.status IN ($2)130 4.6 ms 0.1 ms 0.0 ms 130 SELECT sbom_occurrences_vulnerabilities.id
FROM sbom_occurrences_vulnerabilities
WHERE sbom_occurrences_vulnerabilities.id BETWEEN $1 AND $2 AND sbom_occurrences_vulnerabilities.project_id IS NULL
ORDER BY sbom_occurrences_vulnerabilities.id ASC
LIMIT $3130 5.1 ms 0.1 ms 0.0 ms 130 SELECT batched_background_migrations.*
FROM batched_background_migrations
WHERE batched_background_migrations.id = $1
LIMIT $2Histogram of batch runtimes for BackfillSbomOccurrencesVulnerabilitiesProjectId
Batch Runtime Count 0 seconds - 10 seconds 130 10 seconds - 1 minute 0 1 minute - 2 minutes 0 2 minutes - 3 minutes 0 3 minutes - 5 minutes 0 5 minutes + 0 Histogram across all sampled batches of BackfillSbomOccurrencesVulnerabilitiesProjectId
Query Runtime Count 0 seconds - 0.1 seconds 832 0.1 seconds - 0.5 seconds 2919 0.5 seconds - 1 second 1 1 second - 2 seconds 0 2 seconds - 5 seconds 0 5 seconds + 0 Other information
Other migrations pending on GitLab.com
Migration Type Total runtime Result DB size change 20240531202753 - QueueBackfillOrDropCiPipelineOnProjectId Post deploy 4.9 s +0.00 B 20240602161102 - IndexLast30DayUsageCountUpdatedAtOnCatalogResources Post deploy 5.6 s +40.00 KiB 20240602162649 - ChangeIndexPCatalogResourceComponentUsagesOnCatalogResourceId Post deploy 23.6 s +1.30 MiB 20240605232624 - DropIndexBoardsEpicBoardRecentVisitsOnUserId Post deploy 5.0 s -368.00 KiB 20240610105705 - EnsureBackfillMergeRequestsHeadPipelineIdIsCompleted Post deploy 4.9 s +0.00 B 20240610106705 - PrepareAsyncIndexesForMergeRequestsHeadPipelines Post deploy 8.8 s +0.00 B 20240612075302 - IndexSbomOccurrencesVulnerabilitiesOnProjectId Post deploy 16.8 s +28.02 MiB 20240612075303 - AddSbomOccurrencesVulnerabilitiesProjectIdFk Post deploy 5.5 s 
+0.00 B 20240612075304 - AddSbomOccurrencesVulnerabilitiesProjectIdTrigger Post deploy 4.5 s +8.00 KiB [note] 20240612075305 - QueueBackfillSbomOccurrencesVulnerabilitiesProjectId Post deploy 5.2 s +0.00 B 20240613064359 - IndexVulnerabilityIssueLinksOnProjectId Post deploy 7.4 s +4.56 MiB 20240613064360 - AddVulnerabilityIssueLinksProjectIdFk Post deploy 5.3 s +0.00 B 20240613064361 - AddVulnerabilityIssueLinksProjectIdTrigger Post deploy 4.5 s +0.00 B 20240613064362 - QueueBackfillVulnerabilityIssueLinksProjectId Post deploy 4.9 s +0.00 B Clone details
Clone ID Clone Created At Clone Data Timestamp Expected Removal Time database-testing-3401986-14318090-main2024-06-17T08:23:52Z 2024-06-17T04:09:58Z 2024-06-17 20:57:54 +0000 database-testing-3401986-14318090-ci2024-06-17T08:23:52Z 2024-06-17T04:45:41Z 2024-06-17 20:57:54 +0000 Database migrations (on the ci database)
Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).
Migration Type Total runtime Result DB size change 20240617003021 - CleanupBigintConversionsForPCiBuilds Post deploy 8.1 s +0.00 B Runtime Histogram for all migrations
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 2 0.1 seconds - 1 second 10 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Migration: 20240617003021 - CleanupBigintConversionsForPCiBuilds
- Type: Post deploy
- Duration: 8.1 s
- Database size change: +0.00 B
Calls Total Time Max Time Mean Time Rows Query 1 26.4 ms 26.4 ms 26.4 ms 0 ALTER TABLE "p_ci_builds" DROP COLUMN "auto_canceled_by_id_convert_to_bigint"
1 8.1 ms 8.1 ms 8.1 ms 0 ALTER TABLE "p_ci_builds" DROP COLUMN "commit_id_convert_to_bigint"
1 6.2 ms 6.2 ms 6.2 ms 0 ALTER TABLE "p_ci_builds" DROP COLUMN "user_id_convert_to_bigint"
1 6.2 ms 6.2 ms 6.2 ms 0 ALTER TABLE "p_ci_builds" DROP COLUMN "runner_id_convert_to_bigint"
1 6.2 ms 6.2 ms 6.2 ms 0 DROP TRIGGER IF EXISTS trigger_10ee1357e825 ON "p_ci_builds"
1 5.8 ms 5.8 ms 5.8 ms 0 ALTER TABLE "p_ci_builds" DROP COLUMN "project_id_convert_to_bigint"
1 5.4 ms 5.4 ms 5.4 ms 0 ALTER TABLE "p_ci_builds" DROP COLUMN "trigger_request_id_convert_to_bigint"
1 4.4 ms 4.4 ms 4.4 ms 0 ALTER TABLE "p_ci_builds" DROP COLUMN "upstream_pipeline_id_convert_to_bigint"
1 3.6 ms 3.6 ms 3.6 ms 0 ALTER TABLE "p_ci_builds" DROP COLUMN "erased_by_id_convert_to_bigint"
1 3.2 ms 3.2 ms 3.2 ms 0 DROP FUNCTION IF EXISTS trigger_10ee1357e825()
2 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for CleanupBigintConversionsForPCiBuilds
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 2 0.1 seconds - 1 second 10 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Background Migration: BackfillOrDropCiPipelineOnProjectId
Sampled 1 batches. Estimated Time to complete: 4 years, 6 months, 2 weeks, 6 days, 13 hours, 8 minutes, and 36 seconds
- Interval: 120s
- Total tuple count: 1198180700
- Max batch size: 0
- Estimated seconds to complete: 143781600s
- Estimated number of batches: 1198180
- Average batch time: 90.44s
- Batch size: 1000
- N. of batches sampled: 1
- N. of failed batches: 0
Time estimation is conservative and based on sampling production data in a test environment. It represents the max time that migration could take. The actual time may differ from this estimation.
_Consider changing max_batch_size and interval if this estimate is unacceptable._
Calls Total Time Max Time Mean Time Rows Query 181 549.5 ms 270.2 ms 3.0 ms 181 DELETE
FROM ci_pipelines
WHERE ci_pipelines.id = $1 AND ci_pipelines.lock_version = $247 496.5 ms 207.9 ms 10.6 ms 47 UPDATE ci_pipelines
SET project_id = $1
WHERE ci_pipelines.id = $2 AND ci_pipelines.lock_version = $32 129.0 ms 86.6 ms 64.5 ms 200 SELECT ci_pipelines.*
FROM ci_pipelines
WHERE ci_pipelines.id BETWEEN $1 AND $2 AND ci_pipelines.project_id IS NULL AND ci_pipelines.id >= $3 AND ci_pipelines.id < $41 24.8 ms 24.8 ms 24.8 ms 28 SELECT ci_pipelines.*
FROM ci_pipelines
WHERE ci_pipelines.id BETWEEN $1 AND $2 AND ci_pipelines.project_id IS NULL AND ci_pipelines.id >= $3228 208.4 ms 20.8 ms 0.9 ms 47 SELECT p_ci_builds.project_id
FROM p_ci_builds
WHERE p_ci_builds.commit_id = $1 AND p_ci_builds.project_id IS NOT NULL
ORDER BY p_ci_builds.id ASC
LIMIT $22 19.8 ms 19.5 ms 9.9 ms 2 INSERT INTO batched_background_migration_job_transition_logs (batched_background_migration_job_id, created_at, updated_at, previous_status, next_status) VALUES ($1, $2, $3, $4, $5) RETURNING id
181 13.5 ms 5.6 ms 0.1 ms 0 DELETE
FROM ci_trigger_requests
WHERE ci_trigger_requests.commit_id = $11 0.8 ms 0.8 ms 0.8 ms 1 UPDATE batched_background_migration_jobs
SET updated_at = $1, started_at = $2, status = $3, attempts = $4
WHERE batched_background_migration_jobs.id = $53 0.4 ms 0.1 ms 0.1 ms 2 SELECT ci_pipelines.id
FROM ci_pipelines
WHERE ci_pipelines.id BETWEEN $1 AND $2 AND ci_pipelines.project_id IS NULL AND ci_pipelines.id >= $3
ORDER BY ci_pipelines.id ASC
LIMIT $4
OFFSET $51 0.1 ms 0.1 ms 0.1 ms 1 UPDATE batched_background_migration_jobs
SET updated_at = $1, finished_at = $2, status = $3, metrics = $4
WHERE batched_background_migration_jobs.id = $51 0.1 ms 0.1 ms 0.1 ms 1 SELECT ci_pipelines.id
FROM ci_pipelines
WHERE ci_pipelines.id BETWEEN $1 AND $2 AND ci_pipelines.project_id IS NULL
ORDER BY ci_pipelines.id ASC
LIMIT $32 0.1 ms 0.0 ms 0.0 ms 2 SELECT batched_background_migration_jobs.*
FROM batched_background_migration_jobs
WHERE batched_background_migration_jobs.id = $1
LIMIT $21 0.0 ms 0.0 ms 0.0 ms 1 SELECT batched_background_migrations.*
FROM batched_background_migrations
WHERE batched_background_migrations.id = $1
LIMIT $21 0.0 ms 0.0 ms 0.0 ms 1 SELECT sum(batched_background_migration_jobs.batch_size)
FROM batched_background_migration_jobs
WHERE batched_background_migration_jobs.batched_background_migration_id = $1 AND batched_background_migration_jobs.status IN ($2)Histogram of batch runtimes for BackfillOrDropCiPipelineOnProjectId
Batch Runtime Count 0 seconds - 10 seconds 0 10 seconds - 1 minute 0 1 minute - 2 minutes 1 2 minutes - 3 minutes 0 3 minutes - 5 minutes 0 5 minutes + 0 Histogram across all sampled batches of BackfillOrDropCiPipelineOnProjectId
Query Runtime Count 0 seconds - 0.1 seconds 502 0.1 seconds - 0.5 seconds 331 0.5 seconds - 1 second 0 1 second - 2 seconds 0 2 seconds - 5 seconds 0 5 seconds + 0 Other information
Other migrations pending on GitLab.com
Migration Type Total runtime Result DB size change 20240531202753 - QueueBackfillOrDropCiPipelineOnProjectId Post deploy 8.4 s +0.00 B 20240602161102 - IndexLast30DayUsageCountUpdatedAtOnCatalogResources Post deploy 8.1 s +8.00 KiB [note] 20240602162649 - ChangeIndexPCatalogResourceComponentUsagesOnCatalogResourceId Post deploy 24.8 s +0.00 B 20240605232624 - DropIndexBoardsEpicBoardRecentVisitsOnUserId Post deploy 7.0 s -8.00 KiB 20240610105705 - EnsureBackfillMergeRequestsHeadPipelineIdIsCompleted Post deploy 5.9 s +0.00 B 20240610106705 - PrepareAsyncIndexesForMergeRequestsHeadPipelines Post deploy 10.0 s +8.00 KiB [note] 20240612075302 - IndexSbomOccurrencesVulnerabilitiesOnProjectId Post deploy 7.0 s +8.00 KiB [note] 20240612075303 - AddSbomOccurrencesVulnerabilitiesProjectIdFk Post deploy 7.2 s +0.00 B 20240612075304 - AddSbomOccurrencesVulnerabilitiesProjectIdTrigger Post deploy 6.5 s +0.00 B 20240612075305 - QueueBackfillSbomOccurrencesVulnerabilitiesProjectId Post deploy 6.2 s +0.00 B 20240613064359 - IndexVulnerabilityIssueLinksOnProjectId Post deploy 7.1 s +8.00 KiB [note] 20240613064360 - AddVulnerabilityIssueLinksProjectIdFk Post deploy 7.0 s +0.00 B 20240613064361 - AddVulnerabilityIssueLinksProjectIdTrigger Post deploy 6.4 s +8.00 KiB [note] 20240613064362 - QueueBackfillVulnerabilityIssueLinksProjectId Post deploy 6.1 s +0.00 B Clone details
Clone ID Clone Created At Clone Data Timestamp Expected Removal Time database-testing-3401986-14318090-main2024-06-17T08:23:52Z 2024-06-17T04:09:58Z 2024-06-17 20:57:54 +0000 database-testing-3401986-14318090-ci2024-06-17T08:23:52Z 2024-06-17T04:45:41Z 2024-06-17 20:57:54 +0000
Brought to you by gitlab-org/database-team/gitlab-com-database-testing. Epic
Edited by ****- A deleted user
added databasereview pending label
1 Warning You've made some app changes, but didn't add any tests.
That's OK as long as you're refactoring existing code,
but please consider adding any of the maintenancepipelines, maintenancerefactor, maintenanceworkflow, documentation, QA labels.Reviewer roulette
Category Reviewer Maintainer database @bhrai(UTC+2, 8 hours behind author)
@pshutsin(UTC+2, 8 hours behind author)
Please check reviewer's status!
-
Reviewer is available!
-
Reviewer is unavailable!
Please refer to documentation page for guidance on how you can benefit from the Reviewer Roulette, or use the GitLab Review Workload Dashboard to find other available reviewers.
If needed, you can retry the
danger-reviewjob that generated this comment.Generated by
Edited by Ghost User-
- A deleted user
added Data WarehouseImpact Check label
added database-testing-automation label
changed milestone to %17.1
added 403 commits
-
8f5a53cf...5b615a30 - 402 commits from branch
master - fface3f5 - Remove triggers and old columns for bigint p_ci_builds
-
8f5a53cf...5b615a30 - 402 commits from branch
removed milestone %17.1
changed milestone to %17.2
added pipelinetier-1 label
added 6907 commits
-
fface3f5...f193d1aa - 6906 commits from branch
master - 63dfc2fd - Remove triggers and old columns for bigint p_ci_builds
-
fface3f5...f193d1aa - 6906 commits from branch
added 1 commit
- 455e0a92 - Remove triggers and old columns for bigint p_ci_builds
added 1 commit
- 950b9c64 - Remove triggers and old columns for bigint p_ci_builds
requested review from @morefice
- Resolved by Marius Bobin
@morefice could you help to do the first review? If it looks good to you, could you pass it onto
@mbobin? Thanks in advance! - Last reply by Max Orefice
added pipeline:mr-approved label
added pipelinetier-2 label and removed pipelinetier-1 label
Resolved by Max OreficeBefore you set this MR to auto-merge
This merge request will progress on pipeline tiers until it reaches the last tier: pipelinetier-3. We will trigger a new pipeline for each transition to a higher tier.
Before you resolve this discussion, please check the following:
- You are the last maintainer of this merge request
- The latest pipeline for this merge request is pipelinetier-3 (You can find which tier it is in the pipeline name)
- This pipeline is recent enough (created in the last 8 hours)
If all the criteria above apply, please resolve this discussion and the set auto-merge for this merge request.
See pipeline tiers and merging a merge request for more details.
requested review from @mbobin
removed review request for @mbobin
E2E Test Result Summary
allure-report-publishergenerated test report!e2e-test-on-gdk:
expand test summary
+------------------------------------------------------------------+ | suites summary | +-------------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +-------------+--------+--------+---------+-------+-------+--------+ | Create | 121 | 0 | 11 | 0 | 132 | ✅ | | Manage | 0 | 0 | 1 | 0 | 1 | ➖ | | Plan | 56 | 0 | 2 | 0 | 58 | ✅ | | Package | 19 | 0 | 12 | 0 | 31 | ✅ | | Verify | 30 | 0 | 2 | 0 | 32 | ✅ | | Govern | 65 | 0 | 0 | 0 | 65 | ✅ | | Release | 5 | 0 | 0 | 0 | 5 | ✅ | | Analytics | 2 | 0 | 0 | 0 | 2 | ✅ | | Data Stores | 31 | 0 | 0 | 0 | 31 | ✅ | | Fulfillment | 1 | 0 | 0 | 0 | 1 | ✅ | | Monitor | 8 | 0 | 0 | 0 | 8 | ✅ | +-------------+--------+--------+---------+-------+-------+--------+ | Total | 338 | 0 | 28 | 0 | 366 | ✅ | +-------------+--------+--------+---------+-------+-------+--------+e2e-package-and-test:
expand test summary
+-------------------------------------------------------------+ | suites summary | +--------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +--------+--------+--------+---------+-------+-------+--------+ | Verify | 55 | 0 | 11 | 0 | 66 | ✅ | | Create | 276 | 0 | 30 | 0 | 306 | ✅ | | Govern | 2 | 0 | 0 | 0 | 2 | ✅ | | Plan | 4 | 0 | 0 | 0 | 4 | ✅ | +--------+--------+--------+---------+-------+-------+--------+ | Total | 337 | 0 | 41 | 0 | 378 | ✅ | +--------+--------+--------+---------+-------+-------+--------+e2e-test-on-cng:
expand test summary
+------------------------------------------------------------------+ | suites summary | +-------------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +-------------+--------+--------+---------+-------+-------+--------+ | Package | 23 | 0 | 15 | 0 | 38 | ✅ | | Fulfillment | 2 | 0 | 24 | 0 | 26 | ✅ | | Create | 135 | 0 | 15 | 0 | 150 | ✅ | | Govern | 77 | 0 | 9 | 0 | 86 | ✅ | | Plan | 78 | 1 | 6 | 0 | 85 | ❌ | | Data Stores | 34 | 0 | 9 | 0 | 43 | ✅ | | Growth | 0 | 0 | 2 | 0 | 2 | ➖ | | Manage | 2 | 0 | 8 | 0 | 10 | ✅ | | Monitor | 8 | 0 | 12 | 0 | 20 | ✅ | | Release | 5 | 0 | 1 | 0 | 6 | ✅ | | Ai-powered | 0 | 0 | 1 | 0 | 1 | ➖ | | Secure | 2 | 1 | 1 | 0 | 4 | ❌ | | Verify | 55 | 0 | 11 | 0 | 66 | ✅ | | Configure | 0 | 0 | 3 | 0 | 3 | ➖ | | Analytics | 2 | 0 | 0 | 0 | 2 | ✅ | | ModelOps | 0 | 0 | 1 | 0 | 1 | ➖ | +-------------+--------+--------+---------+-------+-------+--------+ | Total | 423 | 2 | 118 | 0 | 543 | ❌ | +-------------+--------+--------+---------+-------+-------+--------+Edited by Ghost Useradded Data WarehouseNot Impacted label and removed Data WarehouseImpact Check label
added pipelinetier-3 label and removed pipelinetier-2 label
added databaseapproved label and removed databasereview pending label
enabled automatic add to merge train when the pipeline for 1e1f4e5d succeeds
-
Hello @tianwenchen
The database team is looking for ways to improve the database review process and we would love your help!
If you'd be open to someone on the database team reaching out to you for a chat, or if you'd like to leave some feedback asynchronously, just post a reply to this comment mentioning:
@gitlab-org/database-teamAnd someone will be by shortly!
Thanks for your help!
This message was generated automatically. You're welcome to improve it.
mentioned in commit a41b55e7
added workflowstaging-canary label
added workflowcanary label and removed workflowstaging-canary label
added CI PK conversion workflowstaging labels and removed CI PK conversion workflowcanary labels
added CI PK conversion workflowproduction labels and removed CI PK conversion workflowstaging labels
added workflowpost-deploy-db-staging label and removed workflowproduction label
mentioned in incident gitlab-com/gl-infra/production#18176 (closed)
mentioned in merge request !157029 (merged)
20240617003021_cleanup_bigint_conversions_for_p_ci_buildsmigration fails on 2k performance test environments and causes upgrade failure, raised PG::DependentObjectsStillExist: ERROR - db/post... (#468671 - closed) to track this. Could you please take a look how it might be resolved?This migration is reverted in !157029 (merged). It shouldn't cause any failure in latest
mainbranch.Could you restart the pipeline and see how it'd go? And could you tell me how database is setup in this quality pipeline? Is it using proxy to a database clone? Cheers.
-
@tianwenchen thanks for the clarification
@niskhakova I mean do you know if the quality pipeline uses a database clone e.g. from production to run the tests? Since I want to see if I can reproduce it using the same database clone and find out the root cause.
-
@tianwenchen thanks for the clarification, performance pipelines are using its own custom data. Please let me know if that will be helpful to export as DB dump, it would be several GBs probably.
@niskhakova if performance pipelines are using its own custom data, could you help to run this fix from #468541 (comment 1965892789)?
Click to expand the fix
SELECT 'ALTER SEQUENCE ' || quote_ident(min(schema_name)) || '.' || quote_ident(min(seq_name)) || ' OWNED BY ' || quote_ident(min(table_name)) || '.' || quote_ident(min(column_name)) || ';' FROM ( SELECT n.nspname AS schema_name, c.relname AS table_name, a.attname AS column_name, substring(pg_get_expr(d.adbin, d.adrelid) from E'nextval\\(''([^'']*)''(?:::text|::regclass)?\\)') AS seq_name FROM pg_class c JOIN pg_attribute a ON c.oid = a.attrelid JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum JOIN pg_namespace n ON c.relnamespace = n.oid WHERE has_schema_privilege(n.oid, 'USAGE') AND n.nspname NOT LIKE 'pg\_%' ESCAPE '!' AND has_table_privilege(c.oid, 'SELECT') AND NOT a.attisdropped AND pg_get_expr(d.adbin, d.adrelid) ~ '^nextval' ) seq GROUP BY seq_name HAVING count(*) = 1;And you can run the following SQL statement to confirm the changes before and after the above fix:
\d+ x509_issuers_id_seq;Since I'm trying to get the same migration in again at !158711 (merged), you might run into the same problem and need to run the above fix.
-
@tianwenchen thanks for linking the fix, will keep it in mind in case it will be needed. For now the environment issue was resolved when package with reverted MR was installed
mentioned in issue #468671 (closed)
mentioned in merge request !157408 (closed)
mentioned in merge request !158711 (merged)
added releasedcandidate label
added releasedpublished label and removed releasedcandidate label
