PoC for removing partitioned data using foreign keys
Description
Like in #357384 (closed) we want to ship a PoC that will prove that we can efficiently remove resources associated with pipelines (builds, stages, artifacts) using foreign keys and ON DELETE CASCADE
with partitioned CI/CD schema.
We want to better understand if using foreign keys with ON DELETE CASCADE
will have to scan all partitions to perform a removal. Is it even possible to rely ON DELETE CASCADE
with foreign keys in that case https://www.2ndquadrant.com/en/blog/postgresql-12-foreign-keys-and-partitioned-tables/?
Objectives
Check if removing resources through foreign keys actually makes it possible to scan all partitions.
Designs
- Show closed items
Relates to
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- Grzegorz Bizon added grouppipeline execution typemaintenance labels
added grouppipeline execution typemaintenance labels
- Grzegorz Bizon marked this issue as related to #357384 (closed)
marked this issue as related to #357384 (closed)
- Grzegorz Bizon added to epic &5417
added to epic &5417
- Author Maintainer
@jheimbuck_gl would it be possible to schedule this issue? Thanks!
Collapse replies - Contributor
@grzesiek sure thing. Are there any constraints we need to consider (performance, data loss, etc?) for the POC or that it should meet that we can add to the description?
For now the epic this is in is scheduled for FY23Q2 so i'm putting this into %15.1 for now. Are there follow-up items that can/should be created after the POC is complete?
- Contributor
@grzesiek looks like this is still in workflowplanning breakdown can you update description and weight/timebox for this in the coming week or move it to %15.2 please?
cc @marknuzzo
- Author Maintainer
@jheimbuck_gl I think that it should be sufficient to spend on it just a couple of days. We should just check if deleting rows of associated resources will trigger cascading delete that needs to scan all partitions.
- Author Maintainer
I added the weight.
- Contributor
HI @jheimbuck_gl - thanks for the ping here. Is this VerifyP1, VerifyP2 or VerifyP3?
- Contributor
@marknuzzo i'm labeling the CI Partitioning work with VerifyP2 where i remember
1 - Contributor
@grzesiek @marknuzzo if this is ready for dev would you move it to the proper workflow status? Thanks!
- Contributor
Thanks @jheimbuck_gl - based on @grzesiek note above, I think this is ready so I updated it to workflowready for development for %15.2.
2
- Grzegorz Bizon changed the description
Compare with previous version changed the description
- James Heimbuck changed milestone to %Next 1-3 releases
changed milestone to %Next 1-3 releases
- James Heimbuck added 1 deleted label
added 1 deleted label
- James Heimbuck changed milestone to %15.1
changed milestone to %15.1
- James Heimbuck added needs weight label
added needs weight label
- James Heimbuck added workflowplanning breakdown label
added workflowplanning breakdown label
- James Heimbuck mentioned in issue #357384 (closed)
mentioned in issue #357384 (closed)
- Maintainer
Setting label(s) devopsverify sectionops based on grouppipeline execution.
- 🤖 GitLab Bot 🤖 added devopsverify sectionops labels
added devopsverify sectionops labels
- Grzegorz Bizon set weight to 2
set weight to 2
- Mark Nuzzo removed needs weight label
removed needs weight label
- Mark Nuzzo added backend label
added backend label
- James Heimbuck added VerifyP2 label
added VerifyP2 label
- James Heimbuck changed milestone to %15.2
changed milestone to %15.2
- James Heimbuck added needs weight label
added needs weight label
- James Heimbuck mentioned in issue gitlab-org/ci-cd/pipeline-execution#101 (closed)
mentioned in issue gitlab-org/ci-cd/pipeline-execution#101 (closed)
- Sam Beckham removed needs weight label
removed needs weight label
- Mark Nuzzo added workflowready for development label and removed workflowplanning breakdown label
added workflowready for development label and removed workflowplanning breakdown label
- 🤖 GitLab Bot 🤖 added [deprecated] Accepting merge requests label
added [deprecated] Accepting merge requests label
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#7822 (closed)
mentioned in issue gitlab-org/quality/triage-reports#7822 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#7961 (closed)
mentioned in issue gitlab-org/quality/triage-reports#7961 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#8046 (closed)
mentioned in issue gitlab-org/quality/triage-reports#8046 (closed)
- Grzegorz Bizon mentioned in epic &5417
mentioned in epic &5417
- Mark Nuzzo added ciscaling label
added ciscaling label
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#8144 (closed)
mentioned in issue gitlab-org/quality/triage-reports#8144 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#8226 (closed)
mentioned in issue gitlab-org/quality/triage-reports#8226 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#8229 (closed)
mentioned in issue gitlab-org/quality/triage-reports#8229 (closed)
- Mark Nuzzo changed milestone to %15.3
changed milestone to %15.3
- James Heimbuck mentioned in issue gitlab-org/ci-cd/pipeline-execution#104 (closed)
mentioned in issue gitlab-org/ci-cd/pipeline-execution#104 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#8329 (closed)
mentioned in issue gitlab-org/quality/triage-reports#8329 (closed)
- Mark Nuzzo changed milestone to %15.5
changed milestone to %15.5
- Contributor
Moving this out to %15.5 for now so that %15.3 is manageable.
/cc @jheimbuck_gl
1 - Mark Nuzzo mentioned in issue gitlab-org/ci-cd/pipeline-execution#112 (closed)
mentioned in issue gitlab-org/ci-cd/pipeline-execution#112 (closed)
- Marius Bobin assigned to @mbobin
assigned to @mbobin
- Marius Bobin mentioned in merge request !92440 (closed)
mentioned in merge request !92440 (closed)
- Maintainer
Thanks for working on this @mbobin! We've removed the Seeking community contributions label to avoid having multiple people working on the same issue.
- 🤖 GitLab Bot 🤖 removed [deprecated] Accepting merge requests label
removed [deprecated] Accepting merge requests label
- Marius Bobin mentioned in merge request !92535 (merged)
mentioned in merge request !92535 (merged)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#8403 (closed)
mentioned in issue gitlab-org/quality/triage-reports#8403 (closed)
- Marius Bobin mentioned in merge request !92781 (merged)
mentioned in merge request !92781 (merged)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#8498 (closed)
mentioned in issue gitlab-org/quality/triage-reports#8498 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#8563 (closed)
mentioned in issue gitlab-org/quality/triage-reports#8563 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#8708 (closed)
mentioned in issue gitlab-org/quality/triage-reports#8708 (closed)
- Cheryl Li added CI data partitioning label
added CI data partitioning label
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#8781 (closed)
mentioned in issue gitlab-org/quality/triage-reports#8781 (closed)
- Marius Bobin mentioned in epic &7522
mentioned in epic &7522
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#8844 (closed)
mentioned in issue gitlab-org/quality/triage-reports#8844 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#8938 (closed)
mentioned in issue gitlab-org/quality/triage-reports#8938 (closed)
- Author Maintainer
With Add foreign key findings to CI data decay (!92781 - merged) merged we can probably close this issue! Awesome work @mbobin!
- Grzegorz Bizon closed
closed
- Marius Bobin changed milestone to %15.4
changed milestone to %15.4
- James Heimbuck removed 1 deleted label
removed 1 deleted label
- Caroline Simpson changed epic to &11807 (closed)
changed epic to &11807 (closed)
- 🤖 GitLab Bot 🤖 added sectionci label and removed sectionops label
added sectionci label and removed sectionops label