XS Add sharding key YAML config to ci_stages & p_ci_stages (1 table)
What we need to do
The ci_stages
table has a project_id
key, but didn't get the designation automatically a few months ago when the Cells team sent out all the initial MRs.
I'm not sure if there's something blocking this having to do with partitioning.
How to do it
- Find out if there's any reason we can't add the sharding key designation immediately and have all the automation work.
- Yes there's a reason, we'll need to patch invalid records first
- Open up https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/docs/ci_stages.yml
---
table_name: ci_stages
classes:
- Ci::Stage
feature_categories:
- continuous_integration
description: TODO
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/commit/acc22a8422cd1471819510aa375c455b5ea009c5
milestone: '9.3'
gitlab_schema: gitlab_ci
+ sharding_key:
+ project_id: projects
Tasks required to complete this issue
- Remove orphaned stages through post-migration
- add validation for presence of project_id on stage
-
add not valid not null constraint to
pipeline_id
-
validate not null constraint on
pipeline_id
once added - add sharding key to stages
Designs
- Show closed items
Related branches 1
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- drew stachon added to epic &12323
added to epic &12323
- drew stachon changed the description
Compare with previous version changed the description
- drew stachon mentioned in epic &13074 (closed)
mentioned in epic &13074 (closed)
- Maintainer
@drew, Please add a group or category label to identify issue ownership.You can refer to the Features by Group handbook page for guidance.
If you are unsure about the correct group, please do not leave the issue without a group label, and refer to GitLab's shared responsibility functionality guidelines for more information on how to triage this kind of issue.
This message was generated automatically. You're welcome to improve it.
- drew stachon mentioned in epic &12323
mentioned in epic &12323
- drew stachon changed title from Add sharding key designation to ci_stages (1 table) to XS Add sharding key designation to ci_stages (1 table)
changed title from Add sharding key designation to ci_stages (1 table) to XS Add sharding key designation to ci_stages (1 table)
- drew stachon changed milestone to %Backlog
changed milestone to %Backlog
- drew stachon changed title from XS Add sharding key designation to ci_stages (1 table) to XS Add sharding key YAML config to ci_stages (1 table)
changed title from XS Add sharding key designation to ci_stages (1 table) to XS Add sharding key YAML config to ci_stages (1 table)
- drew stachon mentioned in issue #448630 (closed)
mentioned in issue #448630 (closed)
- Caroline Simpson changed milestone to %17.1
changed milestone to %17.1
- Caroline Simpson set weight to 1
set weight to 1
- Caroline Simpson mentioned in issue gitlab-org/ci-cd/pipeline-execution#209 (closed)
mentioned in issue gitlab-org/ci-cd/pipeline-execution#209 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#17612 (closed)
mentioned in issue gitlab-org/quality/triage-reports#17612 (closed)
- Maintainer
Setting health status to
on track
as the milestone has just begun.Issue participants are welcome to override this by setting the health status to another value.
- 🤖 GitLab Bot 🤖 changed health status to on track
changed health status to on track
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#17715 (closed)
mentioned in issue gitlab-org/quality/triage-reports#17715 (closed)
- Developer
Experimental AI Estimation Thread Collapse replies - Developer
Estimated weight: 3.0 Estimated days for development: 2.0 Estimated number of MRs: 3.0
Estimates are generated based on historical work effort from issues that may be similar. The issues that were used to generate this data were as follows.
- S Add sharding_key to p_ci_job_artifacts (#439686 - closed)
- Swap primary key for ci_stages (#429004 - closed)
- Validate referenced FK for ci_job_stages (#436527 - closed)
- Add index for partitioned PK on ci_stages (#429001 - closed)
- Swap primary key for ci_stages (#439064 - closed)
- Ensure that all tables that reference ci_pipeli... (#430722 - closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#17797 (closed)
mentioned in issue gitlab-org/quality/triage-reports#17797 (closed)
- Caroline Simpson changed due date to June 14, 2024
changed due date to June 14, 2024
- Max Fan changed title from XS Add sharding key YAML config to ci_stages (1 table) to XS Add sharding key YAML config to ci_stages & p_ci_stages (1 table)
changed title from XS Add sharding key YAML config to ci_stages (1 table) to XS Add sharding key YAML config to ci_stages & p_ci_stages (1 table)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#17903 (closed)
mentioned in issue gitlab-org/quality/triage-reports#17903 (closed)
- Allison Browne added onboarding label
added onboarding label
- Max Fan changed the description
Compare with previous version changed the description
- Maintainer
Looking at production db, we do have stages that don't have a
project_id
.[ gprd ] production> Ci::Stage.where(project_id: nil).count => 33
[ gprd ] production> Ci::Stage.find(12093957) => #<Ci::Stage:0x00007f1e683a5f10 project_id: nil, created_at: nil, updated_at: nil, name: nil, status: "skipped", lock_version: 0, position: nil, id: 12093957, partition_id: 100, pipeline_id: 11218112>
Because we can't shard on a null id. We'll need to fix these records on production first.
#463405 (closed) should handle some of them, but there's others that don't even have a
pipeline_id
so will need to also be handled - Max Fan changed weight to 4 from 1
changed weight to 4 from 1
- Max Fan changed weight to 3 from 4
changed weight to 3 from 4
- Maintainer
This issue is scheduled for completion in this milestone but doesn't have an assignee. Changing health status to 'needs attention'.
Issue participants are welcome to override this by setting the health status to another value.
- 🤖 GitLab Bot 🤖 changed health status to needs attention
changed health status to needs attention
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#18068 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18068 (closed)
- Caroline Simpson changed milestone to %17.2
changed milestone to %17.2
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#18167 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18167 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#18251 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18251 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#18359 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18359 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#18440 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18440 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#18636 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18636 (closed)
- Maintainer
This issue is scheduled for completion in this milestone but doesn't have an assignee. Changing health status to 'at risk'.
Issue participants are welcome to override this by setting the health status to another value.
- 🤖 GitLab Bot 🤖 changed health status to at risk
changed health status to at risk
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#18723 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18723 (closed)
- Contributor
Hi @mfanGitLab - is this work moving over to %17.3?
/cc @carolinesimpson
Collapse replies - Maintainer
@marknuzzo because of the database release policy, this will be moving all the way into %17.4. This shouldn't block any work though, as the invalid backfills for builds and pipeline are fixed in 17.2
1
- Chun Du mentioned in issue gitlab-org/database-team/team-tasks#447 (closed)
mentioned in issue gitlab-org/database-team/team-tasks#447 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#18805 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18805 (closed)
- Caroline Simpson assigned to @vwolanyk
assigned to @vwolanyk
- Caroline Simpson added candidate17.4 label
added candidate17.4 label
- Caroline Simpson changed milestone to %Next 1-3 releases
changed milestone to %Next 1-3 releases
- Cheryl Li added verify cells label and removed severity2 label
added verify cells label and removed severity2 label
- Vlad Wolanyk changed milestone to %17.4
changed milestone to %17.4
- Vlad Wolanyk removed due date June 14, 2024
removed due date June 14, 2024
- Vlad Wolanyk mentioned in issue gitlab-org/ci-cd/pipeline-execution#226
mentioned in issue gitlab-org/ci-cd/pipeline-execution#226
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#19088 (closed)
mentioned in issue gitlab-org/quality/triage-reports#19088 (closed)
- Maintainer
Setting health status to
on track
as the milestone has just begun.Issue participants are welcome to override this by setting the health status to another value.
- 🤖 GitLab Bot 🤖 changed health status to on track
changed health status to on track
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#19200 (closed)
mentioned in issue gitlab-org/quality/triage-reports#19200 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#19286 (closed)
mentioned in issue gitlab-org/quality/triage-reports#19286 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#19370 (closed)
mentioned in issue gitlab-org/quality/triage-reports#19370 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#19541 (closed)
mentioned in issue gitlab-org/quality/triage-reports#19541 (closed)
- Vlad Wolanyk changed milestone to %17.5
changed milestone to %17.5
- Developer
@mfanGitLab @drew - I ssh'd into prod console and it looks like there are now only 2 Ci::Stage records without a project_id
- I checked to see if they had any other associated records that might provide a project_id like builds and bridges but there is nothing
Any suggestions on how to proceed? Both stages have a status of "skipped" and both were created on the same day around the same time(not sure if that makes a difference)
Collapse replies - Maintainer
Thanks for checking @vwolanyk ! FYI you can use postgres.ai for this and it's much faster & easier than doing an access request: https://console.postgres.ai/gitlab/gitlab-production-ci/sessions/31376/commands/97387
In the execution tab
"Actual Rows": 2,
Aw man, unfortunately that means we'll need to do something similar as what we did in
p_ci_builds
and/orp_ci_pipelines
eg: #458486 (closed)- post_migration to fix/drop the invalid records depending if there's associations and if it's fixable
- Add not-null constraint
- Add sharding key
1 - Developer
@mfanGitLab - Potentially, I met with @drew and my next step is going to be to check in with the cells team to see if there is an automation that will delete these "orphaned records/builds" if an association cannot be found.
- Vlad Wolanyk added workflowin dev label and removed workflowready for development label
added workflowin dev label and removed workflowready for development label
- Vlad Wolanyk mentioned in issue gitlab-org/ci-cd/pipeline-execution#227
mentioned in issue gitlab-org/ci-cd/pipeline-execution#227
- Vlad Wolanyk mentioned in merge request !165094 (closed)
mentioned in merge request !165094 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#19656 (closed)
mentioned in issue gitlab-org/quality/triage-reports#19656 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#19668 (closed)
mentioned in issue gitlab-org/quality/triage-reports#19668 (closed)
- drew stachon mentioned in epic gitlab-org#12323
mentioned in epic gitlab-org#12323
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#19805 (closed)
mentioned in issue gitlab-org/quality/triage-reports#19805 (closed)