XS Add sharding key YAML config to ci_pipelines & p_ci_pipelines
What we need to do
The ci_pipelines
table has a project_id
key, but didn't get the designation automatically a few months ago when the Cells team sent out all the initial MRs.
I'm not sure if there's something blocking this having to do with partitioning.
How to do it
- Find out if there's any reason we can't add the sharding key designation immediately and have all the automation work.
- Open up https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/docs/ci_pipelines.yml
table_name: ci_pipelines
classes:
- Ci::Pipeline
feature_categories:
- continuous_integration
description: TODO
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/commit/c6ae290cea4b88ecaa9cfe0bc9d88e8fd32070c1
milestone: '9.0'
gitlab_schema: gitlab_ci
+ sharding_key:
+ project_id: projects
Designs
- Show closed items
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- drew stachon added to epic &12323
added to epic &12323
- drew stachon changed milestone to %Backlog
changed milestone to %Backlog
- drew stachon changed the description
Compare with previous version changed the description
- drew stachon mentioned in epic &13074 (closed)
mentioned in epic &13074 (closed)
- drew stachon mentioned in epic &12323
mentioned in epic &12323
- drew stachon changed title from Add sharding key designation to ci_pipelines (1 table) to XS Add sharding key designation to ci_pipelines (1 table)
changed title from Add sharding key designation to ci_pipelines (1 table) to XS Add sharding key designation to ci_pipelines (1 table)
- drew stachon changed title from XS Add sharding key designation to ci_pipelines (1 table) to XS Add sharding key YAML config to ci_pipelines (1 table)
changed title from XS Add sharding key designation to ci_pipelines (1 table) to XS Add sharding key YAML config to ci_pipelines (1 table)
- drew stachon mentioned in issue #448630 (closed)
mentioned in issue #448630 (closed)
- Caroline Simpson changed milestone to %17.1
changed milestone to %17.1
- Caroline Simpson set weight to 1
set weight to 1
- Caroline Simpson mentioned in issue gitlab-org/ci-cd/pipeline-execution#209 (closed)
mentioned in issue gitlab-org/ci-cd/pipeline-execution#209 (closed)
- Subashis Chakraborty mentioned in epic &13087 (closed)
mentioned in epic &13087 (closed)
- Caroline Simpson marked this issue as related to #454946 (closed)
marked this issue as related to #454946 (closed)
- Caroline Simpson removed the relation with #454946 (closed)
removed the relation with #454946 (closed)
- Caroline Simpson marked this issue as blocking #454946 (closed)
marked this issue as blocking #454946 (closed)
- Subashis Chakraborty mentioned in issue #454946 (closed)
mentioned in issue #454946 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#17612 (closed)
mentioned in issue gitlab-org/quality/triage-reports#17612 (closed)
- Maintainer
Setting health status to
on track
as the milestone has just begun.Issue participants are welcome to override this by setting the health status to another value.
- 🤖 GitLab Bot 🤖 changed health status to on track
changed health status to on track
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#17715 (closed)
mentioned in issue gitlab-org/quality/triage-reports#17715 (closed)
- Developer
Experimental AI Estimation Thread Collapse replies - Developer
Estimated weight: 3.0 Estimated days for development: 1.0 Estimated number of MRs: 1.0
Estimates are generated based on historical work effort from issues that may be similar. The issues that were used to generate this data were as follows.
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#17797 (closed)
mentioned in issue gitlab-org/quality/triage-reports#17797 (closed)
- Max Fan assigned to @mfanGitLab
assigned to @mfanGitLab
- Maintainer
@drew going through all these cells issues and it looks like we have a lot of other tables that don't have sharding_keys. (ci_trigger_requests, ci_platform_metrics, ci_pipelines_config, etc). Do we need some issues to track all of these?
Also good question on why this wasn't automated by housekeeper. @tigerwnz @DylanGriffith do you have any insights? The
p_ci_pipelines
has aproject_id
so it should have been done in the 1st wave.And are we going to do another wave of MRs by housekeeper? This time for indirect tables (eg
ci_trigger_requests.yml
which referencesci_triggers.yml
which now has a sharding key)? Or should we start patching all the missed tables ourselves? Collapse replies - Maintainer
cc @manojmj also to investigate why this wasn't created automatically.
- Author Maintainer
@mfanGitLab Some of the tables are grouped together in a single issue if the change needed to be made is the same, e.g.
ci_platform_metrics
I started to think about why, but stopped because this is ultimately a one-off task and it's easy enough to write the YAML ourselves ¯\(ツ)/¯
- Maintainer
ci_pipelines.project_id
is nullable, which is why it was skipped by the housekeeper. Adding the sharding key as in the description will cause a failure insharding_key_spec.rb
, as we expect all sharding keys to have aNOT NULL
constraint. 1 Adding that we have created an MR with all tables currently missing sharding keys for the team to work on: !152830 (merged)
It is currently assigned to @hfyngvason, but it was a random assignment. @mfanGitLab - if you are working on this part, feel free to re-assign to yourself
Also adding that adding NOT NULL to
ci_pipelines.project_id
will also unblock quite a few other tables from having a desired sharding key. Those tables will then be able to backfillproject_id
from their relatedci_pipeline
record.It'd be nice if you could give this table some priority in setting up the NOT NULL constraint.
1- Maintainer
ahh gotcha, thank you everyone for the great explanations
Yes it looks like we'll need to prioritize doing that as it blocks this current issue.
- Manoj M J mentioned in merge request !152830 (merged)
mentioned in merge request !152830 (merged)
- Caroline Simpson changed due date to June 14, 2024
changed due date to June 14, 2024
- Max Fan removed the relation with #454946 (closed)
removed the relation with #454946 (closed)
- Max Fan marked this issue as blocked by #454946 (closed)
marked this issue as blocked by #454946 (closed)
- Max Fan added workflowblocked label and removed workflowready for development label
added workflowblocked label and removed workflowready for development label
- Max Fan unassigned @mfanGitLab
unassigned @mfanGitLab
- Max Fan marked this issue as blocking #458483 (closed)
marked this issue as blocking #458483 (closed)
- Caroline Simpson changed title from XS Add sharding key YAML config to ci_pipelines (1 table) to XS Add sharding key YAML config to ci_pipelines & p_ci_pipelines (1 table)
changed title from XS Add sharding key YAML config to ci_pipelines (1 table) to XS Add sharding key YAML config to ci_pipelines & p_ci_pipelines (1 table)
- Caroline Simpson changed milestone to %17.3
changed milestone to %17.3
- Caroline Simpson changed due date to August 09, 2024
changed due date to August 09, 2024
- Cheryl Li mentioned in issue #458483 (closed)
mentioned in issue #458483 (closed)
- Pedro Pombeiro marked this issue as blocking #470152 (closed)
marked this issue as blocking #470152 (closed)
- Pedro Pombeiro changed title from XS Add sharding key YAML config to ci_pipelines & p_ci_pipelines (1 table) to XS Add sharding key YAML config to ci_pipelines & p_ci_pipelines
changed title from XS Add sharding key YAML config to ci_pipelines & p_ci_pipelines (1 table) to XS Add sharding key YAML config to ci_pipelines & p_ci_pipelines
- Tianwen Chen removed the relation with #458483 (closed)
removed the relation with #458483 (closed)
- Chun Du mentioned in issue gitlab-org/database-team/team-tasks#447 (closed)
mentioned in issue gitlab-org/database-team/team-tasks#447 (closed)
- Contributor
Hi @mfanGitLab - is this issue no longer blocked now that #454946 (closed) is complete?
/cc @carolinesimpson
Edited by Mark Nuzzo Collapse replies - Maintainer
ah sorry, no. It's not ready I've updated the linked items. It's blocked until the table has a
not-null
constraint set. Otherwise specs will fail - Developer
The work for this issue is now completed, closing it!
- Max Fan marked this issue as blocked by #463405 (closed)
marked this issue as blocked by #463405 (closed)
- Caroline Simpson changed milestone to %17.4
changed milestone to %17.4
- Caroline Simpson added candidate17.4 label
added candidate17.4 label
- Caroline Simpson changed milestone to %Next 1-3 releases
changed milestone to %Next 1-3 releases
- Cheryl Li added verify cells label and removed severity2 label
added verify cells label and removed severity2 label
- Max Fan assigned to @mfanGitLab
assigned to @mfanGitLab
- Max Fan mentioned in issue gitlab-org/ci-cd/pipeline-execution#226
mentioned in issue gitlab-org/ci-cd/pipeline-execution#226
- Max Orefice mentioned in merge request !162823 (merged)
mentioned in merge request !162823 (merged)
- Max Fan mentioned in merge request !163054 (closed)
mentioned in merge request !163054 (closed)
- Max Orefice closed
closed
- Max Orefice mentioned in issue #470152 (closed)
mentioned in issue #470152 (closed)
- Pedro Pombeiro removed the relation with #470152 (closed)
removed the relation with #470152 (closed)
- Mark Nuzzo added workflowcomplete label and removed workflowblocked label
added workflowcomplete label and removed workflowblocked label
- Mark Nuzzo mentioned in epic gitlab-org#14749
mentioned in epic gitlab-org#14749
- service-epic-status-automation mentioned in epic &14749
mentioned in epic &14749
- service-epic-status-automation mentioned in epic &12383
mentioned in epic &12383
- Max Fan mentioned in issue #458490 (closed)
mentioned in issue #458490 (closed)
- drew stachon mentioned in epic gitlab-org#12323
mentioned in epic gitlab-org#12323