Ci Job Variables has job_id, so we should also be able to backfill project_id from ci_builds.project_id
ci_job_variables
How to do it
Once the designation is set in the db/docs/*.yml config file, automation will handle the rest.
ASK: Populate the db/docs file and then the automation will handle backfilling of the records. This is assuming the sharding key which it's backfilling is non-nullable. As is the case for ci_builds_needs. As it's backfilling from build_id which is already non-nullable.
✓
6 of 6 checklist items completed
· Edited by
Vlad Wolanyk
drew stachonchanged the descriptionCompare with previous version
changed the description
drew stachonchanged title from Add desired sharding keys with backfill from ci_builds.project_id to Add desired sharding keys with backfill from ci_builds.project_id (6 tables)
changed title from Add desired sharding keys with backfill from ci_builds.project_id to Add desired sharding keys with backfill from ci_builds.project_id (6 tables)
drew stachonchanged title from Add desired sharding keys with backfill from ci_builds.project_id (6 tables) to XS Add desired sharding keys with backfill from ci_builds.project_id (6 tables)
changed title from Add desired sharding keys with backfill from ci_builds.project_id (6 tables) to XS Add desired sharding keys with backfill from ci_builds.project_id (6 tables)
drew stachonchanged title from XS Add desired sharding keys with backfill from ci_builds.project_id (6 tables) to XS Add desired sharding key YAML config with backfill from ci_builds.project_id (6 tables)
changed title from XS Add desired sharding keys with backfill from ci_builds.project_id (6 tables) to XS Add desired sharding key YAML config with backfill from ci_builds.project_id (6 tables)
@lauraXD - it sounds like the plan we discussed in !152828 (comment 1920559746) is still the same then for ci_build_needs though the MR will come off of this issue.
@arturoherrero - is the due date still July 31st? I ask because this issue says July 12th.
@marknuzzo This issue is a different scope though, it seems the intention here is to backfill some tables? But this is a 3 so it should probably be added to the board though, given that it's no longer a quick issue
@marknuzzo@arturoherrero I had set it to line up with the code cut-off date of the milestone where we planned to finish so that it would roll-up to the epic's timeline.
@arturoherrero I had missed updating the timeline for this issue when I did the other dependencies. I've now updated it. There is a chance it could be done in 17.3 rather than 17.4 but given the need to backfill and the chain of dependencies I went with 17.4.
@marknuzzo I assume they can be done in parallel once the appropriate fields are made non-nullable and data has been corrected allowing for backfills to happen.
Can you confirm @mfanGitLab?
The actual change needed would be relatively small 1 weight . And can be done once we have backfilled ci_builds, which is in progress and scheduled for this milestone.
@marknuzzo Since I am focusing on the policy for 17.2, I think we should add this to the board like I mentioned so the next available engineer can pick this up. I was selected at random by a bot but anyone should be able to complete it.
@carolinesimpson - so that the ci_builds_needs work shows up in the PA board, I'm going to create a related issue and reference it in the description above. Work will still happen at the appropriate time but for visibility sake, I want to make sure it can be easily found.
The actual change needed would be relatively small 1 weight . And can be done once we have backfilled ci_builds, which is in progress and scheduled for this milestone.
@mfanGitLab is the ask here in the issue to do the backfill? It's not super clear to me with the description above.
@lauraXD for the cells sharding key work, most of the time we only need to populate the db/docs file and then the automation will handle backfilling of the records. This is assuming the sharding key which it's backfilling is non-nullable. As is the case for ci_builds_needs. As it's backfilling from build_id which is already non-nullable.
@arturoherrero I had the same concerns, and given various PTO on the team, we did not assign any others to take this on given the additional context that Max and Drew have while they refined the issue. Once that is completed, we can also "divide and conquer" some of the remaining issues across Verify.
My understanding is that we have until roughly end of Q3 to complete all the work for Cells 1.0, so 17.4 is still within that timeframe (not that we want to cut it that close).
@cheryl.li Cell 1.0 due date is at the end of Q3, but setting up the sharding key is one of the OKRs we have for Q2. It can require different milestones as we have to work with database migrations, but we may need to do more work after setting the sharding keys. Let's try to make progress on each milestone as we have different dependencies, so we make sure there are no delays as we approach the due date.
Chun Du @Mark Nuzzo- that’s a good question and the time is still ambiguous. We know phase 3 is planned to deploy in production by Nov 22. My gut feeling is that Phase 8 will be early next year. @gerir WDYT?
@mfanGitLab I had a similar question to Laura above, does this issue block all the issue where we have to add project_id and backfill from ci_builds.project_id?
ci_build_pending_states
ci_build_trace_chunks
ci_build_trace_metadata
ci_job_variables (using job_id)
If so, we should probably create individual issues to track that effort (which I can do).
@cheryl.li , yes unfortunately it does. If there's NULLproject_ids in ci_builds. Then the backfills won't work properly because it's using the same project_ids.