Database Group - 14.10 Planning
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.
Capacity
No noteworthy PTO is expected that could impact the capacity for %14.10
Boards
Planning
Top Priorities for 14.10
Batched Background Migrations - General Availability
Who: @dfrazao-gitlab, @pbair, @krasio
Our plan is to make the batched background migrations framework the standard for all data related operations and replace the existing way of performing them. We anticipate that this will be a major step forward for the stability and availability of GitLab instances of any size, while making most background data operations complete considerably faster in most cases.
Priority topics for %14.10:
- Support multiple databases for batched background migrations - any pending issues not completed in %14.9
- Split timed out batched background jobs and retry
- Limit or disable batch size optimization for individual migrations
- Add developer docs for the batched background migration framework
- Batch optimizer should key off affected rows
Throttling mechanism for large data changes
Who: @pbair, @abrandl
We want to implement a generic throttling mechanism for large data changes that will monitor the health of the Database for various signals (leading indicators) and react to problems by throttling or even pausing the execution of the updates.
Our plan is to do so by extending the batched background migrations auto-tuning layer to monitor for said signals and actively react by further adjusting the batch sizes of scheduled jobs.
Priority topics for %14.10:
Automated database testing using production clones
Who: @stomlinson @mattkasa
We have completed our coverage of all regular and post migrations and we are now focusing on also testing background migrations.
In %14.9 we have completed our initial support for regular background migrations.
In %14.10 we plan to extend our support to batched background migrations as well and add feedback back to the relevant MRs.
Priority topics for %14.10:
- Instrument batched background migrations
- Display results of instrumenting background migrations
- Estimate length of capped batched background migrations
- Test migrations against background scheduling logic
Database Data Dictionary
Who: @mattkasa
We want to add the ability to label tables by the group that owns them. With more than 400 tables in GitLab's schema, that will provide us with the ability to more efficiently respond to database related incidents or bug / regression reports by finding the group that has the domain knowledge to address the problem in a timely fashion.
This work was deprioritized in previous milestones in favor of the database decomposition and other priorities that the group was working on. We plan to complete our first iteration in %14.10 so that we can then allow each group to label the tables that they own.
Priority topics for %14.10:
- Data Dictionary: Identify objects to be added to the data dictionary and the format of the dictionary content
- Data Dictionary: Write developer docs for data dictionary schema and process
Kickoff Video
Activity
added Category:Database devopssystems groupdatabase sectioncore platform labels
added Engineering Allocation label
@gitlab-bot, please ensure the required labels are present for Engineering Allocation measurements:- An
~Eng-Consumer::*label - An
~Eng-Producer::*label - A
~priority::*label - A
~severity::*label when the type is ~"bug"
- An
Thanks for putting this together, this looks right to me so far @iroussos!
@gitlab-org/database-team I have updated the planning issue for %14.10 following our discussions on priorities. Do you mind quickly checking and telling me if I missed or misrepresented anything? Thank you!
This is a turning point for our Group with much more capacity available
The successful release of the GA of Batched Background Migrations will then allow us to unblock and enable work on gitlab-org&6211 in a few milestones, which is one of the most important initiatives for the availability and performance of GitLab.
Thanks @iroussos! I really appreciate the content and structure of this.
I am particularly excited about the GA of Batched Background migrations!
-
I have updated the description to include the kickoff video for 14.10.
/cc @fzimmer @nhxnguyen
mentioned in issue gitlab-com/Product#3917 (closed)
changed milestone to %14.10
-
FYI @aqualls Database Group's planning issue for 14.10.
We expect to need a technical review on MRs related to:
Excellent. I've read through all three, and they're all labeled with documentation + Technical Writing. Appreciate the planning time!
mentioned in merge request gitlab-com/www-gitlab-com!100753 (merged)
mentioned in issue gitlab-org/technical-writing#569
added devopsdata stores label and removed devopssystems label
added groupdatabase frameworks label and removed groupdatabase [DEPRECATED] label