Database Group - 15.6 Planning
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.
Capacity
In %15.6 the Database group will be at 40% capacity due to continuing capacity issues, onboarding, and planned vacations. Also, given the recent departure of @iroussos, PM duties are performed on a life support basis.
Boards
Planning
We will keep our focus on the initiatives that affect the most the availability and reliability of GitLab.com and self managed instances.
Top Priorities for 15.6
Batched Background Migrations
Who: @dfrazao-gitlab, @krasio, @jon_jenkins
What: Continue our work to provide a stable, reliable framework for executing the most complex, error prone database operations. Throttling mechanics are being included in this scope for now.
Why: This final stages of this effort will help us accomplish some long term goals. First, extra safety on gitlab.com migrations from the automated throttling mechancis. Second, developer observability via chatops will reduce load on database experts by allowing members of the team to self service information about their background migrations in production. The additional batch handling work will also help our self managed customers execute migrations more reliably.
Priority topics for %15.6:
- Handling Batches:
- Throttling Mechanics:
- Identify migration helpers that use non-batched... (gitlab-org/gitlab#366087 - closed)
- Filter by gitlab_schema when displaying batched... (gitlab-org/gitlab#367333 - closed) typebug
CI Partitioning Support
Who: @stomlinson @mattkasa
What: We're starting work aiding the CI groups to partition their tables. The two main aids we're providing are helpers for partitioning tasks, and the initial phase of Single query testing which provides assistance in identifying partitioning keys.
Why: CI tables are very large and partitioning them will help a lot to control some of the table sizes we're experiencing. In addition to alleviating these specific tables, we're also creating a blueprint for further teams to leverage partitioning in the future.
Priority topics for %15.6:
- Partitioning Tooling: Add helper to update fore... (gitlab-org/gitlab#374019 - closed)
- Partitioning Tooling: Add helper for renaming i... (gitlab-org/gitlab#370274 - closed)
- TBD - Still wrapping up topics from 15.5, and we'll designate additional issues as the arrive. We've committed to setting aside time for this effort.
Database query analysis
Who: @mattkasa @jon_jenkins
What: We're extracting queries and identifying added or changed queries from a merge request. Long term, we will provide analysis on these extracted queries to ensure they meet our guidelines.
Why: We're hoping to help people identify their new queries early and give database reviewers a boost by extracting the sql automatically. Additionally, this should aid partitioning efforts by helping teams identify queries that do not contain the partitioning key for their newly partitioned tables.
Priority topics for %15.6:
- Query reporting: Artifact query reporter log (gitlab-org/gitlab#377897 - closed)
- Query Reporting: Deduplicate recorded queries (gitlab-org/gitlab#368741 - closed)
High Severity / Priority issues
Automatic reindexing can block vacuum progress (gitlab-org/gitlab#377720 - closed)
Who: @stomlinson
During a recent incident we identified that automated reindexing blocks auto-vacuum and can lead to incidents. We're investigating possible options to prevent this behavior and avoid future issues.