Database Group - 15.10 Planning
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.
Capacity
In %15.10 the Database group will be at 70% capacity while two out of our seven team members proceed with their onboarding and ramp up. We expect to be at fully capacity in ~1 more milestones.
Boards
Planning
We will keep our focus on the initiatives that affect the most the availability and reliability of GitLab.com and self managed instances.
15.10
Top Priorities for Update Primary Keys to BigInt
Who: @krasio
@dfrazao-gitlab
@jon_jenkins
We recently discovered that our alerting about Primary key exhaustion was broken.
Historically, GitLab database tables used 4-byte integers as primary keys. Since late-2018, we have transitioned to using 8-byte integers as primary keys for all new tables. There are currently 4 legacy tables with > 50% primary key saturation.
We are particularly concerned by merge_request_metrics
, given its recent growth rate. It is the highest risk, at 70% saturation. We are prioritizing work on all four legacy tables; we estimate this will require ~3 milestones to mitigate. We will tackle three of these tables directly, while supporting a self-service approach for the ci_build_needs
table.
Priority topics for %15.10:
- Migrate merge_request_metrics.id to bigint (gitlab-org/gitlab#389516 - closed)
- Migrate sent_notifications.id to bigint (gitlab-org/gitlab#389344 - closed)
- Swap integer and bigint columns for notes and d... (gitlab-org/gitlab#392815 - closed)
Batched Background Migrations
Who: @krasio
@praba.m7n
@dfrazao-gitlab
@l.rosa
What: Continue our work to provide a stable, reliable framework for executing the most complex, error prone database operations. Throttling mechanics are being included in this scope for now.
Why: This final stages of this effort will help us accomplish some long term goals. First, extra safety on gitlab.com migrations from the automated throttling mechancis. Second, developer observability via chatops will reduce load on database experts by allowing members of the team to self service information about their background migrations in production. The additional batch handling work will also help our self managed customers execute migrations more reliably.
Progress update: While our overall goal is to minimize downtime caused by large data migrations, we believe time estimates for large migrations will help reduce customer uncertainty when planning GitLab upgrades. We have developed a mechanism to measure migration runtimes in test and we will observe in %15.10 whether we those estimates can be correlated to production environments. We're are also rolling out improvements to migration parallelization.
Priority topics for %15.10:
- Pause migration when patroni apdex dropped belo... (gitlab-org/gitlab#357250 - closed)
- Estimate length of capped batched background mi... (gitlab-com-database-testing#53 - closed)
- Provide documentation to help internal+external users understand what a migration is doing
Automated Database Testing
Who: @mattkasa
@jon_jenkins
What: We're extracting queries and identifying added or changed queries from a merge request. Long term, we will provide analysis on these extracted queries to ensure they meet our guidelines.
Why: We're hoping to help people identify their new queries early and give database reviewers a boost by extracting the sql automatically. Additionally, this should aid partitioning efforts by helping teams identify queries that do not contain the partitioning key for their newly partitioned tables.
Progress update: We have prototyped a query-interceptor, which will allow us to build a Rails- and postgres- agnostic solution. As we continue to build towards a GitLab-focussed use case, this prototype provides us with optionality if we choose to productize query analysis in the future.
Priority topics for %15.10:
- Develop Architectural Blueprint for Testing Que... (gitlab-org/gitlab#390295 - closed)
- Migration testing pipeline throws errors when i... (gitlab-com-database-testing#66 - closed) typebug
Identify Schema Inconsistencies
Who: @dfrazao-gitlab
@l.rosa
What: As a first phase of Schema validation framework (gitlab-org&9841), we're building tasks to identify if and how a schema has diverged from the reference.
Progress update: We have shipped a mechanism to identify index divergence. We are working to surface this information via usage ping this data back to GitLab
Priority topics for %15.10:
- Service ping integration - Indexes (gitlab-org/gitlab#390273 - closed)
- Implement a rake task to collect schema inconsi... (gitlab-org/gitlab#390719 - closed)
High Severity / Focus Items
Remove old migrations in 15 (gitlab-org/gitlab#370640 - closed)
Who: @jon_jenkins
What: In every major release (once a year), we like to remove old migrations to clean up the repository. It's been a while since we did this, but we should be able to drop migrations introduced before 15.4 (Last required stop). We have begun the process of squashing in 15.8 but there will be additional work items rolling into 15.10. We estimate we are currently ~1/3 of the way complete.
CI Partitioning Support
Who: @stomlinson
What: We're starting work aiding the CI groups to partition their tables. The two main aids we're providing are helpers for partitioning tasks, and the initial phase of Single query testing which provides assistance in identifying partitioning keys.
Why: CI tables are very large (our target table size is ~10 GB, CI are >1 TB) and partitioning them will help a lot to control some of the table sizes we're experiencing. In addition to alleviating these specific tables, we're also creating a blueprint for further teams to leverage partitioning in the future.
Progress update: There are 6 total tables of concern. We have attached a zero-th partition to one table and are now working to implement it on the ci_builds
table as it is the largest of the six. Zero-th partitions are required to support a partitioning framework. Actually splitting CI tables into multiple partitions is still to come.
Priority topics for %15.10:
- Continue to support efforts to partition the
ci_builds
table - ConvertTableToFirstListPartition helper does no... (gitlab-org/gitlab#383310 - closed)