Adding indexes to an existing GitLab column will need to be reviewed by the [database group](/handbook/engineering/data-engineering/data-frameworks/database-framework/) on a case-by-case basis.
Adding indexes to an existing GitLab column will need to be reviewed by the [database group](/handbook/engineering/data-engineering/database-excellence/database-frameworks/) on a case-by-case basis.
In all cases, we annotate the index with a PostgreSQL comment to indicate the index is JiHu-specific.
@@ -63,7 +63,7 @@ Optional: In order to reduce the overhead on GitLab.com, we may want to choose t
#### Additional objects
Other types of database objects, e.g. triggers, functions, extensions, etc., will need to be reviewed by the [database group](/handbook/engineering/data-engineering/data-frameworks/database-framework/) on a case-by-case basis.
Other types of database objects, e.g. triggers, functions, extensions, etc., will need to be reviewed by the [database group](/handbook/engineering/data-engineering/database-excellence/database-frameworks/) on a case-by-case basis.
### Data migrations
@@ -112,4 +112,4 @@ In order to increase the review efficiency when creating a merge request from a
### Questions?
Please engage with the [database group](/handbook/engineering/data-engineering/data-frameworks/database-framework/) for any questions and support.
Please engage with the [database group](/handbook/engineering/data-engineering/database-excellence/database-frameworks/) for any questions and support.
@@ -66,7 +66,7 @@ Although this working group is closing, our work to improve GitLab's database an
### Overview
Our current architecture relies, almost exclusively and by design, on a [single database](/handbook/engineering/data-engineering/data-frameworks/database-framework/doc/strategy/#single-data-store) to be the sole and absolute manager of data in terms of storage, consistency, and query results collation. Strictly speaking, we use a single **logical** database that is implemented across several **physical**[replicas](/handbook/engineering/architecture/practice/scalability/#example-postgres-current-state) to handle load demands on the database backend (with the single pseudo-exception of storing diffs on object storage). Regular analysis of database load, however, is showing that this approach is unsustainable, as the primary RW database server is experiencing high peaks that are approaching the limits of vertical scalability.
Our current architecture relies, almost exclusively and by design, on a [single database](/handbook/engineering/data-engineering/database-excellence/database-frameworks/doc/strategy/#single-data-store) to be the sole and absolute manager of data in terms of storage, consistency, and query results collation. Strictly speaking, we use a single **logical** database that is implemented across several **physical**[replicas](/handbook/engineering/architecture/practice/scalability/#example-postgres-current-state) to handle load demands on the database backend (with the single pseudo-exception of storing diffs on object storage). Regular analysis of database load, however, is showing that this approach is unsustainable, as the primary RW database server is experiencing high peaks that are approaching the limits of vertical scalability.
We explored [sharding](/handbook/company/working-groups/sharding/) last year and scoped it to the database layer. We concluded that while there are solutions available in the market, they did not fit our requirements, both in financial and product fit terms, as they would have forced us into a solution that was difficult (if not impossible) to ship as part of the product.
@@ -19,11 +19,11 @@ We have decided to close this Sharding focused working group and will open a Sca
Additionally, we have come to the consensus that sharding is not the desired approach for our long term scalability needs. This decision was informed through investigation, proofs of concept, research, interviews and various implementation proposals. Here's a brief list of items that helped to inform our decision to close this working group:
-[Introducing PostgreSQL table partitioning to GitLab's database](/handbook/engineering/data-engineering/data-frameworks/database-framework/doc/partitioning/)
-[Table partitioning: Issue group search as an example](/handbook/engineering/data-engineering/data-frameworks/database-framework/doc/issue-group-search-partitioning/)
-[Sharding with CitusDB](/handbook/engineering/data-engineering/data-frameworks/database-framework/doc/citus/)
-[Postgres: Sharding with foreign data wrappers and partitioning](/handbook/engineering/data-engineering/data-frameworks/database-framework/doc/fdw-sharding/)
-[Sharding GitLab by root namespace](/handbook/engineering/data-engineering/data-frameworks/database-framework/doc/root-namespace-sharding/)
-[Introducing PostgreSQL table partitioning to GitLab's database](/handbook/engineering/data-engineering/database-excellence/database-frameworks/doc/partitioning/)
-[Table partitioning: Issue group search as an example](/handbook/engineering/data-engineering/database-excellence/database-frameworks/doc/issue-group-search-partitioning/)
-[Sharding with CitusDB](/handbook/engineering/data-engineering/database-excellence/database-frameworks/doc/citus/)
-[Postgres: Sharding with foreign data wrappers and partitioning](/handbook/engineering/data-engineering/database-excellence/database-frameworks/doc/fdw-sharding/)
-[Sharding GitLab by root namespace](/handbook/engineering/data-engineering/database-excellence/database-frameworks/doc/root-namespace-sharding/)
-[Database Sharding Proposal: Sharding by Namespace](https://drive.google.com/drive/search?q=sharding%20by%20namespace)
@@ -87,7 +87,7 @@ While we continue to investigate Namespace vs. Tenant sharding, we can continue
- Measure results
- Investigation
-[x] [Explore CitusDB as a sharding solution](https://gitlab.com/gitlab-org/gitlab/issues/207833)
-[x] Document impact of [Sharding GitLab by root namespace](/handbook/engineering/data-engineering/data-frameworks/database-framework/doc/root-namespace-sharding/#data-access-pattern)
-[x] Document impact of [Sharding GitLab by root namespace](/handbook/engineering/data-engineering/database-excellence/database-frameworks/doc/root-namespace-sharding/#data-access-pattern)
- Identify application changes required by namespace sharding
- Review Tenant Sharding approach
- Identify application changes required by namespace sharding
@@ -128,7 +128,7 @@ The agenda doc can be found in our Google Drive when searching for "Sharding Wor
- Issue created to investigate issues where sharding would have helped
