## Summary Decomposition is the chosen solution to scale GitLab's database to 10 million monthly active users. This approach relies on moving a feature tables into a separate logical database. We chose this approach because it is iterative and can be implemented in a shorter amount of time than sharding. The first iteration is rolling [Decomposition out to GitLab.com](https://gitlab.com/groups/gitlab-org/-/epics/6168). The second is to enable decomposition for all self-managed customers. ## Self-managed adoption proposal Internal Google Doc: [Issues Table by Stage](https://docs.google.com/document/d/1SzNMHWTWOiff688v2YFTNBZ59hxJtg2Tk1SGwf6hCpM/edit?usp=sharing) 1. [x] 15.X Instances can opt-into two connections 1. [x] [16.X Instances can opt-into decomposed](https://gitlab.com/gitlab-org/gitlab/-/issues/382026) 1. [x] [16.0 Change self-managed to two connections by default](https://gitlab.com/groups/gitlab-org/-/epics/9627). Will bring self-managed to Phase 4. Lots of parity 1. [ ] [17.0 Make two databases beta](https://gitlab.com/groups/gitlab-org/-/epics/11102) 1. [x] [16.8 Make migrating for existing installations opt-in (downtime)](https://gitlab.com/gitlab-org/gitlab/-/issues/368729) - likely via Rake task, or documentation how to clone database for others 1. [x] https://gitlab.com/gitlab-org/gitlab/-/issues/432391+ 1. [ ] [18.X Make two databases GA](https://gitlab.com/groups/gitlab-org/-/epics/13477) 1. [ ] dev.gitlab.org is decomposed 1. [ ] ops.gitlab.net is decomposed 1. [-] ~[16.0 ship new installations decomposed by default](https://gitlab.com/gitlab-org/gitlab/-/issues/368728). This make all new installations have more capacity and easier scaling out~ 1. [ ] [By 19.0 make migration for existing installations mandatory](https://gitlab.com/gitlab-org/gitlab/-/issues/368731). (downtime) - it is to evaluate between 16.0 and 17.0 if we want to do this 1. [ ] By 19.0, we allow schemas to diverge between decomposed databases. ### Testing #### How to spin up a VM to test Omnibus <details><summary>Instructions</summary> 1. Create a GCP project with https://about.gitlab.com/handbook/infrastructure-standards/realms/sandbox/#how-to-get-started 1. Go to your GCP project in console.cloud.google.com. Note it may take 5-10 for your new GCP project to appear in the projects list 1. Go to the Compute Engine section, enable the API (sorry I forgot the exact label) 1. Create a new VM instance. GitLab recommends 4GB memory and 2 CPU, so I chose `Machine type: n2-standard-4` (Omar: This is 4 VCPUs and 8 GB) 1. Download, and install `gcloud` locally. Select the SSH dropdown for the VM instance you want to connect to. Select the `gcloud SSH` option to get the command line to connect via SSH using the gcloud too. https://cloud.google.com/sdk/docs/install 1. Once on the VM, follow https://about.gitlab.com/install/#debian to install GitLab. Select a `http` external url like `http://gitlab-me.example.com` (as long as it's `http`, we will use SSH port forwarding) 1. On another terminal tab, connect using the same SSH command from 5 but with the additional flag `--ssh-flag="-L 8082:localhost:80"`. 1. Open `http://localhost:8082` on your browser. You should see a login screen for your new GitLab instance! 1. Various reconfiguration options are documented in https://docs.gitlab.com/omnibus/index.html#installation-and-configuration-using-omnibus-package </details> #### Charts: See https://gitlab.com/gitlab-org/gitlab/-/issues/385111#how-to-spin-up-a-vm-to-test-gitlab-charts-with-decomposed-external-database-setup <!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION --> *This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.* <!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->