`Planned for execution starting at 2AM Pacific Daylight Time (0900-1100 UTC), Sunday May 10, 2020` **Overview** The version of Postgres being used by GitLab.com is 3 major versions behind the latest release version of Postgres. This limits the usage of newer Postgres features by Development. This project is intended to bring the version of Postgres being used at GitLab.com to more recent version. **Current Situation** Presently Postgres 9.6 is in use on GitLab.com. EOL for PG 9.6 is November 11, 2021. Development is limited to the PG feature set of 9.6 and would like to take advantage of features present in version 11+ **Desired Outcome** Migration of the current database at GitLab.com from version 9.6 to 11.7. This includes full migration of data without data loss and downtime of less than one hour. **Milestones** To execute the postgresql migration we should execute the following steps: | Milestone | Summary | Related artifacts | ECD | Status | | ------ | ------ | ------ | ------ | ------ | | Create test environment to test the pg upgrade process | To execute all the tests needed and do not be disruptive constantly on staging we need to create test environment of the postgresql upgrade | https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9088 | 2020-02-21 | ~"status::on-track" | | Create and publish all Chef cookbooks needed for PG 11 DB environments | In order to deploy and maintain the environments for this upgrade, they needed to be implemented within our Chef repo | https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/2923 | 2020-04-07 | ~"status::on-track" | | Create and publish blueprint | Decide on an approach to the Postgresql database upgrade by exploring several proposals | https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9084 | 2020-04-10 | ~"status::on-track" | | Create and publish a design document | With an approach decided on, detail work needed to be put in place to accomplish the upgrade | https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9090 | 2020-04-10 | ~"status::on-track" | | Test migration to PG 11.7 in Staging | To better assess what additional risk we may see in Production, a test will be conducted at least once in Staging. | https://ops.gitlab.net/gitlab-com/gl-infra/db-migration/-/issues/11 | 2020-04-22 | ~"status::on-track" | | execute database benchmarking on Postgresql 11 | Benchmark queries from actual production on PG 11 | https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9545 | 2020-04-24 | ~"status::on-track" | | Execute migration in Production | Execute migration in Production. | TBD | 2020-04-24 | ~"status::at-risk" | | Execute customer communication plan | - | https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9828 | TBD | ~"status::at-risk" | **Risks** | Risk | Details | Realized? | | ------ | ------ | ------ | | 1. Testing indicates desired downtime is unobtainable | presently the downtime scheduled for this maintenance is 2 hours in duration. Tests of the migration process could indicate that the estimated downtime exceeds this window. | Undetermined until testing is complete | | 2. Rollback procedures determined to not allow for a rollback without data loss/corruption | As part of preparations for this upgrade, rollback procedures will be tested and the database will be compared to it's pre-upgrade state. | Undetermined until testing is complete | | 3. Final test in production not ready - to have a very accurate estimation of the downtime | Our staging End to End test will reveal more about ways to improve our downtime for this change, and have a final estimation. That test can only happen close to 23rd/24th April | Yes (for now) | | 4. PG 11.7 is slated to be in the Omnibus package for 13.0 (May release) as the minimum PG version requirement - impact in times? | Working with Program Mgmt to manage this according to product/customer commitments | Undetermined until we advise a date for the final change in prod | | 5. Plenty of differences between our DB in Production vs Staging | Staging has GEO and few other PG packages that we don't run in production | Not yet since we still have time to mitigate this | | 6. Benchmarks of 11.7 with the current DB load | These benchmark works are slightly delayed - Ongres might have too much on | Yes for now, trying to improve things till May 1st | **Projects Dependent on This Work** 1. [Add support for PostgreSQL 11 in Omnibus package](https://gitlab.com/groups/gitlab-org/-/epics/2414)