Scheduling issue for the 12.9 release
Product Priorities
The product work below is listed in priority order. Most of these buckets of work will still take multiple releases to complete, but I have added some progress goals to work towards in the 12.9 milestone.
PostgreSQL updates
The top priority for %12.9 is to stay on track with PostgreSQL updates. The overall plan for adding PG11 is documented in gitlab-org&2184 (closed). In %12.8 we added PostgreSQL 11 to Omnibus. The goal was to also test HA, Geo, and upgrading with PG11 in %12.8 (WIP at the time of writing this). If necessary, we should continue making improvements to the upgrade experience in the %12.9 milestone in preparation for making PG 11 opt out in %12.10.
To enable testing of index size optimizations, we will add PG 12 to our CI testing to make sure it works. This does not involve adding PG 12 to the Omnibus package, which is currently scheduled for 13.3 (depends on adding Patroni).
FYI @marin @fzimmer @craig-gomes @joshlambert @tpazitny
In %12.9 we will test using PG11 with the GitLab chart. (gitlab-org/charts/gitlab#1848 (closed) and gitlab-org/charts/gitlab#1849 (closed)). We do not support using a bundled PostgreSQL in production for Helm chart installs so there are currently no plans to perform Geo and HA testing of PG11 with the chart.
We should reserve some capacity in %12.9 for responding to feedback we get from internal or external sources regarding issues with using GitLab with PG11, or with upgrading. It would be great if we could find customers that have a test environment for their GitLab instance and are willing to upgrade to PG11 during the %12.9 milestone. We plan to do thorough upgrade testing in %12.8 (gitlab-org/omnibus-gitlab#4970 (closed) and gitlab-org/omnibus-gitlab#4975 (closed)), but it would be helpful to have some real world upgrades completed before making PG11 opt out, in case additional issues arise that we can resolve. https://gitlab.lightning.force.com/lightning/r/Account/00161000010fYg0AAE/view mentioned that they might upgrade to PG 11 before it becomes a minimum requirement. They like to upgrade their database outside of their monthly GitLab upgrade so they can determine if issues are related to a bug in GitLab or if they are related to the database update. I'm not sure if they have a test environment. (cc @bma perhaps this is something you could raise?)
In a recent interview I did with https://gitlab.lightning.force.com/lightning/r/Account/00161000004bZPDAA2/view they mentioned they took a downtime hit of a couple of hours to cut over from PG9.6 to PG10 and would like to avoid that for the PG11 update. We should get more details on that and see if there is further work we can do in %12.9 to reduce the likelihood of downtime (cc @jrreid). There is also the issue raised by @fzimmer in gitlab-org&2184 (comment 275761504) that might need investigation.
Adding support for Patroni in Ominibus
We have recently learned that repmgr 3 is not included in the support matrix for PostgreSQL 11, and both repmgr 3 and 4 don't work with PostgreSQL 12. After reviewing the work that would be required to update repmgr to versions 4 or 5, we have decided to prioritize switching to Patroni. The current version of Patroni supports PostgreSQL 12. There have been two reports of brain split issues with repmgr that we don't think have been resolved in more recent versions.
Initially, Patroni will be available as an optional failover solution and will be shipped alongside repmgr. At this stage we don't expect to remove repmgr until GitLab 14.0. Given our plans to support PostgreSQL 11 and 12, we need to add support for Patroni as soon as possible and make this a high priority for the team. We cannot add support for PostgreSQL 12 until we have added Patroni.
See gitlab-org&2588 (closed) for more details.
Provisioner tooling
The provisioner tool supports creating nodes in GCP and installing GitLab on them. It creates enough nodes for a Geo primary and secondary with HA as described in gitlab-org/gitlab-orchestrator#25 (closed). In %12.8 we are adding the ability to tag nodes as being part of either the primary or secondary cluster. In %12.9 we should complete any remaining work to launch a fully functioning instance of GitLab with HA and Geo working. We should also coordinate with the Geo team to get feedback on the functionality and transition to all using and contributing to the same tool.
cc @rmarshall @fzimmer @rnienaber
Helm Chart enhancements
There are a couple of customers in the process of migrating from Omnibus Docker to using the Helm chart to deploy in EKS. There are some enhancements we could make that would allow them to comply with internal security requirements, while also providing all of our Helm chart users with more options when deploying in EKS. Now that GitLab Chart 3.0 has been released and we have capacity to focus on the Helm chart maturity epic, these enhancements would be some nice wins for our customers:
Use IAM roles for service accounts to access external storage
Support using KMS-encrypted S3 storage
cc @ronkoster and @chloe
Handling secrets
There are passwords used by GitLab that are stored in plain text in configuration files. This is a security concern for customers running self-managed instances. We will be prioritizing a better solution for storing passwords over the coming releases. As a start, there may be some further research required to understand the problem and document the pros and cons of various solutions. This work needs to be broken down into issues, but the issue that describes the problem is gitlab-org/omnibus-gitlab#2183 (closed). ~"group::release management" is also working on this so we will need to coordinate with them.
cc @jmeshell
Update: given the number of priorities we have for 12.9 and the discovered issue that requires us to prioritize Patroni, we will be postponing the work on secrets until 12.10
AWS improvements
Throughout 2020 we will prioritize improving the experience of deploying GitLab on AWS. A large portion of our self-managed customers are deploying their instances on AWS and we should make that process as seamless and efficient as possible to reduce the time it takes for customers to get up and running. A handful of these improvements are targeted for %12.9.
Premium listing The Alliances team has requested a Premium listing for GitLab, similar to the Ultimate listing. When making a private offer through AWS, customers expect that the listing used for the private offer matches the tier of GitLab that they are purchasing. cc @pete_goldberg
Automate release of AMIs The process for updating marketplace AMIs is almost fully automated. There is one manual step left. In %12.9, let's finish automating this process so we have one less thing to think about at release time and our customers on AWS can access the latest version of GitLab through the marketplace on the 22nd of each month.
Improve documentation for HA on AWS. There are a number of issues open that suggest improvements to the documentation for installing GitLab on AWS. These include gitlab-org/gitlab#196728 (closed), gitlab-org&912, and gitlab-org/gitlab#199234 (closed). In gitlab-org/gitlab#196728 (closed), @collen kindly offered to take the lead on making some incremental improvements to the documentation in coordination with @axil. These improvements will likely require review by groupdistribution. We should reserve some capacity in %12.9 to review and merge changes as they become available.
Deliverable Board
Issues listed on this board have already been reviewed and scheduled for the upcoming release.
Priority board
Priority board categorizes the importance of all open issues, regardless of milestone. It's useful when looking for issues to consider for scheduling. Board includes issues from Omnibus, Charts, Team tasks projects.
Scheduling board
For Scheduling board is a work board used specifically during the scheduling process to provide an overview of candidates and get an idea of the next release before assigning the milestone to issues. Board includes issues from Omnibus, Charts, Team tasks projects.
Status
-
Collect initial candidate issues -
Issues added to the Deliverable board via label -
Scheduling boards sorted/arranged by importance -
Scheduling boards contain candidates from both Engineering and Product -
Scheduling boards scoped to a reasonable amount of work for the release -
Managers schedule the agreed issues for the release