Consider creating release instance for release candidates
Problem Statement
Incident: production#15891 (closed)
Preprod has grown its purpose. It is currently considered a lower environment where some configuration testings start and are finalized prior to graduating the change to staging. While this is probably fine, it does add to the number of environments for which all engineers must be acquainted with when making infrastructure changes. teamDelivery leverages this instance during our monthly release as a final stop gap to test Release Candidates prior to tagging our final package that would be released to end users. Recently, noting the above linked incident, the mixture of Infrastructure team members toying with the preprod environment, has caused a problem where PoC's now interfere with regularly occurring needs for teamDelivery. This has the potential to cause unnecessary pain when the infrastructure is not configured to support those needs.
Solution
After discussions we have decided to go ahead with Option B, and follow up in the future with option A. Decision logged on &1070
Option B
Let's refresh preprod and bring it back to a desired working state. This would entail removing any custom work performed to this infrastructure, decomission any PoC work that may be in progress, and removing anything unnecessary which should lower the barrier of entry for management as a whole. This also forces teamDelivery to fully own both the infra and application entirely. We then set a rule where any future infrastructure work that we would normally target preprod today, must be either accomplished on staging, or some other environment that is setup and managed by whatever team.
All other options are detailed below:
Option A
Consider making use of the newly introduced release environments project to create a long standing release instance who receives Release Candidates. There will be some differences in infrastructure compared to preprod today, but with the goal of testing an installation, hopefully this would suffice. This suggested end result would allow us to sunset the current preprod infrastructure and enable any Infrastructure team to leverage this to their desire without interfering with the monthly release procedures. An immediate downside is that we do not have ominbus support in release environments.
Option B
Let's refresh preprod and bring it back to a desired working state. This would entail removing any custom work performed to this infrastructure, decomission any PoC work that may be in progress, and removing anything unnecessary which should lower the barrier of entry for management as a whole. This also forces teamDelivery to fully own both the infra and application entirely. We then set a rule where any future infrastructure work that we would normally target preprod today, must be either accomplished on staging, or some other environment that is setup and managed by whatever team.
Option C
Leave preprod to the wild is it currently stands but modify our release procedures to include a check to validate that preprod is in a working state before and after teamDelivery's needs during monthly releases. This has the least barrier to entry with the downside that we increase the mental load during releases.
Let's leverage this issue for discussion, bringing other potential options, and create issues to address action items.