Provision a Nightly Staging Environment
GitLab follows the mantra "integrate early, integrate often", by building nightly builds on dev.gitlab.org
. This is good practice. The problem is that our nightly built environment is also critical infrastructure. This leads to teams being risk averse in testing early.
For example, the Gitaly team are reluctant to enable feature flags on dev.gitlab.org
for fear of causing instability and creating downstream knock-on effects to other teams.
GitLab also has the staging environment, staging.gitlab.com
, but this is only deployed to post-release-cut, so is neither early, nor often.
With our current staging environment, it can sometimes be over a month between a change being committed to master and it being tested in a full-integrated environment. This has a negative effect on our throughput.
I propose we setup a new environment, call it nightly.gitlab.com
with the following characteristics:
-
Like
dev.gitlab.org
it is rebuilt from master every day -
Unlike
dev.gitlab.org
, but like staging, it is completely throw-away and non-critical -
To enable thorough testing and debugging, either developers can either obtain shell access, OR cog access, and is connected to monitoring systems like ELK, Sentry, Prometheus and Influx (I prefer the latter but this would take more effort)
-
Unlike
dev.gitlab.org
, but like staging, it uses a similar topology to production - ie, gitaly/NFS server separate from worker server -
Preferably, the instances should be a very underpowered (ie cheap) so that it's easy to conduct stress-testing.
With an environment like this, we would be able to start test our migrations a day or two after they've been merged into master, hopefully improving our productivity.
@omame is this something that your terraform work would make easier to manage?
cc @sitschner @pcarranza @gl-gitaly