Run scheduled pipelines on all our stable branches
At present, we run a pipeline on every:
- Merge request
- Merge commit
- Tag
However, our pipelines have a great number of external dependencies. When there's not activity on a given stable branch for some time, it's very easy for the next pipeline created on it to fail because of changes to external components since the last time a pipeline ran. Often, we notice this on master, but forget to fix up the maintained stable branches.
This is a regular feature on gitlab-workhorse and gitlab-pages stables branches, and sometimes it happens on gitlab-ce / gitlab-ee as well. When we need to get a security release out quickly, this often leads to delays as we work out why the pipeline has begun to fail.
I propose we set up scheduled pipelines for all our maintained stable branches - and master - on all our projects. We should run the pipeline once a day, and treat failures in a similar way to ~"master:broken"
failures.
In particular, we should have these scheduled pipelines on :
-
gitaly -
gitlab-ce -
gitlab-ee -
gitlab-elasticsearch-indexer -
gitlab-runner (do we have stable versions of this @steveazz ?) -
gitlab-shell -
gitlab-workhorse -
...
Challenges
Ideally, we'd be able to say "only schedule a pipeline if another pipeline hasn't already happened within 24 hours for this branch", but I don't think that's a product feature at present. Maybe we could add it?
Actually maintaining the list of stable branches to schedule across is going to be painful, since it changes every month. Again, we don't have a product feature for this - being able to say "matches this pattern, and created < X months ago" would be great.
cc @gitlab-org/maintainers/rails-backend @jacobvosmaer-gitlab @nolith