Rails' jobs (i.e. jobs extending from `.rails-cache`) cache wasn't updated between 2021-01-27 and 2021-02-13, leading to an increase in duration
-
!52693 (merged) introduced a new cache for Rails' jobs (i.e. jobs extending from
.rails-cache
) that don't need the.go/pkg/mod/
folder - In the Version 2 of the MR (on 2021-01-27), the cache was populated: !52693 (diffs) => https://gitlab.com/gitlab-org/gitlab/-/jobs/990591107#L506
- The MR was merged on 2021-02-02, and the Rails' jobs were taking advantage of the cache: https://gitlab.com/gitlab-org/gitlab/-/jobs/1001109320#L43
- The MR didn't set the cache policy to
push
for theupdate-rails-cache
job, leading to a stale cache over time... - On 2021-02-04, the cache becoming stale,
bundle install
would take 90 seconds to finish: https://gitlab.com/gitlab-org/gitlab/-/jobs/1009349692#L66 (instead of 14 seconds normally: https://gitlab.com/gitlab-org/gitlab/-/jobs/1001109320#L65) - On 2021-02-11 (approximately two weeks after it was first populated), the cache wasn't there anymore, leading to
bundle install
taking between 500 and 600 seconds (8 to 10 minutes): https://gitlab.com/gitlab-org/gitlab/-/jobs/1022506160#L66 - On 2021-02-15, I've opened !54249 (merged) to fix the cache policy for the
update-rails-cache
job. The MR includes[UPDATE CACHE]
so the cache was re-populated - Soon after, jobs started to take advantage of the cache again: https://gitlab.com/gitlab-org/gitlab/-/jobs/1031425609#L44, finishing 10 minutes faster
😅
We should also think about adding an automatic detection of such regressions.
Related to #300192 (closed).
Edited by Rémy Coutable