Consider increasing the Helm deploy command timeout for Review Apps as migration job duration can take up to 10 minutes
I noticed by looking at some migrations jobs that their duration varies from 4 minutes to 12 minutes at worst:
Start time | Completion time | Duration |
---|---|---|
Jul 23, 2019, 4:27:09 PM | Jul 23, 2019, 4:39:08 PM | 11:59 |
Jul 22, 2019, 9:02:21 PM | Jul 22, 2019, 9:09:34 PM | 07:13 |
Jul 22, 2019, 2:11:04 PM | Jul 22, 2019, 2:17:17 PM | 06:13 |
Jul 22, 2019, 1:23:09 AM | Jul 22, 2019, 1:27:01 AM | 03:52 |
Jul 22, 2019, 1:16:35 AM | Jul 22, 2019, 1:22:22 AM | 05:47 |
Jul 22, 2019, 3:21:45 AM | Jul 22, 2019, 3:25:32 AM | 03:47 |
Jun 4, 2019, 8:34:08 AM | Jun 4, 2019, 8:40:24 AM | 06:16 |
Jul 23, 2019, 1:17:41 PM | Jul 23, 2019, 1:21:22 PM | 03:41 |
Jul 22, 2019, 11:53:50 AM | Jul 22, 2019, 12:03:34 PM | 09:44 |
As we can see, in many cases, the migrations can take up to 5 or 6 minutes, and sometimes even up to more than 10 minutes. Since the unicorn and sidekiq pods are depending on this migrations job to be done, we're likely to hit the Helm deploy timeout of 10 minutes.
I suggest increasing this timeout to 15 minutes for now, and then identify parts of the migrations job than can be optimized (e.g. https://gitlab.com/charts/gitlab/issues/89).
Edited by Rémy Coutable