dealing with post-deploy migrations which may cause problems for rollback if they aren't delayed
Post-deploy migrations may delete data. Once the data is deleted it will be difficult to restore the state of the db if we need to rollback to the previous version.
upgrade (current)
graph LR;
a>prepare] ==> b>migrations];
b ==> c>gitaly deploy];
subgraph fleet;
c -.- d>sidekiq];
c -.- d1>git];
c -.- d2>web];
c -.- d3>api];
c -.- d4>pages];
c -.- d5>registry];
c -.- d6>mailroom];
end
d ==> e>postdeploy migrations];
e ==> f>cleanup];
f ==> g>gitlab-qa];
Note that postdeploy migrations happen immediately after the fleet deploy. If we want to avoid the scenario where this causes problems I think we have the following options:
- Understand better if the post-deploy migrations will have non-reversable changes that can't be automated, this currently is a black box.
- If the post-deploy migrations do have non-reversable changes delay them until the next deployment
- Delay all post-deploy migrations to the next deployment
upgrade (proposed)
- Moves post-deploy migrations to the beginning of the pipeline so that we always run them for old release.
- This job would not be run on rollback.
graph LR;
a>prepare] ==> b>post-deploy migrations at the current version];
b ==> b1>migrations];
b1 ==> c>gitaly deploy];
subgraph fleet;
c -.- d>sidekiq];
c -.- d1>git];
c -.- d2>web];
c -.- d3>api];
c -.- d4>pages];
c -.- d5>registry];
c -.- d6>mailroom];
end
d ==> f>cleanup];
f ==> g>gitlab-qa];
This would mean that post-deploy migrations at the current version (whatever is running on the fleet when the pipeline starts) will be run as the first step.
Edited by John Jarvis