Investigate why cross project Review Apps don't seem to be working properly
We've identified the issue to be a race condition of the pipelines the very first time the manual action is run. Copying from https://gitlab.com/gitlab-com/gitlab-docs/issues/154#note_61672645.
The MR workflow is:
- CE/EE/Omnibys/Runner MR is created
- Manual action to build the docs website begins
-
Script is run which in turn does the following:
-
create_remote_branch
- For the very first time, it creates the remote branch in the docs website in order for the review app to be created. That automatically runs a pipeline and deploys the docs site. No env is used, thus only master branches are pulled. The token that is used is one created for GitLab Docs bot which is added as a Developer to the gitlab-docs project. If the branch is already created, it skips the whole process and no pipeline is run. -
trigger_pipeline
- A pipeline is triggered using the previously created branch, theCI_JOB_TOKEN
and the variables for pulling the right branches.
-
The problem is that there's a race condition, and as described above, 3.1 runs after 3.2 which uses the master branches and thus the review app gets overwritten. That's why running the manual action a second time corrects the problem, since only 3.2 is run (the branch is already created, so 3.1 is skipped).
For example, you can see those two actions in the two different deployments in https://gitlab.com/gitlab-com/gitlab-docs/environments/238533. Notice that one is created by the Docs bot and the other by me. This happens because of the script's functions as described above in 3.1 and 3.2 respectively. We can probably eliminate the need to have a separate token and always use CI_JOB_TOKEN
, but that's for another issue.
Possible solutions:
- Do not run any pipeline when the branch is created for the first time. The quickest way is probably by using
except: api
. Otherwise, we'd need to find the pipeline when the branch is created and cancel it. - Make sure 3.2 is run after 3.1 by putting a
sleep
in the script
Old description
- When we deploy the job to review apps from CE or EE MRs, the review app link doesn't show the latest version of the doc
- When we add more commits and click the manual job again, review apps doesn't seem to update the doc
Examples of weird behavior:
- https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3833#note_54189247
- https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/16372#note_54309419
- https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15411#note_47982019
- https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/16555#note_55142564
- https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4153#note_55157854