Skip to content

Always delete release before review app deployment

Mark Lapierre requested to merge ml-review-app-delete-release-before-each-deploy into master

What does this MR do?

Changing certain parts of a kubernetes deployment is not possible via an update (e.g. !65118 (comment 615933864)) - the resource needs to be recreated.

This MR replaces the review-stop-failed-deployment job with review-delete-deployment, which deletes the release even if the previous deployment was successful. It does not delete the namespace. This will avoid deployment failures due to configuration changes, and may also help resolve transient deployment failures.

This shouldn't affect pipelines in other projects any differently than in this project, as it replaces an existing job.

This change will delete and redeploy an MR's review app in each pipeline. This may reduce performance on average if deletion takes longer than expected, or if deployment into an empty namespace after the previous release was deleted is slower than redeploying in place of an existing release.

However, review-stop currently takes about a minute to delete a namespace, and in a random sample of MRs review-deploy durations are comparable for initial deployments and subsequent redeployments.

Results

review-delete-deployment when there is no deployment: https://gitlab.com/gitlab-org/gitlab/-/jobs/1401064779
Job duration: 37 seconds

$ delete_release
** Checking if review-ml-review-fr5mvf exists in the review-ml-review-fr5mvf namespace... **
Deployment status for review-ml-review-fr5mvf is 1

review-deploy when there is no deployment: https://gitlab.com/gitlab-org/gitlab/-/jobs/1401065490
Job duration: 8 minutes

** Deploying review-ml-review-fr5mvf to https://gitlab-review-ml-review-fr5mvf.gitlab-review.app ... **
** Ensuring the review-ml-review-fr5mvf namespace exists... **
Error from server (NotFound): namespaces "review-ml-review-fr5mvf" not found
namespace/review-ml-review-fr5mvf created
...
Release "review-ml-review-fr5mvf" does not exist. Installing it now.
NAME: review-ml-review-fr5mvf
LAST DEPLOYED: Tue Jul  6 00:51:34 2021
NAMESPACE: review-ml-review-fr5mvf
STATUS: deployed
REVISION: 1

review-delete-deployment in a new pipeline when a release has already been deployed: https://gitlab.com/gitlab-org/gitlab/-/jobs/1401088164
Job duration: 40 seconds

$ delete_release
** Checking if review-ml-review-fr5mvf exists in the review-ml-review-fr5mvf namespace... **
Deployment status for review-ml-review-fr5mvf is 0
...
release "review-ml-review-fr5mvf" uninstalled

review-deploy in a new pipeline when a release has already been deployed: https://gitlab.com/gitlab-org/gitlab/-/jobs/1401088417
Job duration: 6 minutes 44 seconds

** Deploying review-ml-review-fr5mvf to https://gitlab-review-ml-review-fr5mvf.gitlab-review.app ... **
** Ensuring the review-ml-review-fr5mvf namespace exists... **
Name:         review-ml-review-fr5mvf
Labels:       tls=review-apps-tls
Annotations:  <none>
Status:       Active
...
Release "review-ml-review-fr5mvf" does not exist. Installing it now.
NAME: review-ml-review-fr5mvf
LAST DEPLOYED: Tue Jul  6 01:35:45 2021
NAMESPACE: review-ml-review-fr5mvf
STATUS: deployed
REVISION: 1

review-stop is still manual and still deletes the namespace: https://gitlab.com/gitlab-org/gitlab/-/jobs/1398691459

$ delete_k8s_release_namespace
namespace "review-ml-review-fr5mvf" deleted

Related issues

#334342 (closed)
#333998 (closed)

Check-list

Pre-merge

Consider the effect of the changes in this merge request on the following:

If new jobs are added:

This will help keep track of expected cost increases to the GitLab project average pipeline cost per merge request RPI

Post-merge

Edited by Mark Lapierre

Merge request reports