Make it possible to wait for migrations container to complete

Siddharth Kannanrequested to merge
sk/enable-wait-for-migrations into master

What does this MR do?

Make it possible for extraContainers to wait for migrations container to complete

We would like to run some extraContainers which can start running only after the main migrations container has completed running successfully. The specific use-case is that we want to run only regular migrations (SKIP_POST_DEPLOYMENT_MIGRATIONS=1) and collect the number of PDMs after regular migrations are complete as a metric. (gitlab-com/gl-infra/delivery#2357 (closed)).

So, I am proposing this change which makes the script run by the Migrations chart configurable by the Chart user.

Using this configuration script in conjunction with extraContainers, extraVolumes, and extraVolumeMounts, the user can create shared volumes, and write extra containers which will wait for a file to be created in the shared volume.

The following is a simple example of what can be achieved after this change:

- name: pending-pdm-count-metric-uploader
  image: .../gitlab-toolbox-ee:...
  args:
    - /bin/bash
    - -c
    - |
      # Wait for migrations job to complete
      iterations=0
      max_iterations=10
      while [[ ! -f /srv/gitlab/tmp/migrations-complete ]]; do
        sleep 60;
        iterations=$(($iterations+1))
        if [[ $iterations -ge $max_iterations ]];
        then
            echo "Migrations could not be completed within the timeout of 600 seconds. Quitting."
            return 2
        fi;
      done

      # Perform any operations that must run only after migrations are complete
      pending_pdm_count=$(gitlab-rake db:migrate:status | grep post | grep down | wc -l)

      echo "Pending PDM count: $pending_pdm_count"

      # Post this count as a metric to a monitoring system
  volumeMounts:
    - name: migrations-tmp
      mountPath: '/srv/gitlab/tmp'
      readOnly: true

I considered these other options before deciding on this one:

  1. Repeatedly execute gitlab-rake db:migrate:status and check if regular migrations are all in the up status`: This queries the DB unnecessarily. The larger the GitLab installation, the more unnecessary queries the DB will have to deal with. This feels inefficient.
  2. Use kubectl get pod within an extraContainer and wait for the main container using that: This requires the Pod to have RBAC to run kubectl get pod in the namespace where it is deployed. I don't think it is too common either. There will end up having to be two extra containers:
    1. Main container that runs migrations
    2. One extra container that waits for the migrations container to end successfully
    3. Another extra container to actually do the work that needs to be done after migrations have been completed
  3. Use Pod Lifecycle and a preStop hook to write this file: This is possible. It is a bit more implicit than writing a file directly in the main command of the container.So, I did not choose it. This would also require an update to this chart anyway.
  4. Add this line for creating a file after migrations are complete to the db-migrate script: This is also possible and would serve the same purpose. I don't know if we make any assumptions about the paths that are available within CNG. Even if we do this, we will still have to update the chart to declare an empty volume and mount it in the migrations container.
  5. Add an optional post-upgrade hook to the migrations Job: This was an idea from @clemensbeck. We talked about it briefly on Slack. I did not look into it because it would require a bigger change to the migrations subchart. We would be able to allow users to define containers and run what they want to and guarantee that it would run after the migrations Job was complete.

Discussion on Slack

In addition to the above, during the review of this MR and in other places, we considered other options:

  1. Use gitlab-exporter: gitlab-exporter is a metrics exporter which can connect to the database. However, the information about whether something is a post-deployment migration or a regular migration can be found only with access to the Rails codebase. gitlab-exporter does not currently require a full GitLab Rails codebase in order to function properly
  2. Use Helm hooks: Helm post-upgrade hooks are used frequently for this type of use-case. However, GitLab users might be using Helm only for templating, and then applying the change manually using kubectl apply -f or some other type of tool. So, using Helm hooks is also not a viable option.
  3. Add an API endpoint to Rails: This is being done for batched background migrations already. This is beyond my scope, so I did not look into it further.

Related issues

Author checklist

For general guidance, please follow our Contributing guide.

Required

For anything in this list which will not be completed, please provide a reason in the MR discussion.

  • Merge Request Title and Description are up to date, accurate, and descriptive.
  • MR targeting the appropriate branch.
  • MR has a green pipeline.
  • Documentation created/updated.
  • Tests added/updated, and test plan for scenarios not covered by automated tests.
  • Equivalent MR/issue for omnibus-gitlab opened.
    • Not applicable

Reviewers checklist

Edited by Clemens Beck

Merge request reports