Document how backups are supposed to work with containerized Gitlab (docker/kubernetes)
Poor documentation for containerized Gitlab
Most of the documentation seems to assume that Omnibus Gitlab is installed in a "normal" host and not in a container.
For example, https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/raketasks/backup_restore.md#configure-cron-to-make-daily-backups says that For omnibus installations:
To schedule a cron job that backs up your repositories and GitLab metadata, use the root user:
sudo su - crontab -e
> There, add the following line to schedule the backup for everyday at 2 AM:
> ```
0 2 * * * /opt/gitlab/bin/gitlab-rake gitlab:backup:create CRON=1
But what about those of us who are running (Omnibus) GitLab from the (official, I might add) Docker images?
If we were to attempt to actually follow those directions, we would run into the following issues:
root@gitlab-1307966573-fkdk5:/# sudo su -
bash: sudo: command not found2.
root@gitlab-1307966573-fkdk5:/# crontab -e
bash: crontab: command not found
3. Searching the web for why cron might be missing eventually led me to instructions that users are expected to "just install" a cron package ([Readme](https://docs.gitlab.com/omnibus/common_installation_problems/README.html#error-executing-action-create-on-resource-cron-gitlab-ci-schedule-builds), https://gitlab.com/gitlab-org/gitlab-ce/issues/1118, #338). But that again doesn't consider users of the docker image.<p>But even if I were to install cron into my container, both the package and the crontab data would be missing the next time the container got recreated (which is to be considered normal; containers are considered ephemeral and data that needs to be persisted should go in a volume).<p>What are docker users expected to do? Create a custom image `FROM gitlab/gitlab-ce:8.16.4-ce.0` that installs cron, figure out where crontab data is stored, and add a volume for that?
# Add documentation for containerized Gitlab users
I suggest that the following documentation be updated with the recommended solution:
- https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/raketasks/backup_restore.md#configure-cron-to-make-daily-backups
- https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/docker/README.md
- https://gitlab.com/gitlab-org/kubernetes-gitlab-demo
# Add option for automated backups?
Having migrated to the official docker images from sameersbn's [docker-gitlab](https://github.com/sameersbn/docker-gitlab) images, I was actually quite surprised to find this feature missing from all of the "official" ways to run Gitlab in containers.
https://github.com/sameersbn/docker-gitlab#automated-backups gives a *really* easy way to enable daily backups...
> The image can be configured to automatically take backups `daily`, `weekly` or `monthly` using the `GITLAB_BACKUP_SCHEDULE` configuration option.
> Daily backups are created at `GITLAB_BACKUP_TIME` which defaults to `04:00` everyday. Weekly backups are created every Sunday at the same time as the daily backups. Monthly backups are created on the 1st of every month at the same time as the daily backups.
> By default, when automated backups are enabled, backups are held for a period of 7 days. While when automated backups are disabled, the backups are held for an infinite period of time. This behavior can be configured via the `GITLAB_BACKUP_EXPIRY` option.
It sounds like it was already suggested to add a simple way to enable backup cron in https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1265 but that was decided against, so I guess that option is out... :)
# My system environment
We are currently using kubernetes configs based on https://gitlab.com/gitlab-org/kubernetes-gitlab-demo and running in Google Container Engine (GKE).
# Suggested solution for Kubernetes
I created a cronjob resource like this to automatically exec into the gitlab container every day and create a backup.
apiVersion: batch/v2alpha1 kind: CronJob metadata: namespace: gitlab name: create-backup spec:
Time is server time (UTC)
schedule: "00 08 * * " jobTemplate: spec: template: spec: containers: - name: kubectl image: k3integrations/kubectl command: - /bin/bash - -c - pod_name=$(kubectl get pods -n gitlab -l name=gitlab -o jsonpath='{.items[].metadata.name}'); kubectl get po/$pod_name; kubectl exec $pod_name -- gitlab-rake gitlab:backup:create restartPolicy: OnFailure
Seems to be working so far... :)