Adding git-cache-proxy housekeeping service

See merge request !18

The housekeeping that git-cache-proxy is supposed to be doing while
handling connections doesn't work when using systemd.socket.

The housekeeping task fork from the main process which is handling one
connection, but the main process doesn't wait. So once the repo is
served, systemd simply kill all remaining task. The housekeeping task
never managed to finished. Often, it will even leave a "tmp_pack_*"
file that can be big, and fill the disk slowly.

Add a service that is run daily to do the housekeeping task.

Reduce "gc-interval-days" to half the default, to try having this new
.timer do the jobs rather than the .socket one. We can't disable the
housekeeping done automatically at the moment.

Some fixes. Fix "pre-build-script". Fix start of "git-cache-proxy.socket" at reboot.

See merge request !17

This sockets can only be enabled after docker have started as we are
binding to an address that is created with the bridge "docker0" which
docker creates. Also, create this socket when gitlab-runner is
started, instead of having "sockets.targets" depends on "docker.service".

This is to avoid failures to start "git-cache-proxy.socket" on reboot.

Some container don't have "git" installed, so the `git config` fails.
This failure in turn result in a failure for the job.

Check that "git" is installed.

Add git-cache-proxy; update ubuntu, plus other improvement

See merge request !16

This allows to cache git trees on the runners.

Jobs will start using the cache automatically due to the
"pre-build-script" which rewrite git URLs to use the cache.

There's a bug in git-cache-proxy which prevent it from serving newer
git client, so we have to patch the script.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1006695

We can make use of cloud-init to update the new instance. It will also
be able to reboot it if necesssary to for example update the kernel.

https://cloudinit.readthedocs.io/en/latest/topics/modules.html#package-update-upgrade-install

Before doing anything else, we need to wait for cloud-init to finish.

Before destroying an instance of a runner, we can wait for the current
build to finish, by doing a graceful shutdown.

https://docs.gitlab.com/runner/best_practice/index.html#graceful-shutdown

Only unregister the runner after it has stop has otherwise it wouldn't
be able to send the result back.

It will be shared with all provisioners.

replace docker-machine runner by plain docker (remove autoscaling)

See merge request !14

This should replace the autoscale of runners that spawn between 1 and
10 machines to have 10 machine always running. (But will start with 1
as the autoscaling didn't work for other reason.)

We remove docker+machine stuff as we will use "docker" on the machine
setup by teraform. (docker+machine is mainly use for autoscaling.)

This is to workaround "jobs" which fails with `git clone` error for
example:
    Cloning into 'qemu-xen-dir-remote.tmp'...
    error: inflate: data stream error (incorrect data check)
    fatal: serious inflate inconsistency
    fatal: index-pack failed

For the renamed "gitlab-docker-machine" to "gitlab-docker", we can't
really use "count" to increase the number of instance or they would
all have the same name. Instead, we add cound to "random_pet.name" and
create an instance for each one of the pet name.

We now provide a name to `gitlab-runner register` via --name so we can
reference to it to unregister it.

enable docker-privileged so we can build containers

See merge request !13

There is an interest to rebuild the containers periodically and to do so
we need to enable docker-privleged so that we can access Docker itself
and use it to build those containers.

convert to terraform 0.13

See merge request !12

The following commands needed to be run to convert the state with a
terraform 0.13 binary:
terraform state replace-provider registry.terraform.io/-/random registry.terraform.io/hashicorp/random
terraform state replace-provider registry.terraform.io/-/template registry.terraform.io/hashicorp/template
terraform state replace-provider registry.terraform.io/-/openstack registry.terraform.io/terraform-provider-openstack/openstack

We can clean this up manually but in most cases the issue is the user
does not have the generated SSH key.

This upsets future versions of terraform so get rid of it since it works
with the current version.

Break the provision command out to a script to make it easier to follow
what is happening.

add additional notes to the README

See merge request !10

docker: force version 19.03.9

See merge request !11

Seems like using the current version of docker-machine with a version
of docker >= 20.10.0. This has been reported at least in:

gitlab-org/ci-cd/docker-machine#41



Temporary fix this by forcing the install of Docker 19.03.9.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>

switch to dynamically looking up Ubuntu images

See merge request !9

Get the latest Ubuntu 18.04 image instead of hardcoding ourselves to
what may become and outdated image.

upgrade to Terraform 0.12

See merge request !8

Update to the Terraform 0.12 DSL and provide info on how folks can do
updates.

Small fixes

See merge request !7

Without noninteractive set then a package upgrade can prompt us to
answer a question which cannot be answered.

If the runner machines have already been deleted then we don't want to
fail to destroy.

Tag x86 runners

See merge request !6

Signed-off-by: Wei Liu <wei.liu2@citrix.com>

Use noninteractive frontend, so that when there is a conflict in
service configuration files, the default action is taken.

Move changing sshd_config after dist-upgrade, to avoid the "default
action" overrides our own setting. That's not likely to happen, but
better safe than sorry.
Signed-off-by: Wei Liu <wei.liu2@citrix.com>

Fix state deletion issue, remove old fields, add random name for key

See merge request !5

Should this be used with multiple terraform runs or multiple keys be
generated by accident, attempt to link a name to each key and its
resources to help tracking them easier.

Removed the old gitlab-runner instance since we're using the
docker-machine instance exclusively now. Reordered some of the variables
and renamed them to match the docker-machine scheme.

Due to a bug in the terraform.io docs
https://github.com/hashicorp/terraform/issues/19005 the configuration
for the state is actually incorrect and gets deleted after 7 days
instead of being versioned and stored persistently. Switch to using
`archive_container` which is the actual field that the docs should have
described as setting up object versioning.

fix destroying the gitlab-docker-machine

See merge request !3

When destroying the gitlab-docker-machine, the gitlab-runner service
needs to be stopped to avoid it respawning another runner VM. The test
expression in bash was failing until it was quoted.
Signed-off-by: Doug Goldstein <cardoe@cardoe.com>