Commit f6140438 authored by Achilleas Pipinellis's avatar Achilleas Pipinellis 🚀

Add new doc about the init process of the Runner

parent 93c5b063
......@@ -381,7 +381,7 @@ concurrent = 50 # All registered Runners can run up to 50 concurrent jobs
OffPeakIdleTime = 1200 # Each machine can be in Idle state up to 1200 seconds (after this it will be removed) - when Off Peak time mode is on
IdleCount = 5 # There must be 5 machines in Idle state - when Off Peak time mode is off
IdleTime = 600 # Each machine can be in Idle state up to 600 seconds (after this it will be removed) - when Off Peak time mode is off
MaxBuilds = 100 # Each machine can handle up to 100 jobs in a row (after this it will be removed)
MaxBuilds = 100 # Each machine can handle up to 100 jobs in a row (after this it will be removed)
MachineName = "auto-scale-%s" # Each machine will have a unique name ('%s' is required)
MachineDriver = "digitalocean" # Docker Machine is using the 'digitalocean' driver
MachineOptions = [
......
......@@ -10,11 +10,12 @@ well as information how to set up Prometheus metrics:
- [Advanced configuration options](advanced-configuration.md) Learn how to use the [TOML][] configuration file that GitLab Runner uses.
- [Use self-signed certificates](tls-self-signed.md) Configure certificates that are used to verify TLS peer when connecting to the GitLab server.
- [Auto-scaling using Docker machine](autoscale.md) Execute jobs on machines that are created on demand using Docker machine.
- [Autoscaling using Docker machine](autoscale.md) Execute jobs on machines that are created on demand using Docker machine.
- [Autoscaling GitLab Runner on AWS](runner_autoscale_aws/index.md)
- [The init system of GitLab Runner](init.md) Learn how the Runner installs its init service files based on your operating system.
- [Supported shells](../shells/README.md) Learn what shell script generators are supported that allow to execute builds on different systems.
- [Security considerations](../security/index.md) Be aware of potential security implications when running your jobs with GitLab Runner.
- [Prometheus monitoring](../monitoring/README.md) Learn how to use the Prometheus metrics HTTP server.
- [Runner monitoring](../monitoring/README.md) Learn how to monitor the Runner's behavior.
- [Cleanup the Docker images automatically](https://gitlab.com/gitlab-org/gitlab-runner-docker-cleanup) A simple Docker application that automatically garbage collects the GitLab Runner caches and images when running low on disk space.
[TOML]: https://github.com/toml-lang/toml
# The system services of GitLab Runner
GitLab Runner uses the [Go `service` library](https://github.com/kardianos/service)
to detect the underlying OS and eventually install the service file based on
the init system.
NOTE: **Note:**
`service` will install / un-install, start / stop, and run a program as a
service (daemon). Currently supports Windows XP+, Linux/(systemd | Upstart | SysV),
and OSX/Launchd.
Once GitLab Runner [is installed](../install/index.md), the service file will
be automatically be created:
- **systemd:** `/etc/systemd/system/gitlab-runner.service`
- **upstart:** `/etc/init/gitlab-runner`
## Overriding the default service files
In some cases, you might want to override the default behavior of the service.
For example, when upgrading the Runner, you'll want it to stop gracefully
until all running jobs are finished, but systemd, upstart, or some other service
may almost immediately restart the process without even noticing.
So, when upgrading the Runner, the installation script will kill and restart
the Runner's process which would most probably be handling some new jobs at
that time.
### Overriding systemd
For Runners that use systemd create
`/etc/systemd/system/gitlab-runner.service.d/kill.conf` with the following
content:
```bash
[Service]
TimeoutStopSec=7200
KillSignal=SIGQUIT
```
After adding these two settings to the systemd unit configuration, you can
stop the Runner and systemd will use `SIGQUIT` as the kill signal, to stop the
process. Additionally, a 2h timeout is set for the stop command, which
means that if any jobs don't terminate gracefully before this timeout, systemd
will just `SIGKILL` the process.
### Overriding upstart
For Runners that use upstart create `/etc/init/gitlab-runner.override` with the
following content:
```bash
kill signal SIGQUIT
kill timeout 7200
```
After adding these two settings to the upstart unit configuration, you can
stop the Runner and upstart will do exactly the same as systemd above.
......@@ -31,20 +31,47 @@ out of the scope of this documentation. For more details please read the
1. Edit [`config.toml`](../commands/README.md#configuration-file) and configure
the Runner to use Docker machine. Visit the dedicated page covering detailed
information about [GitLab Runner Autoscaling](../configuration/autoscale.md).
1. Try to build your project. In a few seconds, if you run `docker-machine ls`
you should see a new machine being created.
1. The **first time** you're using Docker Machine, it's best to execute manually
`docker-machine create ...` with your chosen driver and all options from the
`MachineOptions` section. This will set up the Docker Machine environment
properly and will also be a good validation of the specified options.
After this, you can destroy the machine with `docker-machine rm [machine_name]`
and start the Runner.
NOTE: **Note:**
Multiple concurrent requests to `docker-machine create` that are done
**at first usage** are not good. When the `docker+machine` executor is used,
the Runner may spin up few concurrent `docker-machine create` commands. If
Docker Machine was not used before in this environment, each started process
tries to prepare SSH keys and SSL certificates (for Docker API authentication
between Runner and Docker Engine on the autoscaled spawned machine), and these
concurrent processes are disturbing each other. This can end with a non-working
environment. That's why it's important to create a test machine manually the
very first time you set up the Runner with Docker Machine.
1. Now, you can try and start a new pipeline in your project. In a few seconds,
if you run `docker-machine ls` you should see a new machine being created.
## Upgrading the Runner
1. Ensure your operating system isn't configured to automatically restart the
Runner if it exits (which is the default configuration on some platforms).
1. Check if your operating system is configured to automatically restart the
Runner (for example, by checking its service file):
- **if yes**, ensure that service manager is [configured to use `SIGQUIT`](../configuration/init.md)
and use the service's tools to stop the process:
```
# For systemd
sudo systemctl stop gitlab-runner
1. Stop the Runner:
# For upstart
sudo service gitlab-runner stop
```
- **if no**, you may stop the Runner's process manually:
```bash
killall -SIGQUIT gitlab-runner
```
```bash
sudo killall -SIGQUIT gitlab-runner
```
NOTE: **Note:**
Sending the [`SIGQUIT` signal](../commands/README.md#signals) will make the
Runner to stop gracefully. It will stop accepting new jobs, and will exit
as soon as the current jobs are finished.
......@@ -59,28 +86,4 @@ out of the scope of this documentation. For more details please read the
done
```
1. You can now safely upgrade the Runner without interrupting any jobs
## Managing the Docker Machines
1. Ensure your operating system isn't configured to automatically restart the
Runner if it exits (which is the default configuration on some platforms).
1. Stop the Runner:
```bash
killall -SIGQUIT gitlab-runner
```
1. Wait until the Runner exits. You can check its status with: `gitlab-runner status`
or await a graceful shutdown for up to 30 minutes with:
```bash
for i in `seq 1 180`; do # 1800 seconds = 30 minutes
gitlab-runner status || break
sleep 10
done
```
1. You can now manage (upgrade or remove) any Docker Machines with the
[`docker-machine` command](https://docs.docker.com/machine/reference/)
1. You can now safely install the new Runner without interrupting any jobs
......@@ -115,11 +115,12 @@ To jump into the specific documentation of each executor, visit:
- [Advanced configuration options](configuration/advanced-configuration.md) Learn how to use the [TOML][] configuration file that GitLab Runner uses.
- [Use self-signed certificates](configuration/tls-self-signed.md) Configure certificates that are used to verify TLS peer when connecting to the GitLab server.
- [Auto-scaling using Docker machine](configuration/autoscale.md) Execute jobs on machines that are created on demand using Docker machine.
- [Autoscaling using Docker machine](configuration/autoscale.md) Execute jobs on machines that are created on demand using Docker machine.
- [Autoscaling GitLab Runner on AWS](configuration/runner_autoscale_aws/index.md)
- [The init system of GitLab Runner](configuration/init.md) Learn how the Runner installs its init service files based on your operating system.
- [Supported shells](shells/README.md) Learn what shell script generators are supported that allow to execute builds on different systems.
- [Security considerations](security/index.md) Be aware of potential security implications when running your jobs with GitLab Runner.
- [Runner monitoring](monitoring/README.md) Learn how to monitor Runner's behavior.
- [Runner monitoring](monitoring/README.md) Learn how to monitor the Runner's behavior.
- [Cleanup the Docker images automatically](https://gitlab.com/gitlab-org/gitlab-runner-docker-cleanup) A simple Docker application that automatically garbage collects the GitLab Runner caches and images when running low on disk space.
## Troubleshooting
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment