Skip to content
Snippets Groups Projects
README.md 5.77 KiB
Newer Older
Rémi Verchère's avatar
Rémi Verchère committed
# Promtail Kubernetes Host Logs
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
[Promtail](https://grafana.com/docs/loki/latest/clients/promtail/) is an efficient log shipping agent. It works very fine with Grafana [Loki](https://grafana.com/docs/loki/latest/) on a Kubernetes environment, scraping containers logs with auto-discovery and labels features.
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
It is very simple to install it as a Daemonset on a Kubernetes cluster, and get containers logs. But sometimes we also need to get logs from the nodes where the containers runs on.
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
Below some quick explanation on how to set it up.
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
## Basic installation
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
*Note*: This article uses Promtail v2.5.0, some configuration should differ when you are reading this article.
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
By default, we install the promtail using the helm chart, and it will work out of the box.

We assume we already have a working Loki and [Grafana](https://grafana.com/docs/grafana/latest/) installation.
Rémi Verchère's avatar
Rémi Verchère committed
```shell
$ helm repo add grafana https://grafana.github.io/helm-charts
$ helm repo update
$ helm --install --namespace observability --create-namespace promtail grafana/promtail
Rémi Verchère's avatar
Rémi Verchère committed
```
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
Now we can see that Promtail is running on nodes, and logs getting to Loki. Grafana helps us to explore the logs. Fine!
Rémi Verchère's avatar
Rémi Verchère committed

```shell
$ kubectl get pods -l app.kubernetes.io/name=promtail
NAME             READY   STATUS    RESTARTS   AGE
promtail-47jwn   1/1     Running   0          13m
promtail-mwkcb   1/1     Running   0          14m
Rémi Verchère's avatar
Rémi Verchère committed
```

Rémi Verchère's avatar
Rémi Verchère committed
![](img/promtail_01.png)
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
## Customization
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
For now we have a simple - but working - setup.
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
That's not enough for our needs, we have to forward ̀`/var/log/syslog` host file to loki. Why not using these daemonsets? Let's configure promtail to get them!
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
### Mount host folder
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
First of all, we need to mount the host folder `/var/log` on the pod. We will mount it to `/var/log/host` to avoid mistakes.
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
We specify custom parameters using `custom-values.yaml` file for the helm chart:
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
```yaml
# Mount folder /var/log from node
extraVolumes:
Rémi Verchère's avatar
Rémi Verchère committed
  - name: node-logs
Rémi Verchère's avatar
Rémi Verchère committed
    hostPath:
      path: /var/log
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
extraVolumeMounts:
Rémi Verchère's avatar
Rémi Verchère committed
  - name: node-logs
Rémi Verchère's avatar
Rémi Verchère committed
    mountPath: /var/log/host
    readOnly: true
```
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
Upgrade the helm chart, and see that the `/var/log/host` folder is here, great!
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
```shell
$ helm upgrade --install --namespace observability promtail grafana/promtail -f values.yaml
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
$ kubectl exec -it promtail-mwkcb -- mount | grep /var/log/host
/dev/sda1 on /var/log/host type ext4 (ro,relatime,data=ordered)
```
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
Before going further, let's check if we can read the file:
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
```shell
Rémi Verchère's avatar
Rémi Verchère committed
$ kubectl exec -it promtail-mwkcb -- tail /var/log/host/syslog
tail: cannot open '/var/log/host/syslog' for reading: Permission denied
command terminated with exit code 1
```
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
Oh, There is something wrong here! Let's check file permissions:
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
```shell
$ kubectl exec -it promtail-mwkcb -- ls -l /var/log/host/syslog 
-rw-r----- 1 102 adm 72327736 Jun 29 20:01 /var/log/host/syslog
```
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
This file is owned by user `102`, and group `adm`. Using `getent` shows us the group number:
Rémi Verchère's avatar
Rémi Verchère committed
```shell
$ kubectl exec -it promtail-mwkcb -- getent group adm
adm:x:4:
```
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
So, If we want promtail pod to read that file, we'll have to set some security context using `fsGroup` on our helm custom values file:
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
```yaml
# Set fsGroup to allow syslog file reading
podSecurityContext:
  fsGroup: 4
```

Rémi Verchère's avatar
Rémi Verchère committed
Upgrade the helm chart, and check file access:
Rémi Verchère's avatar
Rémi Verchère committed
```shell
Rémi Verchère's avatar
Rémi Verchère committed
$ helm upgrade --install --namespace observability promtail grafana/promtail -f values.yaml

$ kubectl exec -it promtail-2dbh9 -- tail /var/log/host/syslog 
Jun 29 20:16:28 worker-pool-node-d5353e kubelet[1596]: I0629 20:16:28.374480    1596 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
Jun 29 20:16:28 worker-pool-node-d5353e docker[1888]: I0629 20:16:28.375267       1 utils.go:81] GRPC call: /csi.v1.Node/NodeGetVolumeStats
```

Rémi Verchère's avatar
Rémi Verchère committed
That seems better! We can now read our node files from the promtail pod.
Rémi Verchère's avatar
Rémi Verchère committed

### Promtail configuration to scrape file

Rémi Verchère's avatar
Rémi Verchère committed
Now, we have to configure promtail to scrape this new file, using the simple [file target discovery](https://grafana.com/docs/loki/latest/clients/promtail/scraping/#file-target-discovery). We'll add some label to easily find the logs in loki:
Rémi Verchère's avatar
Rémi Verchère committed

```yaml
# Scrape config to read syslog file from node
config:
  snippets:
    extraScrapeConfigs: |
      # Add an additional scrape config for syslog
Rémi Verchère's avatar
Rémi Verchère committed
      - job_name: node-syslog
Rémi Verchère's avatar
Rémi Verchère committed
        static_configs:
        - targets:
          - localhost
          labels:
Rémi Verchère's avatar
Rémi Verchère committed
            job: node/syslog
Rémi Verchère's avatar
Rémi Verchère committed
            __path__: /var/log/host/syslog
```
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
Upgrade again the helm chart, and now we have my node syslog file going to loki, as all our containers logs, great!
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
![](img/promtail_02.png)

### Adding labels

Rémi Verchère's avatar
Rémi Verchère committed
We now have our logs, but we would like to have some dynamic extra labels, like the node hostname.
Rémi Verchère's avatar
Rémi Verchère committed

Rémi Verchère's avatar
Rémi Verchère committed
So, we're using the `-config.expand-env` argument value to get `ENV`values, and add the one we want to the promtail configuration:
Rémi Verchère's avatar
Rémi Verchère committed

```yaml
# Allow environment variables usage
extraArgs:
  - -config.expand-env=true

# Scrape config to read syslog file from node
config:
  snippets:
    extraScrapeConfigs: |
      # Add an additional scrape config for syslog
      - job_name: node-syslog
        static_configs:
        - targets:
          - localhost
          labels:
            job: node/syslog
            __path__: /var/log/host/syslog
            node_name: '${HOSTNAME}'
```

Rémi Verchère's avatar
Rémi Verchère committed
Upgrade my helm chart again, check on Grafana logs... Et voilà! We have all our nodes logs, with the labels we specified!
Rémi Verchère's avatar
Rémi Verchère committed

![](img/promtail_03.png)

## Summary

Rémi Verchère's avatar
Rémi Verchère committed
To sum up what we've done, and get nodes logs to loki using promtail:
Rémi Verchère's avatar
Rémi Verchère committed
1. Mount `/var/log` from the node to the pod
1. Use the correct `fsGroup`
1. Add `static_configs` scraping configuration
1. Use environment variables to add labels

You can then see here the complete [custom values file](custom-values.yaml).

Now you can deep dive into your logs, and check what's wrong with your nodes... Enjoy!