Gitaly probes fail due to AppArmor
Summary
When installed on a Ubuntu 16.04-based Kubernetes cluster (or any cluster with similar AppArmor settings), the LivenessProbe and ReadinessProbe will always fail. This causes the pod to never become ready, and continually restart.
Workaround
This can be worked around by reverting to using the tcpProbe health check type (i.e. revert gitlab-org/build/CNG!107 (merged)). This no longer causes the spurious messages described in #324 (closed).
Steps to reproduce
- Deploy a cluster on Ubuntu 16.04
- Deploy the Gitlab charts (default helm config)
- Wait for the gitaly pod to become ready
Configuration used
Entirely default
Current behavior
The pod will continually restart with no errors in it's log, with kernel messages on the host similar to the below:
[540603.089879] audit: type=1400 audit(1550152998.005:89307): apparmor="DENIED" operation="ptrace" profile="docker-default" pid=29489 comm="pgrep" requested_mask="trace" denied_mask="trace" peer="docker-default"
Expected behavior
The pod should become ready
Versions
- Chart: 1.5.3
- Platform:
- Self-hosted: kubeadm-deployed on Ubuntu 16.04 nodes
- Kubernetes: (
kubectl version
)- Client: v1.13.2
- Server: v1.13.3
- Helm: (
helm version
)- Client: v2.12.3
- Server: v2.12.3
Relevant logs
kubectl logs:
Begin parsing .erb files from /etc/gitaly/templates
Writing /etc/gitaly/shell-config.yml
Writing /etc/gitaly/config.toml
Copying other config files found in /etc/gitaly/templates
+ exec /bin/sh -c '"/scripts/process-wrapper"'
Starting Gitaly
==> /var/log/gitaly/gitaly.log <==
==> /var/log/gitaly/gitlab-shell.log <==
==> /var/log/gitaly/gitaly.log <==
time="2019-02-14T13:08:17Z" level=info msg="Starting Gitaly" version="Gitaly, version unknown, built 20190206.112923"
time="2019-02-14T13:08:17Z" level=warning msg="git path not configured. Using default path resolution" resolvedPath=/usr/local/bin/git
time="2019-02-14T13:08:17Z" level=warning msg="git path not configured. Using default path resolution" resolvedPath=/usr/local/bin/git
time="2019-02-14T13:08:18Z" level=info msg="finished tempdir cleaner walk" storage=default time_ms=0
time="2019-02-14T13:08:18Z" level=info msg="listening at tcp address" address="0.0.0.0:8075"
time="2019-02-14T13:08:18Z" level=info msg="Starting prometheus listener" address=":9236"
time="2019-02-14T13:08:18Z" level=info msg="starting RSS monitor" supervisor.name=gitaly-ruby.0 supervisor.rss_threshold=209715200
time="2019-02-14T13:08:18Z" level=info msg="starting RSS monitor" supervisor.name=gitaly-ruby.1 supervisor.rss_threshold=209715200
time="2019-02-14T13:08:18Z" level=warning msg=spawned supervisor.args="[bundle exec bin/ruby-cd / /srv/gitaly-ruby/bin/gitaly-ruby 10 /tmp/gitaly-ruby827289293/socket.1]" supervisor.name=gitaly-ruby.1 supervisor.pid=28
time="2019-02-14T13:08:18Z" level=warning msg=spawned supervisor.args="[bundle exec bin/ruby-cd / /srv/gitaly-ruby/bin/gitaly-ruby 10 /tmp/gitaly-ruby827289293/socket.0]" supervisor.name=gitaly-ruby.0 supervisor.pid=29
time="2019-02-14T13:08:18Z" level=info msg="PID 28 BUNDLE_GEMFILE=/srv/gitaly-ruby/Gemfile" supervisor.args="[bundle exec bin/ruby-cd / /srv/gitaly-ruby/bin/gitaly-ruby 10 /tmp/gitaly-ruby827289293/socket.1]" supervisor.name=gitaly-ruby.1
time="2019-02-14T13:08:18Z" level=info msg="PID 29 BUNDLE_GEMFILE=/srv/gitaly-ruby/Gemfile" supervisor.args="[bundle exec bin/ruby-cd / /srv/gitaly-ruby/bin/gitaly-ruby 10 /tmp/gitaly-ruby827289293/socket.0]" supervisor.name=gitaly-ruby.0
Netstat in the pod: kubectl exec -n gitlab gitlab-gitaly-0 -- netstat -tulnp
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp6 0 0 :::8075 :::* LISTEN -
tcp6 0 0 :::9236 :::* LISTEN -