Pages access control randomly throws 503

This happens randomly. A restart of gitlab-pages, fixes the issue temporarily.

{"duration":0.000284804,"host":"projects.your-instance.io","level":"info","method":"GET","msg":"access","proto":"HTTP/2.0","referer":"http://group.your-instance.io/","remoteAddr":"1.1.1.1:64355","status":302,"system":"http","time":"2019-07-15T09:10:20Z","uri":"/auth","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36","written":165} {"host":"group.your-instance.io","level":"info","msg":"Receive OAuth authentication callback","path":"/auth","state":"","time":"2019-07-15T09:10:20Z"} {"error":"Post https://your-instance.com/oauth/token: dial tcp: lookup your-instance.com on [::1]:53: dial udp [::1]:53: connect: no route to host","host":"group.your-instance.io","level":"error","msg":"Fetching access token failed","path":"/auth","redirect_uri":"http://group.your-instance.io/favicon.ico","state":"","time":"2019-07-15T09:10:20Z"}

This happened 2 times in a 2 month period.

My guess would be that GitLab-Pages doesn't retry, once a 429 error was hit (rate limit)

Possible source of the problem

  • when GitLab Pages are started, we create a jail in $TMPDIR, it includes such files as resolv.conf, pages binary, etc
  • systemd-tmpfiles periodically clears up tmp, so these files are removed, and the issue occurs
  • when pages daemon restarted, files are recreated, and GitLab Pages starts working again.

Possible workaround

Using tempfiles.d allows files not to be removed by systemd-tmpfiles.

Run the following command on your Pages instance(s) (see #231 (comment 374597115))

echo 'x /tmp/gitlab-pages-*' >> /etc/tmpfiles.d/gitlab-pages-jail.conf

Possible solution

Probably it's possible to manage a way to make it generic and include it on the post-inst operations of the RPM packages or as part of the gitlab-ctl configuration handling, for instance.

Edited by Jaime Martinez