Pages access control randomly throws 503
This happens randomly. A restart of gitlab-pages, fixes the issue temporarily.
{"duration":0.000284804,"host":"projects.your-instance.io","level":"info","method":"GET","msg":"access","proto":"HTTP/2.0","referer":"http://group.your-instance.io/","remoteAddr":"1.1.1.1:64355","status":302,"system":"http","time":"2019-07-15T09:10:20Z","uri":"/auth","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36","written":165} {"host":"group.your-instance.io","level":"info","msg":"Receive OAuth authentication callback","path":"/auth","state":"","time":"2019-07-15T09:10:20Z"} {"error":"Post https://your-instance.com/oauth/token: dial tcp: lookup your-instance.com on [::1]:53: dial udp [::1]:53: connect: no route to host","host":"group.your-instance.io","level":"error","msg":"Fetching access token failed","path":"/auth","redirect_uri":"http://group.your-instance.io/favicon.ico","state":"","time":"2019-07-15T09:10:20Z"}
This happened 2 times in a 2 month period.
My guess would be that GitLab-Pages doesn't retry, once a 429 error was hit (rate limit)
Possible source of the problem
- when GitLab Pages are started, we create a jail in $TMPDIR, it includes such files as
resolv.conf, pages binary, etc -
systemd-tmpfilesperiodically clears uptmp, so these files are removed, and the issue occurs - when pages daemon restarted, files are recreated, and GitLab Pages starts working again.
Possible workaround
Using tempfiles.d allows files not to be removed by systemd-tmpfiles.
Run the following command on your Pages instance(s) (see #231 (comment 374597115))
echo 'x /tmp/gitlab-pages-*' >> /etc/tmpfiles.d/gitlab-pages-jail.conf
Possible solution
Probably it's possible to manage a way to make it generic and include it on the post-inst operations of the RPM packages or as part of the gitlab-ctl configuration handling, for instance.