2018-07-16: Intermittent ssl errors on pages when enabling the healthcheck

Summary

On July 16th there were intermittent pages errors for approximately 5 minutes.

The root cause was the result of a configuration change that added the readiness health check. gitlab-com/migration#647 (closed) Although all backends were reporting up we were seeing ssl errors on pages after applying the following change:

     backend pages_https
         mode tcp
         option splice-auto
         balance source
         hash-type consistent
    -    option httpchk GET /-/readiness HTTP/1.1\r\nHost:\ gitlab.com
    -    server web01.fe.gitlab.com 10.69.8.101:1443 check check-ssl inter 2s fastinter 1s downinter 5s fall 3 ssl verify none
    -    server web02.fe.gitlab.com 10.69.8.102:1443 check check-ssl inter 2s fastinter 1s downinter 5s fall 3 ssl verify none
    -    server web03.fe.gitlab.com 10.69.8.103:1443 check check-ssl inter 2s fastinter 1s downinter 5s fall 3 ssl verify none
    -    server web04.fe.gitlab.com 10.69.8.104:1443 check check-ssl inter 2s fastinter 1s downinter 5s fall 3 ssl verify none
    -    server web05.fe.gitlab.com 10.69.8.105:1443 check check-ssl inter 2s fastinter 1s downinter 5s fall 3 ssl verify none
    -    server web06.fe.gitlab.com 10.69.8.106:1443 check check-ssl inter 2s fastinter 1s downinter 5s fall 3 ssl verify none
    -    server web07.fe.gitlab.com 10.69.8.107:1443 check check-ssl inter 2s fastinter 1s downinter 5s fall 3 ssl verify none
    -    server web08.fe.gitlab.com 10.69.8.108:1443 check check-ssl inter 2s fastinter 1s downinter 5s fall 3 ssl verify none
    -    server web09.fe.gitlab.com 10.69.8.109:1443 check check-ssl inter 2s fastinter 1s downinter 5s fall 3 ssl verify none
    -    server web10.fe.gitlab.com 10.69.8.110:1443 check check-ssl inter 2s fastinter 1s downinter 5s fall 3 ssl verify none
    -    server web11.fe.gitlab.com 10.69.8.111:1443 check check-ssl inter 2s fastinter 1s downinter 5s fall 3 ssl verify none
    -    server web12.fe.gitlab.com 10.69.8.112:1443 check check-ssl inter 2s fastinter 1s downinter 5s fall 3 ssl verify none
    -    server web13.fe.gitlab.com 10.69.8.113:1443 check check-ssl inter 2s fastinter 1s downinter 5s fall 3 ssl verify none
    -    server web14.fe.gitlab.com 10.69.8.114:1443 check check-ssl inter 2s fastinter 1s downinter 5s fall 3 ssl verify none
    +    server web01.fe.gitlab.com 10.69.8.101:1443 check inter 5000 fall 3
    +    server web02.fe.gitlab.com 10.69.8.102:1443 check inter 5000 fall 3
    +    server web03.fe.gitlab.com 10.69.8.103:1443 check inter 5000 fall 3
    +    server web04.fe.gitlab.com 10.69.8.104:1443 check inter 5000 fall 3
    +    server web05.fe.gitlab.com 10.69.8.105:1443 check inter 5000 fall 3
    +    server web06.fe.gitlab.com 10.69.8.106:1443 check inter 5000 fall 3
    +    server web07.fe.gitlab.com 10.69.8.107:1443 check inter 5000 fall 3
    +    server web08.fe.gitlab.com 10.69.8.108:1443 check inter 5000 fall 3
    +    server web09.fe.gitlab.com 10.69.8.109:1443 check inter 5000 fall 3
    +    server web10.fe.gitlab.com 10.69.8.110:1443 check inter 5000 fall 3
    +    server web11.fe.gitlab.com 10.69.8.111:1443 check inter 5000 fall 3
    +    server web12.fe.gitlab.com 10.69.8.112:1443 check inter 5000 fall 3
    +    server web13.fe.gitlab.com 10.69.8.113:1443 check inter 5000 fall 3
    +    server web14.fe.gitlab.com 10.69.8.114:1443 check inter 5000 fall 3
Edited Jul 17, 2018 by John Jarvis
Assignee Loading
Time tracking Loading