Intermittent HTTP failures to Ingress controllers

We have been seeing intermittent HTTP failures when trying to connect to ingress controllers.

The first instance was the design.gitlab.com site in this issue: gitlab-org/gitlab-services/design.gitlab.com#432 (closed)

That was resolved by upgrading the K8s cluster and provisioning a new ingress, then re-deploying the app to it. We tried provisioning a new ingress alone, and it didn't resolve it.

The second instance is some suspicious failures on the version.gitlab.com app. We got a Pingdom check failure, and httping is returning some intermittent timeouts.

These may not be related, but this issue can be a place to capture information and explore the possibility.

$  httping -c 15 -t 2 http://version.gitlab.com
PING version.gitlab.com:80 (/):
connected to 104.196.17.203:80 (208 bytes), seq=0 time=294.46 ms
connected to 104.196.17.203:80 (208 bytes), seq=1 time=293.93 ms
connected to 104.196.17.203:80 (208 bytes), seq=2 time=292.10 ms
connected to 104.196.17.203:80 (208 bytes), seq=3 time=296.31 ms
connected to 104.196.17.203:80 (208 bytes), seq=4 time=293.10 ms
connected to 104.196.17.203:80 (208 bytes), seq=5 time=294.27 ms
timeout while receiving reply-headers from host
timeout while receiving reply-headers from host
connected to 104.196.17.203:80 (208 bytes), seq=8 time=293.68 ms
connected to 104.196.17.203:80 (208 bytes), seq=9 time=294.75 ms
connected to 104.196.17.203:80 (208 bytes), seq=10 time=293.74 ms
connected to 104.196.17.203:80 (208 bytes), seq=11 time=295.00 ms
connected to 104.196.17.203:80 (208 bytes), seq=12 time=361.12 ms
connected to 104.196.17.203:80 (208 bytes), seq=13 time=301.54 ms
--- http://version.gitlab.com/ ping statistics ---
15 connects, 12 ok, 13.33% failed, time 22342ms
round-trip min/avg/max = 292.1/300.3/361.1 ms