Gitlab::HTTP not totally compatible with hostnames that resolve to more than one IP address

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Summary

Customer ticket: https://gitlab.zendesk.com/agent/tickets/137634 (internal use)

This is the recap from this ticket (I don't see any customer-identifiable information here):

Summary

The primary node was throwing a 404 error message for the status of the secondary node, while the secondary node showed tracking database cannot be reused. We managed to fix both errors after the steps we took in our call.

Steps taken towards resolution

we ran sudo gitlab-rake gitlab:geo:check on both instances and the secondary was reporting FDW not being configured. After we fixed FDW by running gitlab-rake geo:db:refresh_foreign_tables, the secondary node reported healthy.
we noticed that the 404 error was happening because there was no status for the secondary node
we had a few attempts to manually send the status from the secondary to the primary, which failed
we checked the geo.log and we saw that it was trying to use IPv6 when sending the status to the primary, which didn't work
we solved that by adding a hosts entry pointing to the primary using IPv4
we then retried sending the status to the primary and it worked

To-do

As discussed in the call, you can set the nginx to use IPv6 as well, as per our documentation.

Please have in mind that you would need to run gitlab-ctl reconfigure after changing the nginx settings, so you may want to do this when there is less activity on the instance.

Details

If a hostname resolves to more than one IP address, Gitlab::HTTP replaces a URI's hostname with the first address returned by Addrinfo.getaddrinfo and throws away the rest.

So if a requested URI's hostname resolves to both IPv6 and IPv4 addresses, but the service we are attempting to reach is only listening on IPv4, the request will fail with a connection error.

Possible fixes

There doesn't seem to be an easy answer, because we perform the hostname/IP replacement for security reasons, but in some cases we actually want to attempt to connect to each IP.

cc @WarheadsSE