Patroni configuration: explicitly enabling TLS configuration of client results in Python abort due to 'assert_hostname'

Summary

There is a reproducible bug with Patroni and configuration of the ctl section of patroni.yml, from the recipes and template of Omnibus GitLab. It seems that this has been in place since the upgrade to Patroni 3.x. Either there are more items to document, or we need to find the caused the functional behavior change in the version jump, which resulted in this failure mode.

🤞 This is a matter of configuration, and not a "bug" itself. Either way, we'll need to dive, identify, and possibly backport adjustments.

Details

When enabling HTTPS on the Patroni API, and then setting client mode to required, there is a failure with in the Python library (probably deep in urllib3), which results in TypeError: __init__() got an unexpected keyword argument 'assert_hostname'

This appears to be strictly tied to the following configuration block within gitlab.rb, as documented in Enable TLS support for the Patroni API. Removing this configuration, appears to prevent the occurrence of the problem. Obviously, it is preferable that the configuration as documented results in the desired behavior without this error.

#patroni['tls_client_mode'] = 'required'
#patroni['tls_client_certificate_file'] =  "/etc/gitlab/ssl/server.crt"
#patroni['tls_client_key_file'] = "/etc/gitlab/ssl/server.key"
#patroni['tls_verify'] = true

In particular, we template ctl block, with the affected settings you've listed, in this section.

<% if patroni_helper.verify_client? %>
ctl:
  insecure: false
  <%= "certfile: #{@tls_client_certificate_file}" if @tls_client_certificate_file %>
  <%= "keyfile: #{@tls_client_key_file}" if @tls_client_key_file %>
  <%= "cafile: #{@tls_ca_file}" if @tls_ca_file %>
<% end %>

📚 The reference for this file format is found here for 3.0.1:

Based upon the documentation, it seems that we should not require the configuration of all of these items, however, assert_hostname failing seems to be about the TLS negotiation of the connection and hostnames of the certificates versus the request.

Related items