Skip to content

Document more secure Patroni patterns

Summary

Documented and / or automated procedures for how to secure access to the Patroni API.

As required, configuration changes are put in gitlab.rb and deployed by gitlab-ctl reconfigure. This is a key requirement of the customer, they don't want settings to be removed by gitlab-ctl reconfigure.

Proposal

Some customers will require that access to network services will be restricted.

In the case of Patroni, a security issue with the API was identified by a customer, and they asked how they can lock down port 8008.

host based firewalls

The first tier of protection is logical network access. While the goal is to lock down port 8008, if the servers' firewalls are turned on (eg: firewalld) then all access needs to be defined to some extent.

  • What ports need to be open on servers performing the roles of any or all of Consul, Patroni, and PgBouncer
    • potentially all three components could be on the same servers, so inter-connectivity for all the components needs to be defined.
  • What components access the Patroni API
    • Are the only Patroni clients the other Patroni servers (for example: to elect the Leader?)
    • Do none of PostgreSQL, PgBouncer or Consul attempt to use the API?

The Patroni docs indicate the following about API clients:

The Patroni REST API is used by Patroni itself during the leader race, by the patronictl tool in order to perform failovers/switchovers/reinitialize/restarts/reloads, by HAProxy or any other kind of load balancer to perform HTTP health checks, and of course could also be used for monitoring.

allowlist support

An allowlist was added to Patroni in response to the issue being identified (pull request)

To support this, additional configuration would need to be injected into the patroni configuration from gitlab.rb.

A generic way to add configuration to Patroni is probably the most flexible approach, do we already have support for this in Omnibus GitLab?

This wasn't specifically asked for in this case, but for customers who don't want to have to implement host firewalls, this might be more flexible.

authentication

Patroni documents two ways to ensure Patroni clients have to authenticate to the API

HTTP basic-auth

The unsafe endpoints can be protected with HTTP basic-auth by setting the restapi.authentication.username and restapi.authentication.password parameters. There is no way to protect the safe endpoints without enabling TLS.

The distinction between 'safe endpoints' and 'unsafe endpoints' needs clarification. If the benefit from authentication would only be gained if all endpoints are authenticated, then it seems like TLS is required.

Assuming basic auth is useful, this would involve adding some additional configuration entries, via gitlab.rb to Patroni - for when it's acting as a server or a client.

Do we support adding these keys specifically, and if not, do we have a generic way to add configuration (already mentioned above for allowlist)

Are there any other clients of the Patroni API? This is the same question as in host base firewalls, above.

If there are other clients, do they support basic auth. patronictl is, I assume what underpins the Patroni gitlab-ctl functions and would be a key one that needs to work.

TLS certificates

When TLS for the REST API is enabled and a PKI is established, mutual authentication of the API server and API client is possible for all endpoints.

This would be more complicated. From the discussion we had, I understand the customer would be happy with basic-auth. However, if they wouldn't be able to fully protect access to the API using basic-auth, then this may be required.

References

A customer raised this requirement via Support. GitLab team members can read more in the ticket

Edited by Ben Prescott_