Implement a correct solution for resources of cluster's pods
Currently the cluster is created setting uniquely resources request and limit on the patroni container but a correct solution would be to split the resources limit and request among all containers of the cluster's pods. We should research the correct solution to this problem making sure that:
- Pod total resources limit and request are shared among all containers and the sum of resources limit and request is equals to the one specified in the profile.
- Pod resources limit and request should be balanced in a way that respect the role of each container. We should make sure each container has the right amount of memory and cpu for it to operate correctly and at best performance possible with that profile configuration.
- Resources limit and request are correctly specified in the configuration of a profile. Currently we set a value for cpu and memory and use that in both resource limit and request, we should check this is a correct approach.
Proposal
This is a proposal to enhance SGInstanceProfile
in order to set the resource requirements for containers and init containers and a similar section to include with #820 (closed) or implement with this issue if #820 (closed) get implemented before this one:
apiVersion: stackgres.io/v1
kind: SGInstanceProfile
spec:
cpu: 16000
memory: 64Gi
containers:
pgbouncer:
cpu: 1000
memory: 64Mi
envoy:
cpu: 1000
memory: 64Mi
prometheus-postgres-exporter:
cpu: 1000
memory: 8Mi
postgres-util:
cpu: 1000
memory: 8Mi
fluent-bit:
cpu: 1000
memory: 8Mi
fluentd:
cpu: 4000
memory: 2Gi
cluster-controller: # Could be applied also to distributedlogs-controller
cpu: 1000
memory: 512Mi # Fix #1566 to improve this
initContainers:
setup-arbitrary-user:
cpu: 1000
memory: 8Mi
setup-data-paths:
cpu: 1000
memory: 8Mi
relocate-binaries:
cpu: 1000
memory: 8Mi
setup-scripts:
cpu: 1000
memory: 8Mi
pgbouncer-auth-file:
cpu: 1000
memory: 8Mi
cluster-reconciliation-cycle: # Could be applied also to distributedlogs-reconciliation-cycle
cpu: 1000
memory: 512Mi # Fix #1566 to improve this
major-version-upgrade:
cpu: 16000
memory: 64Gi
reset-patroni:
cpu: 1000
memory: 8Mi
If the user do not specify a container sub-section in the .spec.containers
or .spec.initContainers
section here are the proposed formulas to calculate the values of such sub-sections on creation of sGInstanceProfile
:
Those values may be lowered. Also in those cases the requests may be lower than the limit when it comes to apply this resource requirements to specified containers in order to use the shared CPUs when CPU manager policy is set to static in order to be in the
Burstable
group and use the shared pool of CPUs.
-
For
pgbouncer
:-
cpu
as millicpu:min(1000, floor(toMillicpu(".spec.cpu") / 16))
-
memory
as Mi:64
-
With 64Mi
PgBouncer should be able to handle 4096 connections. See https://www.pgbouncer.org/features.html
Low memory requirements (2 kB per connection by default). This is because PgBouncer does not need to see full packets at once.
-
For
envoy
:-
cpu
as millicpu:min(4000, floor(toMillicpu(".spec.cpu") / 4))
-
memory
as Mi:64
-
Scaling connections with pgbench from 1 connection to 100 only add 1Mi of used memory to the 20Mi initially used and with another increment of 11Mi with 1000 connections. If we consider 12Mi each 1000 connections with 64Mi envoy should be able to handle up to 3500 connections.
-
For
prometheus-postgres-exporter
:-
cpu
as millicpu:min(1000, floor(toMillicpu(".spec.cpu") / 16))
-
memory
as Mi:8Mi
-
-
For
postgres-util
:-
cpu
as millicpu:min(1000, floor(toMillicpu(".spec.cpu") / 16))
-
memory
as Mi:8Mi
-
-
For
fluent-bit
:-
cpu
as millicpu:min(1000, floor(toMillicpu(".spec.cpu") / 16))
-
memory
as Mi:8Mi
-
-
For
fluentd
:-
cpu
as millicpu:min(4000, floor(toMillicpu(".spec.cpu") / 4))
-
memory
as Mi:2Gi
-
Memory usage is quite high. Starting from 512Mi it easily increases to 612Mi with a cluster of 3 instances with low log usage. So 2Gi
seems a quite safe value.
For cluster-controller
/ distributedlogs-controller
/ cluster-reconciliation-cycle
/ distributedlogs-reconciliation-cycle
:
-
cpu
as millicpu:min(1000, floor(toMillicpu(".spec.cpu") / 4))
-
memory
as Mi:512Mi
See #1566 (closed).
For major-version-upgrade
:
-
cpu
as millicpu:toMillicpu(".spec.cpu")
-
memory
as Mi:toMi(".spec.memory")
Major version upgrade will run pg_upgrade
command that may or may not require all of that CPU and memory. In any case better to give all the available resources to this container since it runs alone and we may find out a faster alternative that requires more memory and CPU in the future.
-
For
prometheus-postgres-exporter
/postgres-util
/fluent-bit
/setup-arbitrary-user
/setup-data-paths
/relocate-binaries
/setup-scripts
/pgbouncer-auth-file
/reset-patroni
:-
cpu
as millicpu:toMillicpu(".spec.cpu")
-
memory
as Mi:toMi(".spec.memory")
-
Init containers may or may not require all of that CPU and memory. In any case better to give all the available resources to any init container since they runs alone.
** Acceptance criteria: **
-
Implement the feature -
Implement the changhe in the REST API. -
create test -
Documentation