Too many open files in cluster controller
Summary
After some time the cluster-controller
container will throw the error Too many open files
that may prevent some functionality from working.
Current Behaviour
The cluster-controller
container will throw the error Too many open files
Steps to reproduce
- Create a cluster with SSL enabled
Expected Behaviour
The cluster-controller
container do not throw any error
Possible Solution
Seems that the problem is related to when the operator checks for SSL certificates
Environment
- StackGres version: 1.6.0
- Kubernetes version: *
- Cloud provider or hardware configuration: *
Relevant logs and/or screenshots
An example of the error is as follow:
An error occurred while reconciling patroni configuration: /etc/patroni/config.yml: Too many open files
The limits of the process and the file descriptors present indicate that a lot of FD on /etc/ssl are present:
$ kubectl exec -ti -n "$CLUSTER_NAMESPACE" "$CLUSTER_NAME-0" -c postgres-util -- bash -c 'PID="$(for FILE in /proc/[0-9]*; do echo "${FILE##*/} $(tr "\0" " " < "$FILE/cmdline")"; done | grep [j]ava | cut -d " " -f 1)"; cat "/proc/$PID/limits"; echo; ls -1 "/proc/$PID/fd" | wc -l;'
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size unlimited unlimited bytes
Max resident set unlimited unlimited bytes
Max processes unlimited unlimited processes
Max open files 1048576 1048576 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 126812 126812 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
2045
kubectl exec -ti -n "$CLUSTER_NAMESPACE" "$CLUSTER_NAME-0" -c postgres-util -- bash -c 'PID="$(for FILE in /proc/[0-9]*; do echo "${FILE##*/} $(tr "\0" " " < "$FILE/cmdline")"; done | grep [j]ava | cut -d " " -f 1)"; ls -l "/proc/$PID/fd" | cut -d ">" -f 2 | cut -d " " -f 2 | while read FILE; do echo "$FILE"; done | sort | uniq -c'
1 0
17 anon_inode:[eventfd]
17 anon_inode:[eventpoll]
1 /app/app/stackgres-cluster-controller.jar
1 /app/lib/boot/io.github.crac.org-crac-0.1.3.jar
1 /app/lib/boot/io.quarkus.quarkus-bootstrap-runner-2.15.3.Final.jar
1 /app/lib/boot/io.quarkus.quarkus-development-mode-spi-2.15.3.Final.jar
1 /app/lib/boot/io.quarkus.quarkus-vertx-latebound-mdc-provider-2.15.3.Final.jar
1 /app/lib/boot/io.smallrye.common.smallrye-common-io-1.13.2.jar
1 /app/lib/boot/org.graalvm.sdk.graal-sdk-22.3.0.jar
1 /app/lib/boot/org.jboss.logging.jboss-logging-3.5.0.Final.jar
1 /app/lib/boot/org.jboss.logmanager.jboss-logmanager-embedded-1.0.11.jar
1 /app/lib/boot/org.wildfly.common.wildfly-common-1.5.4.Final-format-001.jar
1 /app/lib/main/com.aayushatharva.brotli4j.brotli4j-1.8.0.jar
1 /app/lib/main/com.aayushatharva.brotli4j.native-linux-x86_64-1.8.0.jar
1 /app/lib/main/com.fasterxml.classmate-1.5.1.jar
1 /app/lib/main/com.fasterxml.jackson.core.jackson-annotations-2.14.1.jar
1 /app/lib/main/com.fasterxml.jackson.core.jackson-core-2.14.1.jar
1 /app/lib/main/com.fasterxml.jackson.core.jackson-databind-2.14.1.jar
1 /app/lib/main/com.fasterxml.jackson.dataformat.jackson-dataformat-properties-2.14.1.jar
1 /app/lib/main/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.14.1.jar
1 /app/lib/main/com.fasterxml.jackson.datatype.jackson-datatype-jdk8-2.14.1.jar
1 /app/lib/main/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.14.1.jar
1 /app/lib/main/com.fasterxml.jackson.jaxrs.jackson-jaxrs-base-2.14.1.jar
1 /app/lib/main/com.fasterxml.jackson.jaxrs.jackson-jaxrs-json-provider-2.14.1.jar
1 /app/lib/main/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.14.1.jar
1 /app/lib/main/com.fasterxml.jackson.module.jackson-module-parameter-names-2.14.1.jar
1 /app/lib/main/com.github.java-json-tools.btf-1.3.jar
1 /app/lib/main/com.github.java-json-tools.jackson-coreutils-2.0.jar
1 /app/lib/main/com.github.java-json-tools.json-patch-1.13.jar
1 /app/lib/main/com.github.java-json-tools.msg-simple-1.2.jar
1 /app/lib/main/com.google.errorprone.error_prone_annotations-2.16.jar
1 /app/lib/main/com.google.guava.failureaccess-1.0.1.jar
1 /app/lib/main/com.google.guava.guava-31.1-jre.jar
1 /app/lib/main/com.google.j2objc.j2objc-annotations-1.3.jar
1 /app/lib/main/com.ibm.async.asyncutil-0.1.0.jar
1 /app/lib/main/commons-codec.commons-codec-1.15.jar
1 /app/lib/main/commons-io.commons-io-2.11.0.jar
1 /app/lib/main/com.ongres.fluent-process-1.0.1.jar
1 /app/lib/main/com.squareup.okhttp3.logging-interceptor-3.14.9.jar
1 /app/lib/main/com.squareup.okhttp3.okhttp-3.14.9.jar
1 /app/lib/main/com.squareup.okio.okio-1.17.2.jar
1 /app/lib/main/com.sun.activation.jakarta.activation-1.2.1.jar
1 /app/lib/main/io.fabric8.kubernetes-client-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-client-api-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-httpclient-okhttp-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-model-admissionregistration-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-model-apiextensions-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-model-apps-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-model-autoscaling-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-model-batch-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-model-certificates-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-model-common-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-model-coordination-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-model-core-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-model-discovery-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-model-events-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-model-extensions-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-model-flowcontrol-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-model-gatewayapi-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-model-metrics-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-model-networking-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-model-node-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-model-policy-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-model-rbac-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-model-scheduling-6.2.0.jar
1 /app/lib/main/io.fabric8.kubernetes-model-storageclass-6.2.0.jar
1 /app/lib/main/io.fabric8.zjsonpatch-0.3.0.jar
1 /app/lib/main/io.netty.netty-buffer-4.1.86.Final.jar
1 /app/lib/main/io.netty.netty-codec-4.1.86.Final.jar
1 /app/lib/main/io.netty.netty-codec-dns-4.1.86.Final.jar
1 /app/lib/main/io.netty.netty-codec-haproxy-4.1.86.Final.jar
1 /app/lib/main/io.netty.netty-codec-http2-4.1.86.Final.jar
1 /app/lib/main/io.netty.netty-codec-http-4.1.86.Final.jar
1 /app/lib/main/io.netty.netty-codec-socks-4.1.86.Final.jar
1 /app/lib/main/io.netty.netty-common-4.1.86.Final.jar
1 /app/lib/main/io.netty.netty-handler-4.1.86.Final.jar
1 /app/lib/main/io.netty.netty-handler-proxy-4.1.86.Final.jar
1 /app/lib/main/io.netty.netty-resolver-4.1.86.Final.jar
1 /app/lib/main/io.netty.netty-resolver-dns-4.1.86.Final.jar
1 /app/lib/main/io.netty.netty-transport-4.1.86.Final.jar
1 /app/lib/main/io.netty.netty-transport-native-unix-common-4.1.86.Final.jar
1 /app/lib/main/io.quarkus.arc.arc-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-apache-httpclient-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-arc-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-core-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-credentials-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-fs-util-0.0.9.jar
1 /app/lib/main/io.quarkus.quarkus-hibernate-validator-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-jackson-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-jdbc-postgresql-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-jsonp-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-kubernetes-client-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-kubernetes-client-internal-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-mutiny-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-netty-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-rest-client-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-rest-client-config-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-resteasy-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-resteasy-common-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-resteasy-jackson-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-resteasy-server-common-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-security-runtime-spi-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-smallrye-context-propagation-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-smallrye-health-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-vertx-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-vertx-http-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.quarkus-vertx-http-dev-console-runtime-spi-2.15.3.Final.jar
1 /app/lib/main/io.quarkus.security.quarkus-security-1.1.4.Final.jar
1 /app/lib/main/io.smallrye.common.smallrye-common-annotation-1.13.2.jar
1 /app/lib/main/io.smallrye.common.smallrye-common-classloader-1.13.2.jar
1 /app/lib/main/io.smallrye.common.smallrye-common-constraint-1.13.2.jar
1 /app/lib/main/io.smallrye.common.smallrye-common-expression-1.13.2.jar
1 /app/lib/main/io.smallrye.common.smallrye-common-function-1.13.2.jar
1 /app/lib/main/io.smallrye.common.smallrye-common-vertx-context-1.13.2.jar
1 /app/lib/main/io.smallrye.config.smallrye-config-2.13.1.jar
1 /app/lib/main/io.smallrye.config.smallrye-config-common-2.13.1.jar
1 /app/lib/main/io.smallrye.config.smallrye-config-core-2.13.1.jar
1 /app/lib/main/io.smallrye.config.smallrye-config-source-yaml-2.13.1.jar
1 /app/lib/main/io.smallrye.config.smallrye-config-validator-2.13.1.jar
1 /app/lib/main/io.smallrye.jandex-3.0.5.jar
1 /app/lib/main/io.smallrye.reactive.mutiny-1.8.0.jar
1 /app/lib/main/io.smallrye.reactive.mutiny-smallrye-context-propagation-1.8.0.jar
1 /app/lib/main/io.smallrye.reactive.smallrye-mutiny-vertx-auth-common-2.29.0.jar
1 /app/lib/main/io.smallrye.reactive.smallrye-mutiny-vertx-bridge-common-2.29.0.jar
1 /app/lib/main/io.smallrye.reactive.smallrye-mutiny-vertx-core-2.29.0.jar
1 /app/lib/main/io.smallrye.reactive.smallrye-mutiny-vertx-runtime-2.29.0.jar
1 /app/lib/main/io.smallrye.reactive.smallrye-mutiny-vertx-uri-template-2.29.0.jar
1 /app/lib/main/io.smallrye.reactive.smallrye-mutiny-vertx-web-2.29.0.jar
1 /app/lib/main/io.smallrye.reactive.smallrye-mutiny-vertx-web-common-2.29.0.jar
1 /app/lib/main/io.smallrye.reactive.vertx-mutiny-generator-2.29.0.jar
1 /app/lib/main/io.smallrye.smallrye-context-propagation-1.2.2.jar
1 /app/lib/main/io.smallrye.smallrye-context-propagation-api-1.2.2.jar
1 /app/lib/main/io.smallrye.smallrye-context-propagation-storage-1.2.2.jar
1 /app/lib/main/io.smallrye.smallrye-fault-tolerance-vertx-5.6.0.jar
1 /app/lib/main/io.smallrye.smallrye-health-3.3.1.jar
1 /app/lib/main/io.smallrye.smallrye-health-api-3.3.1.jar
1 /app/lib/main/io.smallrye.smallrye-health-provided-checks-3.3.1.jar
1 /app/lib/main/io.stackgres.stackgres-common-1.6.0.jar
1 /app/lib/main/io.stackgres.stackgres-operator-framework-1.6.0.jar
1 /app/lib/main/io.sundr.builder-annotations-0.94.0.jar
1 /app/lib/main/io.sundr.resourcecify-annotations-0.94.0.jar
1 /app/lib/main/io.sundr.sundr-adapter-api-0.94.0.jar
1 /app/lib/main/io.sundr.sundr-adapter-apt-0.94.0.jar
1 /app/lib/main/io.sundr.sundr-adapter-reflect-0.94.0.jar
1 /app/lib/main/io.sundr.sundr-codegen-api-0.94.0.jar
1 /app/lib/main/io.sundr.sundr-codegen-apt-0.94.0.jar
1 /app/lib/main/io.sundr.sundr-core-0.94.0.jar
1 /app/lib/main/io.sundr.sundr-model-0.94.0.jar
1 /app/lib/main/io.sundr.sundr-model-base-0.94.0.jar
1 /app/lib/main/io.sundr.sundr-model-repo-0.94.0.jar
1 /app/lib/main/io.sundr.sundr-model-utils-0.94.0.jar
1 /app/lib/main/io.vertx.vertx-auth-common-4.3.6.jar
1 /app/lib/main/io.vertx.vertx-bridge-common-4.3.6.jar
1 /app/lib/main/io.vertx.vertx-codegen-4.3.6.jar
1 /app/lib/main/io.vertx.vertx-core-4.3.6.jar
1 /app/lib/main/io.vertx.vertx-uri-template-4.3.6.jar
1 /app/lib/main/io.vertx.vertx-web-4.3.6.jar
1 /app/lib/main/io.vertx.vertx-web-common-4.3.6.jar
1 /app/lib/main/jakarta.annotation.jakarta.annotation-api-1.3.5.jar
1 /app/lib/main/jakarta.el.jakarta.el-api-3.0.3.jar
1 /app/lib/main/jakarta.enterprise.jakarta.enterprise.cdi-api-2.0.2.jar
1 /app/lib/main/jakarta.inject.jakarta.inject-api-1.0.jar
1 /app/lib/main/jakarta.interceptor.jakarta.interceptor-api-1.2.5.jar
1 /app/lib/main/jakarta.transaction.jakarta.transaction-api-1.3.3.jar
1 /app/lib/main/jakarta.validation.jakarta.validation-api-2.0.2.jar
1 /app/lib/main/org.apache.commons.commons-compress-1.22.jar
1 /app/lib/main/org.apache.httpcomponents.httpasyncclient-4.1.5.jar
1 /app/lib/main/org.apache.httpcomponents.httpclient-4.5.13.jar
1 /app/lib/main/org.apache.httpcomponents.httpcore-4.4.15.jar
1 /app/lib/main/org.apache.httpcomponents.httpcore-nio-4.4.15.jar
1 /app/lib/main/org.checkerframework.checker-qual-3.27.0.jar
1 /app/lib/main/org.eclipse.microprofile.config.microprofile-config-api-2.0.1.jar
1 /app/lib/main/org.eclipse.microprofile.context-propagation.microprofile-context-propagation-api-1.2.jar
1 /app/lib/main/org.eclipse.microprofile.health.microprofile-health-api-3.1.jar
1 /app/lib/main/org.eclipse.microprofile.rest.client.microprofile-rest-client-api-2.0.jar
1 /app/lib/main/org.glassfish.jakarta.el-3.0.4.jar
1 /app/lib/main/org.glassfish.jakarta.json-1.1.6.jar
1 /app/lib/main/org.hibernate.validator.hibernate-validator-6.2.5.Final.jar
1 /app/lib/main/org.jboss.logging.commons-logging-jboss-logging-1.0.0.Final.jar
1 /app/lib/main/org.jboss.logging.jboss-logging-annotations-2.2.1.Final.jar
1 /app/lib/main/org.jboss.resteasy.resteasy-cdi-4.7.7.Final.jar
1 /app/lib/main/org.jboss.resteasy.resteasy-client-4.7.7.Final.jar
1 /app/lib/main/org.jboss.resteasy.resteasy-client-api-4.7.7.Final.jar
1 /app/lib/main/org.jboss.resteasy.resteasy-client-microprofile-4.7.7.Final.jar
1 /app/lib/main/org.jboss.resteasy.resteasy-client-microprofile-base-4.7.7.Final.jar
1 /app/lib/main/org.jboss.resteasy.resteasy-core-4.7.7.Final.jar
1 /app/lib/main/org.jboss.resteasy.resteasy-core-spi-4.7.7.Final.jar
1 /app/lib/main/org.jboss.resteasy.resteasy-jackson2-provider-4.7.7.Final.jar
1 /app/lib/main/org.jboss.slf4j.slf4j-jboss-logmanager-1.2.0.Final.jar
1 /app/lib/main/org.jboss.spec.javax.ws.rs.jboss-jaxrs-api_2.1_spec-2.0.1.Final.jar
1 /app/lib/main/org.jboss.spec.javax.xml.bind.jboss-jaxb-api_2.3_spec-2.0.0.Final.jar
1 /app/lib/main/org.jboss.threads.jboss-threads-3.4.3.Final.jar
1 /app/lib/main/org.jooq.jool-0.9.15.jar
1 /app/lib/main/org.postgresql.postgresql-42.5.1.jar
1 /app/lib/main/org.reactivestreams.reactive-streams-1.0.3.jar
1 /app/lib/main/org.slf4j.slf4j-api-1.7.36.jar
1 /app/lib/main/org.yaml.snakeyaml-1.33.jar
1 /app/quarkus/generated-bytecode.jar
1 /app/quarkus-run.jar
1 /app/quarkus/transformed-bytecode.jar
1 /dev/null
1 /dev/random
1 /dev/urandom
1800 /etc/ssl
1 pipe:[147374691]
1 pipe:[147374692]
1 socket:[147372004]
1 socket:[147382369]
1 socket:[147386136]
1 socket:[147392760]
1 socket:[157365115]
1 /usr/lib/jvm/java-17-openjdk-17.0.9.0.9-2.el8.x86_64/lib/modules
Edited by Matteo Melli