File Descriptor Leak to /usr/bin

Summary

The stackgres-cluster-controller process leaks file descriptors pointing to /usr/bin (a directory). The leak is persistent and slow but causes system-level issues over time, including lsof hanging and eventual exhaustion of open file descriptors.

Current Behaviour

After launching the controller pod, it begins with a low number of open file descriptors (e.g., ~150). Over time, this number steadily increases — by 5–10 descriptors every minute — reaching over 100,000 within days.

99.9% of the leaked file descriptors point to /usr/bin, as shown by:

find /proc/<pid>/fd -type l -exec readlink -f {} + | grep -v '^/usr/bin$' | wc -l

All the FDs look like this:

lr-x------ 1 999 systemd-journal 64 Aug 1 08:07 /proc/3412072/fd/1000 -> /usr/bin

Steps to reproduce

  • Deploy one or more StackGres clusters (we run ~10 on a single node in this instance).
  • Find the PID of the controller:
ps aux | grep stackgres-cluster-controller
  • Monitor open FDs over time:
while true; do
  ls /proc/<pid>/fd | wc -l
  sleep 10
done
  • The count increases slowly.
  • Nearly all FDs are read-only references to /usr/bin:
find /proc/<pid>/fd -type l -exec readlink -f {} + | grep '^/usr/bin$' | wc -l

Expected Behaviour

  • File descriptors should not be persistently opened to /usr/bin
  • All directory handles must be properly closed after use
  • The open FD count should remain relatively stable over time

Possible Solution

  • Check the controller logic for:
  • Repeated directory reads like new File("/usr/bin").listFiles() or Java NIO Files.walk(...) without proper closing
  • Periodic health checks or background threads that scan /usr/bin

Use of java.io.File.listFiles() or similar API in a loop or thread pool without resource cleanup

Environment

  • StackGres version: 1.16.2
  • Kubernetes version: v1.31.7
  • Cloud provider or hardware configuration: Scaleway Kubernetes (Kapsule), Ubuntu 24.04, containerd runtime

Relevant logs and/or screenshots

Not much visible in logs, but clear via these shell outputs:

Number of file descriptors increasing:

while true; do
  ls /proc/<pid>/fd | wc -l
  sleep 10
done

Typical output:

161
230
246
254
261
269
...

Count of leaked descriptors pointing to /usr/bin:

find /proc/<pid>/fd -type l -exec readlink -f {} + | grep '^/usr/bin$' | wc -l
# → 100000+ after several days

Confirming very few non-/usr/bin FDs:

find /proc/<pid>/fd -type l -exec readlink -f {} + | grep -v '^/usr/bin$' | wc -l
# → ~14

Temporary Workaround Restarting the affected stackgres-cluster-controller pods reclaims the leaked FDs. However, the leak resumes immediately afterward.