Patroni fails to start with MemoryError

Summary

Patroni fails to start with MemoryError and a stack trace.

Steps to reproduce

Unknown

I've raised this issue to solicit input, and to log it for future reference.

What is the current bug behavior?

When Omnibus starts Patroni, it fails to start with the following error, and this repeats frequently as Omnibus keeps trying to start it up.

MemoryError
Traceback (most recent call last):
  File "/opt/gitlab/embedded/bin/patroni", line 8, in <module>
    sys.exit(main())
  File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/__init__.py", line 170, in main
    return patroni_main()
  File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/__init__.py", line 138, in patroni_main
    abstract_main(Patroni, schema)
  File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/daemon.py", line 98, in abstract_main
    controller = cls(config)
  File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/__init__.py", line 30, in __init__
    self.watchdog = Watchdog(self.config)
  File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/watchdog/base.py", line 94, in __init__
    self.impl = self.config.get_impl()
  File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/watchdog/base.py", line 64, in get_impl
    from patroni.watchdog.linux import LinuxWatchdogDevice
  File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/watchdog/linux.py", line 2, in <module>
    import ctypes
  File "/opt/gitlab/embedded/lib/python3.7/ctypes/__init__.py", line 551, in <module>
    _reset_cache()
  File "/opt/gitlab/embedded/lib/python3.7/ctypes/__init__.py", line 273, in _reset_cache
    CFUNCTYPE(c_int)(lambda: None)

What is the expected correct behavior?

Patroni starts.

Relevant logs

Relevant logs

Details of package version

Omnibus 13.10.3

Environment details

• Operating System: RHEL 8.3
• Installation Target, remove incorrect values:
  • VM: Other VMWare
• Installation Type, remove incorrect values:
  • New Installation
• Is there any other software running on the machine: Various commercial logging, security agents etc.
• Is this a single or multiple node installation?
• Resources
  • CPU: 4 * Intel Xeon
  • Memory total: 8gb

While MemFree is low, swap is unused, and most of the memory is being used for Cache.

MemTotal:        7956120 kB
MemFree:          982880 kB
MemAvailable:    6937856 kB
Buffers:            8380 kB
Cached:          5424876 kB
SwapCached:            0 kB
Active:          5496916 kB
Inactive:         315368 kB
Active(anon):     372792 kB
Inactive(anon):      300 kB
Active(file):    5124124 kB
Inactive(file):   315068 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       4194300 kB
SwapFree:        4194300 kB
Dirty:               412 kB
Writeback:             0 kB
AnonPages:        346160 kB
Mapped:           314100 kB
Shmem:               648 kB
KReclaimable:     821072 kB
Slab:             976348 kB
SReclaimable:     821072 kB
SUnreclaim:       155276 kB
KernelStack:        6016 kB
PageTables:        12912 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     8172360 kB
Committed_AS:    1462612 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:             3536 kB
HardwareCorrupted:     0 kB
AnonHugePages:    133120 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      325504 kB
DirectMap2M:     8062976 kB
DirectMap1G:     2097152 kB

SeLinux is permissive, so if it was that, it'd only be logging there was a problem

SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          permissive
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      32

I've reviewed /etc/security/limits* - nothing to see there. The ulimit -a output looks like standard RHEL/Centos.

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 30931
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 30931
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

We have a GitLab SOS on the ticket, and I've looked at the logs and cannot see any other issues.

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb`