Containerd: too many file descriptors

We need to diagnose and debug this issue. Is it that there were actually too many open file descriptors, or is this a red-herring. There were 54 active experiments on the testbed. Can this be reproduced in a single virtual machine? How many containers becomes the limit before reaching this error.

Running containerd v1.3.2

Jan 11 14:41:22 site1 rex[73187]: time="2022-01-11T14:41:22-08:00" level=error msg="task failed" error="stage failed" id=rVCqXHMVr
Jan 11 14:41:17 site1 rex[73187]: time="2022-01-11T14:41:17-08:00" level=error msg="pull image"
Jan 11 14:41:17 site1 rex[73187]: time="2022-01-11T14:41:17-08:00" level=error msg="pull image" caller="/builds/mergetb/tech/cogs/pkg/runtime/containers.go:92" error="unpack: failed to extract layer sha256:2a0ae9201830dc8c540407473a5551a11d818fe095a17cfc65095e60c1990512: open /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/69353/fs/usr/lib/python2.7/compiler/pyassem.pyc: too many open files: unknown"
Jan 11 14:41:17 site1 rex[73187]: time="2022-01-11T14:41:17-08:00" level=info msg="apply failure, attempting cleanup" error="failed to extract layer sha256:2a0ae9201830dc8c540407473a5551a11d818fe095a17cfc65095e60c1990512: open /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/69353/fs/usr/lib/python2.7/compiler/pyassem.pyc: too many open files: unknown" key="extract-562996232-Nvjs sha256:a64516a67e5c64fda244637a60a555d840acbc14586a89c98ebc4bb589cd1ea1"
Jan 11 14:41:12 site1 rex[73187]: time="2022-01-11T14:41:12-08:00" level=error msg="pull image"
Jan 11 14:41:12 site1 rex[73187]: time="2022-01-11T14:41:12-08:00" level=error msg="pull image" caller="/builds/mergetb/tech/cogs/pkg/runtime/containers.go:92" error="unpack: failed to extract layer sha256:16d5d206885d919d6213c9e3ff688cf6aea7516636292d98abfcbb2d62f33035: failed to get stream processor for application/vnd.docker.image.rootfs.diff.tar.gzip: pipe2: too many open files: unknown"
Jan 11 14:41:12 site1 rex[73187]: time="2022-01-11T14:41:12-08:00" level=info msg="apply failure, attempting cleanup" error="failed to extract layer sha256:16d5d206885d919d6213c9e3ff688cf6aea7516636292d98abfcbb2d62f33035: failed to get stream processor for application/vnd.docker.image.rootfs.diff.tar.gzip: pipe2: too many open files: unknown" key="extract-82929411-ypn5 sha256:16d5d206885d919d6213c9e3ff688cf6aea7516636292d98abfcbb2d62f33035"
Jan 11 14:41:12 site1 rex[73187]: time="2022-01-11T14:41:12-08:00" level=error msg="pull image"
Jan 11 14:41:12 site1 rex[73187]: time="2022-01-11T14:41:12-08:00" level=error msg="pull image" caller="/builds/mergetb/tech/cogs/pkg/runtime/containers.go:92" error="unpack: failed to extract layer sha256:8c02234b86056c009036ff0c31efb9a726412392d9872dacf95103767ac3b101: failed to get stream processor for application/vnd.docker.image.rootfs.diff.tar.gzip: fork/exec /usr/bin/unpigz: too many open files: unknown"
Jan 11 14:41:12 site1 rex[73187]: time="2022-01-11T14:41:12-08:00" level=info msg="apply failure, attempting cleanup" error="failed to extract layer sha256:8c02234b86056c009036ff0c31efb9a726412392d9872dacf95103767ac3b101: failed to get stream processor for application/vnd.docker.image.rootfs.diff.tar.gzip: fork/exec /usr/bin/unpigz: too many open files: unknown" key="extract-946430018-s2Zi sha256:8c02234b86056c009036ff0c31efb9a726412392d9872dacf95103767ac3b101"