Bst open too many files which causes `OSError`s for the next calls

Summary

While trying to build gnome-build-meta today, I hit the following bug. bst seems like it have opened too many File descriptors and the rest syscalls failed with OSError: [Errno 24] Too many open files

Steps to reproduce

Not really sure how to reproduce it, It only happened once thus far. I had a recent checkout and build artifacts of the project, did a git pull and tried building again. Most of the fetches should have pulled down little if anything at all.

What is the current bug behavior?

For each pull task the following happened: An unhandled exception is thrown that keeps. Upon control-C neither quit nor terminate options seem to be able to exit.

What is the expected correct behavior?

Buildstream never gets to have too many files open at which point the syscalls start to fail.

Relevant logs and/or screenshots

I couldn't copy more than a task due to a weird bug with my terminal, but the following trace was identical for the rest of the too.

Unknown exception in SIGCHLD handler
Traceback (most recent call last):
  File "/usr/lib64/python3.6/asyncio/unix_events.py", line 819, in _sig_chld
  File "/usr/lib64/python3.6/asyncio/unix_events.py", line 885, in _do_waitpid_all
  File "/usr/lib64/python3.6/asyncio/unix_events.py", line 919, in _do_waitpid
  File "/usr/lib/python3.6/site-packages/buildstream/_scheduler/jobs/job.py", line 557, in _parent_child_completed
  File "/usr/lib/python3.6/site-packages/buildstream/_scheduler/scheduler.py", line 238, in job_completed
  File "/usr/lib/python3.6/site-packages/buildstream/_scheduler/scheduler.py", line 288, in _schedule_queue_jobs
  File "/usr/lib/python3.6/site-packages/buildstream/_scheduler/queues/queue.py", line 174, in enqueue
  File "/usr/lib/python3.6/site-packages/buildstream/_scheduler/queues/queue.py", line 174, in <listcomp>
  File "/usr/lib/python3.6/site-packages/buildstream/_scheduler/queues/queue.py", line 171, in create_job
  File "/usr/lib/python3.6/site-packages/buildstream/_scheduler/jobs/elementjob.py", line 69, in __init__
  File "/usr/lib/python3.6/site-packages/buildstream/_scheduler/jobs/job.py", line 112, in __init__
[--:--:--][????????][ pull:bootstrap-junction.bst:coreutils.bst] START   freedesktop-sdk-bootstrap/coreutils/????????-pull.4976.log
[--:--:--][????????][ pull:bootstrap-junction.bst:coreutils.bst] INFO    Pulling ???????? <- https://cache.gitlab.gnome.org:11001
[--:--:--][????????][ pull:bootstrap-junction.bst:coreutils.bst] INFO    Pulling ???????? <- https://testcache.codethink.co.uk:11001
[--:--:--][????????][ pull:bootstrap-junction.bst:coreutils.bst] INFO    Pulling ???????? <- https://cache.gitlab.gnome.org:11001
[--:--:--][????????][ pull:bootstrap-junction.bst:coreutils.bst] INFO    Pulling ???????? <- https://testcache.codethink.co.uk:11001
[00:00:01][????????][ pull:bootstrap-junction.bst:coreutils.bst] SUCCESS freedesktop-sdk-bootstrap/coreutils/????????-pull.4976.log
[--:--:--][????????][ main:bootstrap-junction.bst:coreutils.bst] BUG     Unhandled exception in post processing

    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/buildstream/_scheduler/queues/queue.py", line 310, in _job_done
      File "/usr/lib/python3.6/site-packages/buildstream/_scheduler/queues/pullqueue.py", line 65, in done
      File "/usr/lib/python3.6/site-packages/buildstream/_scheduler/scheduler.py", line 332, in _check_cache_size_real
      File "/usr/lib/python3.6/site-packages/buildstream/_scheduler/jobs/cachesizejob.py", line 25, in __init__
      File "/usr/lib/python3.6/site-packages/buildstream/_scheduler/jobs/job.py", line 112, in __init__
      File "/usr/lib64/python3.6/multiprocessing/context.py", line 102, in Queue
      File "/usr/lib64/python3.6/multiprocessing/queues.py", line 42, in __init__
      File "/usr/lib64/python3.6/multiprocessing/context.py", line 67, in Lock
      File "/usr/lib64/python3.6/multiprocessing/synchronize.py", line 163, in __init__
      File "/usr/lib64/python3.6/multiprocessing/synchronize.py", line 60, in __init__
    OSError: [Errno 24] Too many open files

Possible fixes

Other relevant information

• BuildStream version affected: 1.1.6 /milestone %BuildStream_v1.1

---