Ocamlnet Shell.call takes a VERY long time when max no of file descriptors is large

Gerd Stolpmann
2018-04-26T16:00:57Z

A simple Shell.(call [cmd "cat" ["/etc/passwd"]]) can take minutes to execute when the max number of file descriptors is large (100,000 or 1,000,000) as is the case by default on our large FreeBSD servers.

With ulimit -n 1024:

ocaml debug.ml > /dev/null  0.20s user 0.04s system 99% cpu 0.240 total

With the FreeBSD default (3772809 on this smaller server):

ocaml debug.ml > /dev/null  1.14s user 97.55s system 100% cpu 1:38.65  total

Tracing the slow-running executable shows thousands of dup2's fcntls and closes in groups like this:

 24442 debug.native RET   close 0
 24442 debug.native CALL  dup2(0x3,0x37733c)
 24442 debug.native RET   dup2 3633980/0x37733c
 24442 debug.native CALL  fcntl(0x37733c,F_SETFD,0)
 24442 debug.native RET   fcntl 0
 24442 debug.native CALL  close(0x37733c)

The problem doesn't occur for Sys.command "cat /etc/passwd" or Unix.execvp "cat" [|"cat";"/etc/passwd"|], or Unix.open_process_in. All of those run equally fast no matter how many file descriptors, but of course they don't have the nice high level abstraction that Shell provides. debug.ml