The 'grpc' gem is prone to breakage after process forking
Upstream issue: https://github.com/grpc/grpc/issues/10658
This bit us in particular with grpc 1.2.2. But it seems to be a general problem we have to deal with.
The gitlab-ce (and gitlab-ee) code base uses two concurrency mechanisms: pre-forking (Unicorn, for HTTP requests) and multi-threading (Sidekiq, for background jobs). Pre-forking is a crucial mechanism for us because it allows us to keep gitlab-ce memory use in check. Forking is not compatible with multi-threading within the same process (a child process forked from a multi-threaded parent will have only one thread). Network sockets opened by the parent can also cause problems when used in the child.
Unicorn has a mechanism to deal with this problem: the after_fork
hook. This allows you to re-start threads and re-open connections in a Unicorn worker process. Unfortunately, the 'grpc' gem currently does not have a mechanism for this: once its C library has been initialized you can run into trouble if you then fork the process. There is no way to 'repair' the forked child.
I am creating this issue so we can discuss how to deal with this in the gitlab-ce code.