Consider alternatives to Ruby grpc client
In terms of number of issues and debugging time, the gRPC ruby client has unfortunately been one of the most problematic components within Gitaly.
An incomplete list of issues we (and other GitLab teams) have had to deal with so far:
Memory Leaks
- Out of Memory
Unicorn / Forking Issues (related to memory leaks)
- Deadlock on worker termination:
Build/Compile Issues
- Compile issues:
- FreeBSD #154 (closed)
- OpenBSD https://gitlab.com/gitlab-org/gitlab-ce/issues/29562
- CentOS (protobuf issues, below)
- Protobuf Gemfile issues:
- https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9665 (workaround)
- omnibus-gitlab#2042 (closed)
- omnibus-gitlab!1369 (closed)
- https://github.com/google/protobuf/issues/2853
- https://github.com/google/protobuf/issues/2783
- https://github.com/google/protobuf/pull/2804
- https://gitlab.com/gitlab-com/infrastructure/issues/1354
- https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9932
- https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9971
- https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10014
- Gemfile issues
- https://github.com/grpc/grpc/issues/10058
- https://github.com/grpc/grpc/pull/10059
- https://github.com/grpc/grpc/issues/9998
- https://github.com/grpc/grpc/pull/10019
- https://gitlab.com/gitlab-org/gitlab-ce/issues/30766
- https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10589
- https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10700 (revert)
- General worries about grpc and protobuf
Currently, the unicorn threading issue and memory leak issue are yet to be resolved and a risk to our next delivery.
I feel that the grpc team has been doing a fantastic job of addressing these issues, but it remains a big risk for Gitaly.
I propose that we investigate ways to mitigate the risk posed by this component by looking at alternatives, while continuing to work at resolving the gRPC issues.
grpc-gateway
Option 1: Investigate grpc-gateway
is a gRPC to JSON proxy generator. It can emit Swagger definitions for the JSON proxy. We could then generate a Ruby client stub with a similar interface to the existing gRPC client stub that does not depend on gRPC C-bindings and uses native Ruby code only.
Advantages
- No portability issues (FreeBSD, CentOS, OpenBSD, etc) as we could generate a Ruby-only Swagger client
- Restful interfaces are a boring solution
- No threading issues
- Client is still autogenerated using Swagger
- Switching back to gRPC later on may be fairly simple, depending on how different the interface of the Swagger client stub is from the gRPC client stub
Disadvantages
- Extra step in proto generation requires additional cognitive overhead
- Possible performance impact
- Switching back to gRPC later on may be fairly difficult, depending on how different the interface of the Swagger client stub is from the gRPC client stub
Option 2:
Alternative proposals welcome