GitLab CI/CD service daemon
Description
We recently talked with @ayufan about the expected CI/CD performance in the future. I also had a nice friendly chat about it with a friend who has a lot of experience regarding improving performance and designing salable services. Other people also raise this idea from time to time, so let's create an issue to write it down somewhere, and to have a place to discuss GitLab CI/CD service daemon.
GitLab CI/CD is currently implemented mostly on a Rails side, except GitLab Runner, which is a separate service, because of obvious reasons.
The more features we add to rails app, the more difficult it becomes to foresee future performance and availability implications. The implementation is currently quite complex (long polling, fair usage etc).
One of the directions for improving future performance / availability / scalability is to extract most of complex and domain-specific CI/CD implementation to a separate microservice - GitLab CI/CD daemon.
GitLab CI/CD daemon could be responsible for:
- Implementing Runner -> GitLab communication protocol, through an API.
- Implementing other GitLab <-> Runner communication protocols, like websockets, other pub/sub mechanism.
- Exposing jobs API and evenly distributing jobs across Runners fleet.
- Receiving and storing jobs' traces using various protocols, API, GRPC.
- Implementing various performance improvement techniques, like exclusive locks etc.
- Updating rails app and database to preserve as much resources as possible.
We talked about with @ayufan and we might think about something like this in the future, maybe in a year from now given the current growth.
For now there are more simple/boring solutions available, like introducing performance improvements using Redis etc.
/cc @stanhu @bikebilly