WIP: Make it simple to build and use "Decoupled Services"
TL;DR
This issue describes a concrete proposal how to enable GitLab to use a composable architecture with easy to build Microservices, but keeping the benefits of single codebase
This is multi-dimensional problem. It tries to describe the end goal, but proposes a steps to reach the end goal. This continues using single codebase, but propose to make it more component oriented. The components are being a Gems, with lifecycle being managed by the application.
Problem
We don't know how to easily build composable and smaller components, as this involves a multiple stakeholders in order to run. Thus we fallback to extend Monolith by default, instead of making us to build interfaces/APIs to interact between parts of application.
GitLab grows in complexity due to it's monolithic architecture:
- one big codebase,
- one big database,
- one big test suite.
This introduces a number of problems:
- you have to test everything to ensure that it continues working,
- there's no clear domain-oriented code structure that would allow contain some features,
- there's a number of drop-bags namespaces, like
lib/, - we have to run everything and load everything to run application.
- single monolith is harder to manage, the boring solution says add to monolith refactor later,
- adding to single monolith makes us to not care that much on writing re-usable interfaces, creating abstractions, using a well documented APIs, as all data are easily accessible,
- creating a Microservice is job near to impossible, as number of components that needs to be updated is astonishing: 1. create new repository, 2. introduce CONTRIBUTING and code practices, 3. connect versioning between GitLab and Microservice, 4. delegate maintainers for projects, 5. update Omnibus, source installation, Cloud Native, 6. make it observable, 7. create runbooks, 8. make Infrastructure team be aware and allow to manage it,
- measuring memory consumption is very hard problem, as you have multiple responsibilities in single codebase
- optimising application is very hard problem, as you have to be aware of multiple factors when running application,
- moving some parts of application into separate services/microservices as above is mission impossible,
This results in:
- long testing pipelines,
- a big flat codebase,
- hard to manage and run components,
- we have to load everything, even if most of application is not used,
- even the smallest failure in single part of application usually results in catastrophic failure of the whole application, as this is interconnected body,
- optimising and moving some components of the application usually results in a multi-month efforts with one of the most knowledgeable engineers at GitLab,
The state today
GitLab consists of multiple components today:
- GitLab Rails CE and EE: the big monolith that rarely interfaces with other gems, a. Unicorn/Puma: to serve all types of web requests, b. Sidekiq: to process background jobs,
- Some our Gems: https://gitlab.com/gitlab-org/labkit-ruby, https://gitlab.com/gitlab-org/gitlab_emoji, https://gitlab.com/gitlab-org/gitlab-styles, https://gitlab.com/gitlab-org/gitlab-sidekiq-fetcher, https://gitlab.com/gitlab-org/gitlab-markup.
- GitLab Workhorse: https://gitlab.com/gitlab-org/gitlab-workhorse: effectively Nginx proxy in front of Unicorn,
- GitLab Pages: https://gitlab.com/gitlab-org/gitlab-pages/,
- Gitaly and corresponding Ruby-gem: https://gitlab.com/gitlab-org/gitaly/
We currently work on merging CE and EE codebase into a single GitLab Codebase as part of: https://gitlab.com/gitlab-org/gitlab-ee/issues/2952. This seems to be progressing well and be very near the completion.
Description of the proposal
TBD
Steps
TBD