GitLab believes in a world where everyone can contribute. As one of the company's core values, Diversity, Inclusion and Belonging (DIB) is fundamental to the success of GitLab. Team member diversity leads to better decisions and a greater sense of team member belonging. When you have a diverse, inclusive and belonging environment where team members feel safe to show up as their full selves, you empower everyone to contribute.
@@ -29,7 +29,7 @@ Given this dataset is small and read very often, we can expect data to nearly al
In the example of license data above, the query to read license data was [identified](https://gitlab.com/gitlab-org/gitlab/-/issues/292900) to stand out in terms of query frequency. In fact, we were seeing around 6,000 queries per second (qps) on the cluster during peak times. With the cluster size at that time, we were seeing about 1,000 qps on each replica and less than 400 qps on the primary at peak times. The difference is explained by our [database load balancing for scaling reads](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/database/load_balancing.rb), which favors replicas for pure read-only transactions.
The overall transaction throughput on the database primary at the time varied between 50,000 and 70,000 transactions per second (tps). In comparison to this, this query frequency only takes a small portion of the overall query frequency. However, we do expect this to still have considerable overhead in terms of context switches and hence it is worth removing this overhead, if we can.
@@ -60,15 +60,15 @@ avg by (relname, fqdn) (
This yields a good impression of which tables are much more often read than written (on the database primary):
From here, we can [zoom](https://thanos.gitlab.net/graph?g0.range_input=1d&g0.end_input=2021-04-07%2023%3A11&g0.max_source_resolution=0s&g0.expr=sum(rate(pg_stat_user_tables_idx_tup_fetch%7Benvironment%3D%22gprd%22%2C%20relname%3D%22gitlab_subscriptions%22%7D%5B1h%5D))&g0.tab=0&g1.range_input=1d&g1.end_input=2021-04-07%2023%3A11&g1.max_source_resolution=0s&g1.expr=sum(rate(pg_stat_user_tables_n_tup_ins%7Benvironment%3D%22gprd%22%2C%20relname%3D%22gitlab_subscriptions%22%7D%5B1h%5D))%20by%20(instance)%20%2B%20sum(rate(pg_stat_user_tables_n_tup_upd%7Benvironment%3D%22gprd%22%2C%20relname%3D%22gitlab_subscriptions%22%7D%5B1h%5D))%20by%20(instance)%20%2B%20%20sum(rate(pg_stat_user_tables_n_tup_del%7Benvironment%3D%22gprd%22%2C%20relname%3D%22gitlab_subscriptions%22%7D%5B1h%5D))%20by%20(instance)&g1.tab=0&g2.range_input=1h&g2.max_source_resolution=0s&g2.expr=pg_stat_user_tables_idx_tup&g2.tab=0) into e.g. `gitlab_subscriptions` and realize that index reads peak at above 10k tuples per second overall (there are no seq scans):
Additionally, the table is only 400 MB in size - so this may be another candidate we may want to consider in this pattern (see [#327483](https://gitlab.com/gitlab-org/gitlab/-/issues/327483)).
@@ -84,7 +84,7 @@ In order to reduce the database overhead, we implement a cache for the data and
Continuing the above example, we had a `RequestStore` in place to cache license information on a per-request basis. However, that still leads to one query per request. When we [started to cache license information using a process-wide in-memory cache](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50318) for 1 second, query frequency dramatically dropped:
The choice of caching here highly depends on the characteristics of data in question. A very small dataset like license data that is nearly never updated is a good candidate for in-memory caching. A per-process cache is favorable here since this unties the cache refresh rate from the incoming request rate.
@@ -31,7 +31,7 @@ This Working Group has the following goals:
As per Wikipedia, **MLOps** or **ML Ops** is a set of practices that aims to deploy and maintain machine learning models in production reliably and efficiently.
@@ -321,7 +321,7 @@ Additionally Cells may solve some issues with canaries today, for example not ha
A Cell is a set of infrastructure components that contains multiple top-level Groups that belong to different Organizations. The components include both data stores (PostgreSQL, Redis, etc.) and stateless services (web, etc.). The infrastructure components provided within a Cell are shared among Organizations and their top-level Groups, but are not shared with other Cells. This isolation of infrastructure components means that Cells are independent from each other.
An Organization is the umbrella for one or multiple top-level Groups. Organizations are isolated from each other by default meaning that cross-namespace features will only work for namespaces that exist in a single Organization.
See the [Organization blueprint](../organization/).
@@ -393,7 +393,7 @@ The top-level Group has served as the de facto Organization entity. With the cre
Over time, there won't be a distinction between a top-level Group and a Group. All features that make top-level Groups different from Groups will move to the Organization.
- There's a 1:n relationship between a Cell and a Runner, i.e, one Cell can have many Runners, while a Runner is only registered to one Cell.
- Legacy Cell (GitLab.com) is using Runners that are managed via [config-mgmt](https://gitlab.com/gitlab-com/gl-infra/config-mgmt) and [chef-repo](https://gitlab.com/gitlab-com/gl-infra/chef-repo), while the Cells are using [Transistor](https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/transistor/)-like setup, which vendors in GRIT.
