Skip to content
Snippets Groups Projects

Update Load Balancer sizing in Ref Arch docs

Merged Grant Young requested to merge gy-update-lb-sizings-ref-arch-docs into master
All threads resolved!
7 files
+ 233
239
Compare changes
  • Side-by-side
  • Inline
Files
7
@@ -28,34 +28,38 @@ specifically the [Before you start](index.md#before-you-start) and [Deciding whi
| Service | Nodes | Configuration | GCP | AWS | Azure |
|------------------------------------------|-------|-------------------------|------------------|----------------|-----------|
| External load balancing node <sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
| Consul <sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
| PostgreSQL <sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` |
| PgBouncer <sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
| Internal load balancing node <sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
| Redis/Sentinel - Cache <sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
| Redis/Sentinel - Persistent <sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
| Gitaly <sup>5</sup> | 3 | 16 vCPU, 60 GB memory <sup>6</sup> | `n1-standard-16` | `m5.4xlarge` | `D16s v3` |
| Praefect <sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
| Praefect PostgreSQL <sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
| Sidekiq <sup>7</sup> | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
| GitLab Rails <sup>7</sup> | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` |
| External load balancer<sup>3</sup> | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5n.xlarge` | `F4s v2` |
| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
| PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` |
| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
| Internal load balancer<sup>3</sup> | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5n.xlarge` | `F4s v2` |
| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
| Gitaly<sup>5</sup> | 3 | 16 vCPU, 60 GB memory<sup>6</sup> | `n1-standard-16` | `m5.4xlarge` | `D16s v3` |
| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
| Sidekiq<sup>7</sup> | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
| GitLab Rails<sup>7</sup> | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` |
| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` |
| Object storage <sup>4</sup> | - | - | - | - | - |
**Footnotes:**
<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix -->
<!-- markdownlint-disable MD029 -->
1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
1. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instances](#provide-your-own-redis-instances) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instances](#provide-your-own-redis-instances) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
- Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. For this size of architecture it's strongly recommended having separate Cache and Persistent instances as specified to achieve optimum performance.
1. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
1. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information.
1. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management.
3. Recommended to be run with a reputable third-party load balancer or service (LB PaaS) which can provide HA capabilities.
Also note that sizing depends on selected Load Balancer as well as additional factors such as Network Bandwidth. Refer to [Load Balancers](index.md#load-balancers) for more information.
4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information.
5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management.
Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`.
1. Gitaly specifications are based on high percentiles of both usage patterns and repository sizes in good health.
6. Gitaly specifications are based on high percentiles of both usage patterns and repository sizes in good health.
However, if you have [large monorepos](index.md#large-monorepos) (larger than several gigabytes) or [additional workloads](index.md#additional-workloads) these can *significantly* impact Git and Gitaly performance and further adjustments will likely be required.
1. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes).
7. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes).
However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node.
<!-- markdownlint-enable MD029 -->
NOTE:
For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices.
@@ -238,22 +242,14 @@ The following list includes descriptions of each server and its assigned IP:
## Configure the external load balancer
In a multi-node GitLab configuration, you'll need a load balancer to route
In a multi-node GitLab configuration, you'll need an external load balancer to route
traffic to the application servers.
The specifics on which load balancer to use, or its exact configuration
is beyond the scope of GitLab documentation. It is expected however that any
reputable load balancer should work and as such this section will focus on the specifics of
is beyond the scope of GitLab documentation but refer to [Load Balancers](index.md) for more information around
general requirements. This section will focus on the specifics of
what to configure for your load balancer of choice.
### Balancing algorithm
We recommend that a least-connection load balancing algorithm or equivalent
is used wherever possible to ensure equal spread of calls to the nodes and good performance.
We don't recommend the use of round-robin algorithms as they are known to not
spread connections equally in practice.
### Readiness checks
Ensure the external load balancer only routes to working services with built
@@ -365,10 +361,14 @@ for details on managing SSL certificates and configuring NGINX.
## Configure the internal load balancer
The Internal Load Balancer is used to balance any internal connections the GitLab environment requires
In a multi-node GitLab configuration, you'll need an internal load balancer to route
traffic for select internal components if configured
such as connections to [PgBouncer](#configure-pgbouncer) and [Praefect](#configure-praefect) (Gitaly Cluster).
It's a separate node from the External Load Balancer and shouldn't have any access externally.
The specifics on which load balancer to use, or its exact configuration
is beyond the scope of GitLab documentation but refer to [Load Balancers](index.md) for more information around
general requirements. This section will focus on the specifics of
what to configure for your load balancer of choice.
The following IP will be used as an example:
@@ -422,14 +422,6 @@ backend praefect
Refer to your preferred Load Balancer's documentation for further guidance.
### Balancing algorithm
We recommend that a least-connection-based load balancing algorithm or equivalent
is used wherever possible to ensure equal spread of calls to the nodes and good performance.
We don't recommend the use of round-robin algorithms as they are known to not
spread connections equally in practice.
<div align="right">
<a type="button" class="btn btn-default" href="#setup-components">
Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i>
@@ -2293,28 +2285,32 @@ services where applicable):
| Service | Nodes | Configuration | GCP | AWS |
|------------------------------------------|-------|-----------------------|------------------|--------------|
| Consul <sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
| PostgreSQL <sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` |
| PgBouncer <sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
| Internal load balancing node <sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
| Redis/Sentinel - Cache <sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` |
| Redis/Sentinel - Persistent <sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` |
| Gitaly <sup>5</sup> | 3 | 16 vCPU, 60 GB memory <sup>6</sup> | `n1-standard-16` | `m5.4xlarge` |
| Praefect <sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
| Praefect PostgreSQL <sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
| Object storage <sup>4</sup> | - | - | - | - |
| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
| PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` |
| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
| Internal load balancer<sup>3</sup> | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5n.xlarge` |
| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` |
| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` |
| Gitaly<sup>5</sup> | 3 | 16 vCPU, 60 GB memory<sup>6</sup> | `n1-standard-16` | `m5.4xlarge` |
| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` |
| Object storage<sup>4</sup> | - | - | - | - |
**Footnotes:**
<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix -->
<!-- markdownlint-disable MD029 -->
1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
1. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instances](#provide-your-own-redis-instances) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Provide your own Redis instances](#provide-your-own-redis-instances) and [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
- Redis is primarily single threaded and doesn't significantly benefit from an increase in CPU cores. For this size of architecture it's strongly recommended having separate Cache and Persistent instances as specified to achieve optimum performance.
1. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information.
1. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information.
1. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management.
3. Recommended to be run with a reputable third-party load balancer or service (LB PaaS) which can provide HA capabilities.
Also note that sizing depends on selected Load Balancer as well as additional factors such as Network Bandwidth. Refer to [Load Balancers](index.md#load-balancers) for more information.
4. Should be run on reputable Cloud Provider or Self Managed solutions. See [Configure the object storage](#configure-the-object-storage) for more information.
5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management.
Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`.
1. Gitaly specifications are based on high percentiles of both usage patterns and repository sizes in good health.
6. Gitaly specifications are based on high percentiles of both usage patterns and repository sizes in good health.
However, if you have [large monorepos](index.md#large-monorepos) (larger than several gigabytes) or [additional workloads](index.md#additional-workloads) these can *significantly* impact Git and Gitaly performance and further adjustments will likely be required.
<!-- markdownlint-enable MD029 -->
NOTE:
For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices.
Loading