index.md 16.2 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
# GitLab.com settings

In this page you will find information about the settings that are used on
[GitLab.com](https://about.gitlab.com/pricing).

## SSH host keys fingerprints

Below are the fingerprints for GitLab.com's SSH host keys.

| Algorithm | MD5 | SHA256  |
| --------- | --- | ------- |
|  DSA      | `7a:47:81:3a:ee:89:89:64:33:ca:44:52:3d:30:d4:87` | `p8vZBUOR0XQz6sYiaWSMLmh0t9i8srqYKool/Xfdfqw` |
|  ECDSA    | `f1:d0:fb:46:73:7a:70:92:5a:ab:5d:ef:43:e2:1c:35` | `HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw` |
|  ED25519  | `2e:65:6a:c8:cf:bf:b2:8b:9a:bd:6d:9f:11:5c:12:16` | `eUXGGm1YGsMAS7vkcx6JOJdOGHPem5gQp4taiCfCLB8` |
|  RSA      | `b6:03:0e:39:97:9e:d0:e7:24:ce:a3:77:3e:01:42:09` | `ROQFvPThGrW4RuWLoL9tq9I9zJ42fK4XywyRtbOz/EQ` |

## Mail configuration

GitLab.com sends emails from the `mg.gitlab.com` domain via [Mailgun] and has
its own dedicated IP address (`198.61.254.240`).

## Alternative SSH port

GitLab.com can be reached via a [different SSH port][altssh] for `git+ssh`.

| Setting     | Value               |
| ---------   | ------------------- |
| `Hostname`  | `altssh.gitlab.com` |
| `Port`      | `443`               |

An example `~/.ssh/config` is the following:

```
Host gitlab.com
  Hostname altssh.gitlab.com
  User git
  Port 443
  PreferredAuthentications publickey
  IdentityFile ~/.ssh/gitlab
```

## GitLab Pages

Below are the settings for [GitLab Pages].

| Setting                 | GitLab.com        | Default       |
| ----------------------- | ----------------  | ------------- |
| Domain name             | `gitlab.io`       | -             |
49
| IP address              | `35.185.44.232`   | -             |
50 51 52 53
| Custom domains support  | yes               | no            |
| TLS certificates support| yes               | no            |

The maximum size of your Pages site is regulated by the artifacts maximum size
54
which is part of [GitLab CI/CD](#gitlab-cicd).
55 56 57 58 59 60 61 62

## GitLab CI/CD

Below are the current settings regarding [GitLab CI/CD](../../ci/README.md).

| Setting                 | GitLab.com        | Default       |
| -----------             | ----------------- | ------------- |
| Artifacts maximum size  | 1G                | 100M          |
63
| Artifacts [expiry time](../../ci/yaml/README.md#artifactsexpire_in)   | kept forever           | deleted after 30 days unless otherwise specified    |
64

65 66
## Repository size limit

67 68
The maximum size your Git repository is allowed to be, including LFS. If you are near
or over the size limit, you can [reduce your repository size with Git](../project/repository/reducing_the_repo_size_using_git.md).
69 70 71 72 73

| Setting                 | GitLab.com        | Default       |
| -----------             | ----------------- | ------------- |
| Repository size including LFS | 10G         | Unlimited     |

Cynthia Ng's avatar
Cynthia Ng committed
74 75 76 77 78 79 80 81
## IP range

GitLab.com, CI/CD, and related services are deployed into Google Cloud Platform (GCP). Any 
IP based firewall can be configured by looking up all 
[IP address ranges or CIDR blocks for GCP](https://cloud.google.com/compute/docs/faq#where_can_i_find_product_name_short_ip_ranges). 

[Static endpoints](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5071) are being considered.

82 83 84
## Shared Runners

Shared Runners on GitLab.com run in [autoscale mode] and powered by
Sid Sijbrandij's avatar
Sid Sijbrandij committed
85
Google Cloud Platform. Autoscaling means reduced
86 87
waiting times to spin up CI/CD jobs, and isolated VMs for each project,
thus maximizing security.
88 89 90 91
They're free to use for public open source projects and limited to 2000 CI
minutes per month per group for private projects. Read about all
[GitLab.com plans](https://about.gitlab.com/pricing/).

Sid Sijbrandij's avatar
Sid Sijbrandij committed
92
All your CI/CD jobs run on [n1-standard-1 instances](https://cloud.google.com/compute/docs/machine-types) with 3.75GB of RAM, CoreOS and the latest Docker Engine
93 94
installed. Instances provide 1 vCPU and 25GB of HDD disk space. The default
region of the VMs is US East1.
Sid Sijbrandij's avatar
Sid Sijbrandij committed
95
Each instance is used only for one job, this ensures any sensitive data left on the system can't be accessed by other people their CI jobs.
96

97 98 99 100
Jobs handled by the shared Runners on GitLab.com (`shared-runners-manager-X.gitlab.com`),
**will be timed out after 3 hours**, regardless of the timeout configured in a
project. Check the issues [4010] and [4070] for the reference.

101 102 103 104 105 106
Below are the shared Runners settings.

| Setting                               | GitLab.com                                        | Default    |
| -----------                           | -----------------                                 | ---------- |
| [GitLab Runner]                       | [Runner versions dashboard][ci_version_dashboard] | -          |
| Executor                              | `docker+machine`                                  | -          |
107
| Default Docker image                  | `ruby:2.5`                                        | -          |
108 109
| `privileged` (run [Docker in Docker]) | `true`                                            | `false`    |

110
[ci_version_dashboard]: https://dashboards.gitlab.com/dashboard/db/ci?from=now-1h&to=now&refresh=5m&orgId=1&panelId=12&fullscreen&theme=light
111 112 113 114 115

### `config.toml`

The full contents of our `config.toml` are:

116 117
**DigitalOcean**

118
```toml
119 120 121 122 123
concurrent = X
check_interval = 1
metrics_server = "X"
sentry_dsn = "X"

124 125 126
[[runners]]
  name = "docker-auto-scale"
  request_concurrency = X
127
  url = "https://gitlab.com/"
128 129 130 131 132
  token = "SHARED_RUNNER_TOKEN"
  executor = "docker+machine"
  environment = [
    "DOCKER_DRIVER=overlay2"
  ]
133
  limit = X
134
  [runners.docker]
135
    image = "ruby:2.5"
136 137
    privileged = true
  [runners.machine]
138
    IdleCount = 20
139
    IdleTime = 1800
140 141 142 143
    OffPeakPeriods = ["* * * * * sat,sun *"]
    OffPeakTimezone = "UTC"
    OffPeakIdleCount = 5
    OffPeakIdleTime = 1800
144
    MaxBuilds = 1
145
    MachineName = "srm-%s"
146 147
    MachineDriver = "digitalocean"
    MachineOptions = [
148
      "digitalocean-image=X",
149 150
      "digitalocean-ssh-user=core",
      "digitalocean-region=nyc1",
151
      "digitalocean-size=s-2vcpu-2gb",
152
      "digitalocean-private-networking",
153 154 155
      "digitalocean-tags=shared_runners,gitlab_com",
      "engine-registry-mirror=http://INTERNAL_IP_OF_OUR_REGISTRY_MIRROR",
      "digitalocean-access-token=DIGITAL_OCEAN_ACCESS_TOKEN",
156 157 158
    ]
  [runners.cache]
    Type = "s3"
159 160 161 162
    BucketName = "runner"
    Insecure = true
    Shared = true
    ServerAddress = "INTERNAL_IP_OF_OUR_CACHE_SERVER"
163 164
    AccessKey = "ACCESS_KEY"
    SecretKey = "ACCESS_SECRET_KEY"
165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210
```

**Google Cloud Platform**

```toml
concurrent = X
check_interval = 1
metrics_server = "X"
sentry_dsn = "X"

[[runners]]
  name = "docker-auto-scale"
  request_concurrency = X
  url = "https://gitlab.com/"
  token = "SHARED_RUNNER_TOKEN"
  executor = "docker+machine"
  environment = [
    "DOCKER_DRIVER=overlay2"
  ]
  limit = X
  [runners.docker]
    image = "ruby:2.5"
    privileged = true
  [runners.machine]
    IdleCount = 20
    IdleTime = 1800
    OffPeakPeriods = ["* * * * * sat,sun *"]
    OffPeakTimezone = "UTC"
    OffPeakIdleCount = 5
    OffPeakIdleTime = 1800
    MaxBuilds = 1
    MachineName = "srm-%s"
    MachineDriver = "google"
    MachineOptions = [
      "google-project=PROJECT",
      "google-disk-size=25",
      "google-machine-type=n1-standard-1",
      "google-username=core",
      "google-tags=gitlab-com,srm",
      "google-use-internal-ip",
      "google-zone=us-east1-d",
      "google-machine-image=PROJECT/global/images/IMAGE",
      "engine-registry-mirror=http://INTERNAL_IP_OF_OUR_REGISTRY_MIRROR"
    ]
  [runners.cache]
    Type = "s3"
211
    BucketName = "runner"
212
    Insecure = true
213
    Shared = true
214 215 216
    ServerAddress = "INTERNAL_IP_OF_OUR_CACHE_SERVER"
    AccessKey = "ACCESS_KEY"
    SecretKey = "ACCESS_SECRET_KEY"
217 218 219 220 221 222 223 224 225 226 227 228 229 230 231
```

## Sidekiq

GitLab.com runs [Sidekiq][sidekiq] with arguments `--timeout=4 --concurrency=4`
and the following environment variables:

| Setting                                 | GitLab.com | Default   |
|--------                                 |----------- |--------   |
| `SIDEKIQ_MEMORY_KILLER_MAX_RSS`         | `1000000`  | `1000000` |
| `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_SIGNAL` | `SIGKILL`  | -         |
| `SIDEKIQ_LOG_ARGUMENTS`                 | `1`        | -         |

## Cron jobs

232
Periodically executed jobs by Sidekiq, to self-heal GitLab, do external
233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291
synchronizations, run scheduled pipelines, etc.:

| Setting                     | GitLab.com   | Default      |
|--------                     |------------- |------------- |
| `pipeline_schedule_worker`  | `19 * * * *` | `19 * * * *` |

## PostgreSQL

GitLab.com being a fairly large installation of GitLab means we have changed
various PostgreSQL settings to better suit our needs. For example, we use
streaming replication and servers in hot-standby mode to balance queries across
different database servers.

The list of GitLab.com specific settings (and their defaults) is as follows:

| Setting                             | GitLab.com                                                          | Default                               |
|:------------------------------------|:--------------------------------------------------------------------|:--------------------------------------|
| archive_command                     | `/usr/bin/envdir /etc/wal-e.d/env /opt/wal-e/bin/wal-e wal-push %p` | empty                                 |
| archive_mode                        | on                                                                  | off                                   |
| autovacuum_analyze_scale_factor     | 0.01                                                                | 0.01                                  |
| autovacuum_max_workers              | 6                                                                   | 3                                     |
| autovacuum_vacuum_cost_limit        | 1000                                                                | -1                                    |
| autovacuum_vacuum_scale_factor      | 0.01                                                                | 0.02                                  |
| checkpoint_completion_target        | 0.7                                                                 | 0.9                                   |
| checkpoint_segments                 | 32                                                                  | 10                                    |
| effective_cache_size                | 338688MB                                                            | Based on how much memory is available |
| hot_standby                         | on                                                                  | off                                   |
| hot_standby_feedback                | on                                                                  | off                                   |
| log_autovacuum_min_duration         | 0                                                                   | -1                                    |
| log_checkpoints                     | on                                                                  | off                                   |
| log_line_prefix                     | `%t [%p]: [%l-1] `                                                  | empty                                 |
| log_min_duration_statement          | 1000                                                                | -1                                    |
| log_temp_files                      | 0                                                                   | -1                                    |
| maintenance_work_mem                | 2048MB                                                              | 16 MB                                 |
| max_replication_slots               | 5                                                                   | 0                                     |
| max_wal_senders                     | 32                                                                  | 0                                     |
| max_wal_size                        | 5GB                                                                 | 1GB                                   |
| shared_buffers                      | 112896MB                                                            | Based on how much memory is available |
| shared_preload_libraries            | pg_stat_statements                                                  | empty                                 |
| shmall                              | 30146560                                                            | Based on the server's capabilities    |
| shmmax                              | 123480309760                                                        | Based on the server's capabilities    |
| wal_buffers                         | 16MB                                                                | -1                                    |
| wal_keep_segments                   | 512                                                                 | 10                                    |
| wal_level                           | replica                                                             | minimal                               |
| statement_timeout                   | 15s                                                                 | 60s                                   |
| idle_in_transaction_session_timeout | 60s                                                                 | 60s                                   |

Some of these settings are in the process being adjusted. For example, the value
for `shared_buffers` is quite high and as such we are looking into adjusting it.
More information on this particular change can be found at
<https://gitlab.com/gitlab-com/infrastructure/issues/1555>. An up to date list
of proposed changes can be found at
<https://gitlab.com/gitlab-com/infrastructure/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=database&label_name[]=change>.

## Unicorn

GitLab.com adjusts the memory limits for the [unicorn-worker-killer][unicorn-worker-killer] gem.

Base default:
292 293 294

- `memory_limit_min` = 750MiB
- `memory_limit_max` = 1024MiB
295 296

Web front-ends:
297 298 299

- `memory_limit_min` = 1024MiB
- `memory_limit_max` = 1280MiB
300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352

## GitLab.com at scale

In addition to the GitLab Enterprise Edition Omnibus install, GitLab.com uses
the following applications and settings to achieve scale. All settings are
located publicly available [chef cookbooks](https://gitlab.com/gitlab-cookbooks).

### ELK

We use Elasticsearch, logstash, and Kibana for part of our monitoring solution:

- [gitlab-cookbooks / gitlab-elk · GitLab](https://gitlab.com/gitlab-cookbooks/gitlab-elk)
- [gitlab-cookbooks / gitlab_elasticsearch · GitLab](https://gitlab.com/gitlab-cookbooks/gitlab_elasticsearch)

### Prometheus

Prometheus complete our monitoring stack:

- [gitlab-cookbooks / gitlab-prometheus · GitLab](https://gitlab.com/gitlab-cookbooks/gitlab-prometheus)

### Grafana

For the visualization of monitoring data:

- [gitlab-cookbooks / gitlab-grafana · GitLab](https://gitlab.com/gitlab-cookbooks/gitlab-grafana)

### Sentry

Open source error tracking:

- [gitlab-cookbooks / gitlab-sentry · GitLab](https://gitlab.com/gitlab-cookbooks/gitlab-sentry)

### Consul

Service discovery:

- [gitlab-cookbooks / gitlab_consul · GitLab](https://gitlab.com/gitlab-cookbooks/gitlab_consul)

### Haproxy

High Performance TCP/HTTP Load Balancer:

- [gitlab-cookbooks / gitlab-haproxy · GitLab](https://gitlab.com/gitlab-cookbooks/gitlab-haproxy)

[autoscale mode]: https://docs.gitlab.com/runner/configuration/autoscale.html "How Autoscale works"
[runners-post]: https://about.gitlab.com/2016/04/05/shared-runners/ "Shared Runners on GitLab.com"
[GitLab Runner]: https://gitlab.com/gitlab-org/gitlab-runner
[altssh]: https://about.gitlab.com/2016/02/18/gitlab-dot-com-now-supports-an-alternate-git-plus-ssh-port/ "GitLab.com now supports an alternate git+ssh port"
[GitLab Pages]: https://about.gitlab.com/features/pages "GitLab Pages"
[docker in docker]: https://hub.docker.com/_/docker/ "Docker in Docker at DockerHub"
[mailgun]: https://www.mailgun.com/ "Mailgun website"
[sidekiq]: http://sidekiq.org/ "Sidekiq website"
[unicorn-worker-killer]: https://rubygems.org/gems/unicorn-worker-killer "unicorn-worker-killer"
353 354
[4010]: https://gitlab.com/gitlab-com/infrastructure/issues/4010 "Find a good value for maximum timeout for Shared Runners"
[4070]: https://gitlab.com/gitlab-com/infrastructure/issues/4070 "Configure per-runner timeout for shared-runners-manager-X on GitLab.com"