GitLab Geo configuration

Note: This is the documentation for installations from source. For installations using the Omnibus GitLab packages, follow the Omnibus GitLab Geo nodes configuration guide.

Note: Stages of the setup process must be completed in the documented order. Before attempting the steps in this stage, complete all prior stages.

  1. Install GitLab Enterprise Edition on the server that will serve as the secondary Geo node. Do not login or set up anything else in the secondary node for the moment.
  2. Upload the GitLab License you purchased for GitLab Enterprise Edition to unlock GitLab Geo.
  3. Setup the database replication (primary (read-write) <-> secondary (read-only) topology).
  4. Configure SSH authorizations to use the database
  5. Configure GitLab to set the primary and secondary nodes.
  6. Follow the after setup steps.

This is the final step you need to follow in order to setup a Geo node.

You are encouraged to first read through all the steps before executing them in your testing/production environment.

Setting up GitLab

Notes:

  • Do not setup any custom authentication in the secondary nodes, this will be handled by the primary node.
  • Do not add anything in the secondaries Geo nodes admin area (Admin Area ➔ Geo Nodes). This is handled solely by the primary node.

After having installed GitLab Enterprise Edition in the instance that will serve as a Geo node and set up the database replication, the next steps can be summed up to:

  1. Configure the primary node
  2. Replicate some required configurations between the primary and the secondaries
  3. Configure a second, tracking database on each secondary
  4. Configure every secondary node in the primary's Admin screen
  5. Start GitLab on the secondary node's machine

Prerequisites

This is the last step of configuring a Geo node. Make sure you have followed the first two steps of the Setup instructions:

  1. You have already installed on the secondary server the same version of GitLab Enterprise Edition that is present on the primary server.
  2. You have set up the database replication.
  3. Your secondary node is allowed to communicate via HTTP/HTTPS with your primary node (make sure your firewall is not blocking that).
  4. Your nodes must have an NTP service running to synchronize the clocks. You can use different timezones, but the hour relative to UTC can't be more than 60 seconds off from each node.
  5. You have set up another PostgreSQL database that can store writes for the secondary. Note that this MUST be on another instance, since the primary replicated database is read-only.

Some of the following steps require to configure the primary and secondary nodes almost at the same time. For your convenience make sure you have SSH logins opened on all nodes as we will be moving back and forth.

Step 1. Adding the primary GitLab node

  1. SSH into the primary node and login as root:

    sudo -i
  2. Add this node as the Geo primary by running:

    bundle exec rake geo:set_primary_node

Step 2. Copying the database encryption key

GitLab stores a unique encryption key in disk that we use to safely store sensitive data in the database. Any secondary node must have the exact same value for db_key_base as defined in the primary one.

  1. SSH into the primary node and login as root:

    sudo -i
  2. Execute the command below to display the current encryption key and copy it:

     bundle exec rake geo:db:show_encryption_key
  3. SSH into the secondary node and login as root:

    sudo -i
  4. Open the secrets file and paste the value of db_key_base you copied in the previous step:

     editor /etc/gitlab/gitlab-secrets.json
  5. Save and close the file.

The secondary will start automatically replicating missing data from the primary in a process known as backfill. Meanwhile, the primary node will start to notify changes to the secondary, which will act on those notifications immediately. Make sure the secondary instance is running and accessible.

Step 3. Enabling hashed storage (from GitLab 10.0)

  1. Visit the primary node's Admin Area ➔ Settings (/admin/application_settings) in your browser
  2. In the Repository Storages section, check Create new projects using hashed storage paths:

Using hashed storage significantly improves Geo replication - project and group renames no longer require synchronization between nodes - so we recommend it is used for all GitLab Geo installations.

Step 4. Managing the secondary GitLab node

You can monitor the status of the syncing process on a secondary node by visiting the primary node's Admin Area ➔ Geo Nodes (/admin/geo_nodes) in your browser.

Please note that if git_data_dirs is customized on the primary for multiple repository shards you must duplicate the same configuration on the secondary.

GitLab Geo dashboard

Disabling a secondary node stops the syncing process.

The two most obvious issues that replication can have here are:

  1. Database replication not working well
  2. Instance to instance notification not working. In that case, it can be something of the following:
    • You are using a custom certificate or custom CA (see the Troubleshooting section)
    • Instance is firewalled (check your firewall rules)

Currently, this is what is synced:

  • Git repositories
  • Wikis
  • LFS objects
  • Issue, merge request, snippet and comment attachments
  • User, group, and project avatars

You can monitor the status of the syncing process on a secondary node by visiting the primary node's Admin Area ➔ Geo Nodes (/admin/geo_nodes) in your browser.

Next steps

Your nodes should now be ready to use. You can login to the secondary node with the same credentials as used in the primary. Visit the secondary node's Admin Area ➔ Geo Nodes (/admin/geo_nodes) in your browser to check if it's correctly identified as a secondary Geo node and if Geo is enabled.

If your installation isn't working properly, check the troubleshooting section.

Point your users to the after setup steps.

Selective replication

Read Selective replication.

Adding another secondary Geo node

To add another Geo node in an already Geo configured infrastructure, just follow the steps starting from step 2. Just omit the first step that sets up the primary node.

Replicating wikis and repositories over SSH

Read Replicating wikis and repositories over SSH.

Troubleshooting

Read the troubleshooting document.