Move gitlab-foss and gitlab to file-cny-01

Production Change - Criticality 2 C2

Change Objective Describe the objective of the change
Change Type ConfigurationChange
Services Impacted gitlab-org/gitlab and gitlab-org/gitlab-foss (gitaly)
Change Team Members @jarv
Change Severity C2
Change Reviewer TBD
Tested in staging This will be run in canary
Dry-run output If the change is done through a script, it is mandatory to have a dry-run capability in the script, run the change in dry-run mode and output the result
Due Date Date and time in UTC timezone for the execution of the change, if possible add the local timezone of the engineer executing the change
Time tracking To estimate and record times associated with changes ( including a possible rollback )

Overview

To alleviate recent performance issues on file-02 https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8407 we will be moving gitlab-org/gitlab-foss and gitlab-org/gitlab to the new file-cny-01 file server. This is a dedicated file node for gitaly, that will eventually be added to the canary fleet. At this time, it will be treated like any other file node.

Monitoring

Procedure

Preparation

Add the new storage shard in the configuration:

GITLAB-FOSS

gitlab-foss is on file-08, we are moving this project first as it should give us a general idea for the time to migrate gitlab-org/gitlab

  • Announce that gitlab-foss will be switched to read-only for project migration in the following channels:
  • #announcements
:exclamation: IMPORTANT ANNOUNCEMENT :exclamation:
To alleviate performance issues on one of the gitlab.com storage servers we are going to migrate the 
repository for the https://gitlab.com/gitlab-org/gitlab project to a new storage server. During the migration, 
the project repository will be set to read-only, meaning that we will not be able to do any write activities 
on the repo.

We will start with a migration of `gitlab-org/gitlab-foss`, this migration is starting now and we do not expect it to have any impact on activies.

See https://gitlab.com/gitlab-com/gl-infra/production/issues/1440 for more information
  • Switch the gitlab-foss project to read-only from the rails console
p = Project.find_by_full_path('gitlab-org/gitlab-foss')
p.repository_read_only
  • Initiate the move of gitlab-foss
curl --request PUT --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" -d repository_storage=nfs-file-cny01 https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab-foss
p = Project.find_by_full_path('gitlab-org/gitlab-foss')
p.repository_storage

GITLAB

  • Announce that gitlab will be switched to read-only for project migration in the following channels:
  • #announcments
  • #backend
  • #company-announcements
:exclamation: IMPORTANT ANNOUNCEMENT :exclamation:
To alleviate performance issues on one of the gitlab.com storage servers we are going to migrate the 
repository for the https://gitlab.com/gitlab-org/gitlab project to a different server. During the migration, 
the project repository will be set to read-only, meaning that we will not be able to do any write activities 
on the repo.

This migration is starting now and we will expect it to take XX minutes.

See https://gitlab.com/gitlab-com/gl-infra/production/issues/1440 for more information
  • Switch the gitlab project to read-only from the rails console
p = Project.find_by_full_path('gitlab-org/gitlab')
p.repository_storage
p.repository_read_only
  • Initiate the move of gitlab
curl --request PUT --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" -d repository_storage=nfs-file-cny01 https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab
p = Project.find_by_full_path('gitlab-org/gitlab')
p.repository_storage

Rollback

  • Initiate a migration back to file-08 for gitlab-foss
curl --request PUT --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" -d repository_storage=nfs-file08 https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab-foss
  • Initiate a migration back to file-02 for gitlab
curl --request PUT --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" -d repository_storage=nfs-file02 https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab
Edited by John Jarvis