Move gitlab-foss and gitlab to file-cny-01 (second attempt)

Production Change - Criticality 2 C2

Change Objective Describe the objective of the change
Change Type ConfigurationChange
Services Impacted gitlab-org/gitlab and gitlab-org/gitlab-foss (gitaly)
Change Team Members @jarv
Change Severity C2
Change Reviewer TBD
Tested in staging This will be run in canary
Dry-run output If the change is done through a script, it is mandatory to have a dry-run capability in the script, run the change in dry-run mode and output the result
Due Date Date and time in UTC timezone for the execution of the change, if possible add the local timezone of the engineer executing the change
Time tracking To estimate and record times associated with changes ( including a possible rollback )

Overview

To alleviate recent performance issues on file-02 https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8407 we will be moving gitlab-org/gitlab-foss and gitlab-org/gitlab to the new file-cny-01 file server. This is a dedicated file node for gitaly, that will eventually be added to the canary fleet. At this time, it will be treated like any other file node.

Monitoring

Procedure

Preparation

See previous change issue #1440 (closed)

test-public-project

https://gitlab.com/reprazents-test-space-renamed/test-public-project is on file-09, we are moving this project first for testing and ensuring that forks are not broken after the migration

curl --request PUT --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" -d repository_storage=nfs-file-cny01 https://gitlab.com/api/v4/projects/reprazents-test-space-renamed%2ftest-public-project

GITLAB-FOSS

gitlab-foss is on file-08, we are moving this project as it should give us a general idea for the time to migrate gitlab-org/gitlab

  • Announce that gitlab-foss will be switched to read-only for project migration in the following channels:
  • #announcements
:exclamation: IMPORTANT ANNOUNCEMENT :exclamation:
To alleviate performance issues on one of the gitlab.com storage servers we are going to migrate the 
repository for the https://gitlab.com/gitlab-org/gitlab project to a new storage server. During the migration, 
the project repository will be set to read-only, meaning that we will not be able to do any write activities 
on the repo.

We will start with a migration of `gitlab-org/gitlab-foss`, this migration is starting now and we do not expect it to have any impact on activies.

See https://gitlab.com/gitlab-com/gl-infra/production/issues/1477 for more information
  • Initiate the move of gitlab-foss
curl --request PUT --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" -d repository_storage=nfs-file-cny01 https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab-foss
p = Project.find_by_full_path('gitlab-org/gitlab-foss')
p.repository_storage

GITLAB

  • Announce that gitlab will be switched to read-only for project migration in the following channels:
  • #announcments
  • #backend
  • #company-announcements
:exclamation: IMPORTANT ANNOUNCEMENT :exclamation:
To alleviate performance issues on one of the gitlab.com storage servers we are going to migrate the 
repository for the https://gitlab.com/gitlab-org/gitlab project to a different server. During the migration, 
the project repository will be set to read-only, meaning that we will not be able to do any write activities 
on the repo.

This migration is starting now and we will expect it to take 20 minutes.

See https://gitlab.com/gitlab-com/gl-infra/production/issues/1477 for more information
  • Initiate the move of gitlab
curl --request PUT --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" -d repository_storage=nfs-file-cny01 https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab
p = Project.find_by_full_path('gitlab-org/gitlab')
p.repository_storage

Rollback

  • Initiate a migration back to file-08 for gitlab-foss
curl --request PUT --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" -d repository_storage=nfs-file08 https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab-foss
  • Initiate a migration back to file-02 for gitlab
curl --request PUT --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" -d repository_storage=nfs-file02 https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab
Edited by John Jarvis