Create nfs-lfs-02 with 2TB disks

As mentioned in gitlab-com/infrastructure#2541, we are slowly running out of space for LFS. We should build a new nfs-lfs-02 server with the new Azure 2TB disks. This will double our capacity from 16TB to 32TB and should last for quite some time. While we are only at 60% used space on nfs-lfs-01, it would be wise to deal with this sooner rather than later.

Our tasks are:

• Build nfs-lfs-02 with 2TB disks
• Sync data from nfs-lfs-01
• Cut over mounts from nfs-lfs-01 to nfs-lfs-02(https://gitlab.com/gitlab-com/infrastructure/issues/2603)
• Sync data once more (https://gitlab.com/gitlab-com/infrastructure/issues/2603)

cc/ @northrup @pcarranza

changed the description

mentioned in issue #2541 (closed)

changed milestone to %WoW ending 2017-08-29

added critical label

cc/ @jtevnan just to be aware that this is also happening tonight in the scheduled downtime.

@ahanselka we may need to create a change issue while things are getting in sync.

@pcarranza Agree. The server isn't ready yet, so we can't do much with it. Plus, syncing of LFS objects will probably take quite some time, as we will be moving nearly 10TB.

mentioned in issue #2603 (closed)

marked this issue as related to #2603 (closed)

marked this issue as related to #2541 (closed)

assigned to @ahanselka

We can create the server once https://gitlab.com/gitlab-com/gitlab-com-infrastructure/merge_requests/118 is merged and applied. We can then begin the fun of data transfer.

marked the checklist item Build nfs-lfs-02 with 2TB disks as completed

The server is created and the data is rsyncing over now. This is nearly 10TB of data, so it will take quite some time to transfer. I'll keep an eye on it in the meantime.

Awww Yeah.. Nicely done @ahanselka

Well, to my surprise, we are making really good time with the rsync. It has already transferred 4.7 TB!

/dev/mapper/gitlab_vg-gitlab_var   30T  4.7T   26T  16% /var/opt/gitlab

changed milestone to %WoW ending 2017-09-19

added moved 1 label

marked the checklist item Sync data from nfs-lfs-01 as completed

changed the description

The sync is finished. I am impressed. We can now schedule turning this over, which will be followed in https://gitlab.com/gitlab-com/infrastructure/issues/2603.

closed

removed milestone

changed milestone to %WoW ending 2017-09-05