Swap the current LFS server for the newly built

We have two options to migrate. The first one is to do this the same we did last time. Disable the old mount, mount the new one, resync one last time.

The alternative is to bundle it in with https://gitlab.com/gitlab-com/infrastructure/issues/2123 (nfs-flake-01 into production).

We have time to make the switchover. nfs-lfs-01 is only 61% full. I say we bundle it in with #2123 (closed). The mountpoints will already be changing when that happens, it makes sense to put all our mount changes at one time. Right before we begin to rebuild the fleet, I can perform another sync so it will be as close to accurate as possible. Another resync can happen immediately after the rebuild is complete.

https://gitlab.com/gitlab-com/infrastructure/issues/2543

added blocked label

marked this issue as related to #2543 (closed)

mentioned in issue #2597 (closed)

mentioned in issue #2543 (closed)

changed the description

cc/ @pcarranza @northrup @omame

By the way, I'd like to add that if we do this when we are rebuilding the fleet, we won't have to take downtime.

changed milestone to %WoW ending 2017-09-19

Alright, due to the continuing troubles with making images and rebuilding, I think the best bet would be to update the mounts.

chef-repo!1113 updates chef with the correct new nfs-lfs-02 mountpoint. The procedure to update would be as follows.

• Sync nfs-lfs-01 to nfs-lfs-02
• Update chef-repo and run chef-client on all the things
• Sync nfs-lfs-01 to nfs-lfs-02 once more
• Verify LFS still does what it should.

This would leave a small window where LFS might go to the old server or the new server, but the final sync should take care of that. This is the same method we used in https://gitlab.com/gitlab-com/infrastructure/issues/750.

cc/ @northrup @pcarranza

removed blocked label

I've started another rsync so that we can do this this week.

I also thought I posted this last night, so I started this yesterday afternoon and it is finished.

This is blocked until we figure out gitlab-com/infrastructure#2744. We currently have chef-client turned off on the servers that need to run chef-client to put this into production. We could update chef-repo but we don't know how long the Azure troubleshooting will take.

added blocked label

removed milestone

removed blocked label

changed milestone to %WoW ending 2017-09-26

This can proceed and is scheduled in the next WoW.

Do we still require downtime for this @ahanselka ?

@pcarranza I don't think so. The main concern is being sure the old mount is unmounted and the new one mounted. This won't cause downtime, though there may be a delay in LFS objects updating for a bit as we do a final rsync between the old and new servers.

changed milestone to %WoW ending 2017-10-03

I spoke with @jtevnan about this, and it seems like it might be best to lump this in with a coming downtime. We tried to find a way to do this without downtime, but it seems like it would be easier and safer to just do it when we are already having downtime. As such, I'm going to remove this from the WoW as it still isn't urgent and can wait for an opportunity.

removed milestone

Ok, this has just been sitting for too long and we just need to deal with it. As such, here is my newly proposed plan.

1. Schedule LFS only downtime
2. rsync data
3. Disable LFS in gitlab.rb and apply to fleet
4. rsync data
5. Switch out mounts on all the servers
6. Enable LFS in gitlab.rb and apply to fleet
7. Test to ensure all is well.

This is similar to how we did it last time and I think it would be the most expedient and least impactful way to apply this.

Thoughts?

cc/ @northrup @pcarranza @ilyaf

I'd like to schedule this for Monday (2017-12-04) US evening.

I've prepared a few MRs for this lovely occasion. The process will be as follows:

1. rsync lfs-objects from nfs-lfs-01 to nfs-lfs-02
2. Disable chef-client on prod so it doesn't try to run in the middle of all of thisv
3. Disable LFS on canary, make sure LFS is disabled (chef-repo!1354)
4. Change mount point in chef for canary (chef-repo!1355)
5. Manually unmount nfs-lfs-01 from canary and sed away the old entry in fstab, sed -i '/10.70.2.121/d' /etc/fstab
6. Run chef-client on canary and ensure the proper mount points
7. Disable LFS on prod, make sure LFS is disabled
8. begin final rsync of data
9. Manually unmount nfs-lfs-01 from canary and sed away the old entry in fstab
10. Run chef-client on prod and ensure the proper mount points
11. Ensure final rsync is complete
12. Re-enable LFS on canary and test that it works (chef-repo!1356)
13. Re-enable LFS on prod and test that it works
14. Success!

How do you feel about this, @ilyaf @northrup

Also, to be clear, nfs-lfs-02 is already rsynced mostly. I'm running another one now and yet another one before we start.

removed milestone

So this has just sat for a while again since we've had so many other "fun" events recently. It is now the holidays, thus many people will be off and there is no reason to potentially cause problems at this time, we have enough already. Thus, I'm going to do this early January.

As of now, nfs-lfs-01 is at a bit under 75% usage. It grows fairly slowly so we will have plenty of time and be fine into January. Additionally, we are hoping to roll out object storage for LFS "soon" which would be super awesome.

I'm leaving this assigned to me however, so it doesn't fall through the cracks as it has already.

mentioned in issue #3453 (moved)

With the migration to object storage, we no longer need this.

closed