Move LFS storage to its own host

file-storage1 grows a lot all the time. It seems the largest offenders are artifacts and lfs-objects. We need to set up alerts to be sure we keep on top of this and don't let everything catch on fire.

/dev/mapper/gitlab_vg-gitlab_var   23T   20T  4.0T  83% /var/opt/gitlab

root@file-storage1:/var/opt/gitlab/gitlab-rails/shared# nice -n19 ionice -c3 du -h --max-depth=1 | tee /root/usage-summary-gitlab-rails-shared-amh.txt
9.7T    ./artifacts
91G     ./cache
7.7T    ./lfs-objects
1.5T    ./pages
20G     ./tmp
12K     ./registry
19T     .

cc/ @pcarranza @maratkalibek

Ok, so we need 2 things here:

• Reduce artifacts pressure (I think @ayufan is already working on this)
• Split artifacts from LFS objects into separate hosts

Mentioned in issue #412 (closed)

@stanhu Looks like we need to evict old artifacts as they are taking most of the space, along with LFS objects which we can't delete.

How are we storing 1.5Tb of pages?

BTW, we need to pull the uploads into a separate NFS server so we can take CephFS down completely.

@ahanselka please check these last comments as this is something that is burning right now

@pcarranza I'm interested in the pages thing too. I'm going to run a du on that and find huge sites and see if any are abusive or all are normal.

I will also create a new LFS NFS server. How big do we want the LFS NFS to be? What do we even want it named? At a later date I also need to know if there a way we can block LFS uploads or make it read only for a few minutes at some point to finalize the switch over?

cc/ @ayufan @stanhu @pcarranza @northrup

Reassigned to @ahanselka

I have created nfs-lfs01.stor.gitlab.com. It is the same setup as our other NFS servers. I am going to get with @northrup on Monday to sync up the LFS folder and then we can make plans to switch over.

@pcarranza so I checked on the pages as I said I would and there is a giant directory at /var/opt/gitlab/gitlab-rails/shared/pages/tmp. It is 919GB and I have no idea what it is for. There are also a few large pages sites, but none are over 1GB which is our max, and it doesn't appear to be abuse.

I've begun the LFS sync to the new nfs-lfs01 server. It is currently estimating it will take about 1.5 days.

cc/ @pcarranza @northrup @maratkalibek @stanhu

@ahanselka

How many projects we have pending to be deleted?

@ayufan I actually don't know how to find that out...

@northrup basically, yes

Added availability cleanup ~366510 labels

FYI I added a gauge for this new host in the storage dashboard.

At this time, the sync for LFS objects claims it will take about 12 more hours.

Interesting, can we trace LFS objects back to their owner easily? It seems like it's just an object storage and a giant black box?

I've no idea how LFS is implemented, if we can have a positive lookup then we could run a process to clean up. But we will need to have an actual solution for the future to keep it under control.

I'm calling people I have an impression has touched LFS at a given point in time.

cc/ @jacobvosmaer-gitlab @patricio @brodock

Milestone changed to %Milestone-6

The sync finished! What would the next steps be to actually make the switch? I assume we need to make LFS read-only, sync again to get any changed files, and then mount the new location on all the workers?

cc/ @ayufan @northrup

@ahanselka those sound like logical next stems to me. Let's proceed down that path please!

@northrup Is there a way to mark LFS read only or do we have to turn it off? Should we use the FDT tool or rsync for the second pass? Should we plan a time to do this so we can alert our users to the downtime? Perhaps next week after the holiday? I don't like the idea of doing it before a holiday really, even if it is just a US holiday.

Sorry, playing 20 questions over here

I believe that there's no read-only switch.

Just take downtime

I mean we definitely have to take the downtime, I'm curious what the user experience will be while LFS is turned off, though. I will be sure to do this Monday after US Thanksgiving.

Cyber Monday fun!

I'll try to figure out if there is a good way of doing this.

LFS objects have a 1-many relation with projects. If needed we can create a choke point for LFS uploads in Projects::LfsApiController.

(The reason it's 1-many with projects is because of deduplication: if two unrelated projects upload the same file we only store it once.)

That's interesting data @jacobvosmaer-gitlab

Are you aware of any way of knowing if a given file is actually not used at all? (as in, we can delete it)

Ok, it is Monday! @pcarranza any ideas?

@northrup can I use the same FDT tool or should I use something like rsync for the second run through?

@ahanselka just take downtime. That's fine, we should use the downtime for moving both uploads out of cephfs and LFS out of this other host.

Changed title: Identify and alert for used space on file-storage1 → Move LFS storage to its own host

@northrup I still need your answer for FDT vs. rsync

I'd use rsync for the second catch. It's better about delta computation and then only copying what's needed.

Ok! I'm running an rsync now to get it as in sync as possible before taking downtime.

cc/ @pcarranza @northrup

The rsync finished very quickly. I am going to prepare merge requests with @northrup to switch the LFS mount over. Once these are completed, we can take the downtime and make the migration pretty painless due to the speed of the rsync.

Excellent Job!

Alright! We've migrated all of the data and all of the workers are pointed to the new LFS NFS server! YAY!

I think that is all that is left for this issue, so I'm closing it.

Status changed to closed

@ahanselka please create an issue for cleaning up the LFS data once we are comfortable with the new storage

@pcarranza I don't think we have a deletion mechanism at the moment. It's a bit finnicky because you have to be careful about concurrent uploads of the file you're about to delete. If you're not careful, you end up with missing LFS objects etc.

It's similar to other kinds of file uploads which also don't really have a good deletion regime. We thought of it as an append-only thing.

@jacobvosmaer-gitlab this is about deleting a folder that is not in use anymore. We moved the LFS objects to a different host completely. It's about cleaning up behind a move. rm -rf is the deletion mechanism to use in this case, there should be no upload at all in the old storage.

Mentioned in issue #818 (closed)

#818 (closed)

mentioned in issue #2603 (closed)