Moving from CephFS Back to NFS

This is to track the status and progress of moving from CephFS back to NFS by way of the new nfs-file0[1-8] servers.

We need to:

• Enable NFS Shards in GitLab
• Change Default Creation Point away from Ceph
• Migrate repos from CephFS to NFS fleet

/cc @pcarranza @jnijhof @stanhu

I have enabled the NFS shards in GitLab and made the default project creation space nfs-file01 for now.

Added ~476270 performance labels

Marked the task Enable NFS Shards in GitLab as completed

Marked the task Change Default Creation Point away from Ceph as completed

@northrup Great, thanks!

Current status: We've now moved 88,000 projects to NFS, with 430,000 to go for CephFS

 repository_storage | count
--------------------+--------
 nfs-file04         |  11135
 nfs-file02         |  10894
 nfs-file07         |  10915
 alternate          |    133
 nfs-file05         |  10914
 default            | 983434
 nfs-file01         |  11504
 nfs-file06         |  10954
 ceph               | 430893
 nfs-file03         |  10822
 nfs-file08         |  10877

Thanks for the update, @northrup! That was 88,000 projects in 8 days. Does that mean we expect another 40 days or so?

@stanhu no, I've had a significant uptick in movement... I'm estimating ~7 days to finish moving everything. We're currently processing ~2,500 projects an hour now on average.

Current Status:

 repository_storage | count
--------------------+--------
 nfs-file04         |  33913
 nfs-file02         |  33692
 nfs-file07         |  33653
 alternate          |    133
 nfs-file05         |  33641
 default            | 982949
 nfs-file01         |  34353
 nfs-file06         |  33776
 ceph               | 262571
 nfs-file03         |  33704

Current Status:

 repository_storage | count
--------------------+--------
 nfs-file04         |  56134
 nfs-file02         |  55964
 nfs-file07         |  55878
 alternate          |    133
 nfs-file05         |  55920
 default            | 982532
 nfs-file01         |  56609
 nfs-file06         |  56071
 nfs-file03         |  55983
 ceph               |  95250
 nfs-file08         |  56006

Current Status:

DONE!

 repository_storage | count
--------------------+--------
 nfs-file04         |  67891
 nfs-file02         |  67742
 nfs-file07         |  67654
 alternate          |    133
 nfs-file05         |  67676
 default            | 982511
 nfs-file01         |  68353
 nfs-file06         |  67843
 nfs-file03         |  67780
 ceph               |   2277
 nfs-file08         |  67782

of those, a fair amount are flagged for delete, just not deleted

gitlabhq_production=# select count(*) from projects where repository_storage = 'ceph' and pending_delete = 't';
 count
-------
   559

the remaining also have errors and issues. I'll perform the cleanup Tuesday (11/22) and turn the servers down.

So only thing missing is the uploads and taking a look around to see that we are not missing anything.

Great work @northrup

And we need to cleanup default into all the others too.

Added storage label

Marked the task Migrate repos from CephFS to NFS fleet as completed

Milestone changed to %Milestone-5

Milestone changed to %Milestone-6

Milestone changed to %Milestone-5

Status changed to closed

Can you document the NFS mount options in use on GitLab.com? See https://gitlab.com/gitlab-com/infrastructure/issues/1314