CephFS Turn Down

Things left to do before completely shutting down CephFS

• Storage Location
• Spin up nfs-share01 & bootstrap
• Uploads
• Final Sync of Uploads to NFS (nfs-share01)
• repoint and cutover
• CI builds
• sync to nfs-share01
• repoint and cutover
• Power Off and 24 Hour Cool Down
• Deletion of hosts and disk storage

/cc @pcarranza @stanhu @ahanselka @jnijhof

Sounds great. Eager to do this.

Somewhat bittersweet don't you think?

@northrup Did we still have projects in pending_delete on CephFS?

Where are we with this?

marked the checklist item Storage Location as completed

marked the checklist item Spin up nfs-share01 & bootstrap as completed

@stanhu I spun up nfs-share01 and I'm just finishing the last of the sync of 'uploads' this weekend and then I'll shut the CephFS cluster down on Sunday.

Unexpected discovery of the CI build logs being hosted on CephFS. Started syncing those on Saturday and now that is the only item remaining.

marked the checklist item Uplaods as completed

marked the checklist item Final Sync of Uploads to NFS (nfs-share01) as completed

marked the checklist item repoint and cutover as completed

marked the checklist item repoint and cutover as incomplete

marked the checklist item repoint and cutover as completed

@northrup I hear the builds are done... next step? :)

@stanhu augmenting the NFS cookbooks now for the new mount, and swapping over in the low point of the night... standby for the hurrah of a powered down CephFS

marked the checklist item CI builds as completed

marked the checklist item sync to nfs-share01 as completed

marked the checklist item repoint and cutover as completed

marked the checklist item Power Off and 24 Hour Cool Down as completed

@stanhu @pcarranza @jnijhof Ceph cluster is in cooldown. All nodes have been powered down.

@ahanselka @jnijhof Did this get turned back on? If so, can you update this issue about why and what we need to do to really shut this down?

@jnijhof can you update this issue with the last actions that were performed with the missing projects please?

So there were 1706 projects still on ceph, after I restarted our ceph cluster I started to copy data from there to nfs-file03. After the copy finished I changed the repository storage to nfs-file03 which fixed these stale projects.

Some minor issues happened as well for 600 projects which already started a move but did not finish it some how, ending up with repository directories being renamed with +moved+ but was not actually moved. I renamed these projects and synced them as well to nfs-file03.

Currently there are no projects on ceph storage anymore.

@jnijhof have we turned it off yet?

marked the checklist item Deletion of hosts and disk storage as completed

@ahanselka ^

@jnijhof hehehe amazing

I'm also removing ceph related prometheus jobs from chef-repo

I think we can close this one now

closed