## Background The shared NFS server is a SPOF for GitLab.com and the source of various issues that impact our availability * We are seeing soft lockups resulting in spontaneous reboots on servers that have these mounts, because these paths are also used as temporary directories for object storage uploads there are many unnecessary writes to these NFS mounted paths. https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7872#note_227712046 * The shared NFS server has also been the source of production incidents: * https://gitlab.com/gitlab-com/gl-infra/production/issues/733 * https://gitlab.com/gitlab-com/gl-infra/production/issues/759 ### Out of Scope * Removing Pages NFS: https://gitlab.com/groups/gitlab-org/-/epics/3901 ## Project management * Issue board https://gitlab.com/groups/gitlab-com/gl-infra/-/boards/1705970?scope=all&utf8=%E2%9C%93&state=opened&epic_id=13429&include_subepics=%E2%9C%93 ## GitLab.com unmounts on all servers - [x] Unmount `/var/opt/gitlab/gitlab-rails/shared/tmp` - [x] Unmount `/var/opt/gitlab/gitlab-rails/shared/lfs-objects` - [x] Unmount `/var/opt/gitlab/gitlab-rails/uploads` - [x] Unmount `/var/opt/gitlab/gitlab-rails/shared/cache` Needs to remain because of https://gitlab.com/gitlab-org/gitlab/-/issues/216000 - [x] Unmount `/var/opt/gitlab/gitlab-ci/builds` Needs to remain mounted until `ci_enable_live_trace` is enabled on production https://gitlab.com/groups/gitlab-org/-/epics/4275 - [x] Unmount `/var/opt/gitlab/gitlab-rails/shared/artifacts` Needs to remain mounted until `ci_enable_live_trace` is enabled on production https://gitlab.com/groups/gitlab-org/-/epics/4275 ## GitLab.com unmounts on the git fleet - [x] Unmount `/var/opt/gitlab/gitlab-rails/shared/tmp` - [x] Unmount `/var/opt/gitlab/gitlab-rails/shared/lfs-objects` - [x] Unmount `/var/opt/gitlab/gitlab-rails/uploads` - [x] Unmount `/var/opt/gitlab/gitlab-rails/shared/cache` - [x] Unmount `/var/opt/gitlab/gitlab-ci/builds` - [x] Unmount `/var/opt/gitlab/gitlab-rails/shared/artifacts` ## GitLab.com prod and non-prod environment configuration ### Direct upload Direct upload is **enabled** on GitLab.com environments: * artifacts_object_store_direct_upload * lfs_object_store_direct_upload * packages_object_store_direct_upload * uploads_object_store_direct_upload ### Workhorse/Rails Workhorse is deployed 1:1 with rails on all nodes so files that are shared between workhorse and unicorn on the filesystem are not an issue https://gitlab.com/gitlab-org/gitlab/issues/21511 ## Validation The following shared paths are NFS mounted across all front-end and back-end nodes: ``` /var/opt/gitlab/gitlab-rails/shared/lfs-objects /var/opt/gitlab/gitlab-rails/shared/cache /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared/artifacts /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-rails/shared/tmp ``` ## Blocking issues * Until live traces are enabled https://gitlab.com/groups/gitlab-org/-/epics/4275 we cannot unmount `/builds` or `/artifacts`