Mount every shared subdir as its own mount to avoid nested mounts - recycle the front end fleet
Nested mount points continue to cause problems, particularly with LFS. In order to alleviate this issue, we can use symlinks in /var/opt/gitlab
to point to the appropriate mount points elsewhere.
For example, we could have lfs-objects
mounted to /mnt/lfs-opjects
and then have a symlink pointing from /var/opt/gitlab/gitlab-rails/shared/lfs-objects
to /mnt/lfs-objects
. This would make mount order not relevant and would prevent the LFS woes we've experienced recently.
The currently nested mounts are:
/var/opt/gitlab/gitlab-rails/shared/pages
/var/opt/gitlab/gitlab-rails/shared/lfs-objects
/var/opt/gitlab/gitlab-rails/shared/artifacts/2017_04
/var/opt/gitlab/gitlab-rails/shared/artifacts/2017_05
/var/opt/gitlab/gitlab-rails/shared/artifacts/2017_06
/var/opt/gitlab/gitlab-rails/shared/artifacts/2017_07
/var/opt/gitlab/gitlab-rails/shared/artifacts/2017_08
This should be done before we can put nfs-flake-01
into production: https://gitlab.com/gitlab-com/infrastructure/issues/2123
Proposed structure:
/var/opt/gitlab/gitlab-rails/shared/artifacts/2017_04 -> /mnt/artifacts/2017_04
/var/opt/gitlab/gitlab-rails/shared/artifacts/2017_05 -> /mnt/artifacts/2017_05
/var/opt/gitlab/gitlab-rails/shared/artifacts/2017_06 -> /mnt/artifacts/2017_06
/var/opt/gitlab/gitlab-rails/shared/artifacts/2017_07 -> /mnt/artifacts/2017_07
/var/opt/gitlab/gitlab-rails/shared/artifacts/2017_08 -> /mnt/artifacts/2017_08
/var/opt/gitlab/gitlab-rails/shared/lfs-objects -> /mnt/lfs-objects
Previous description text:
Currently our nfs recipe doesn't take into account any kind of mount order. The order in
/etc/fstab
matters. The file should be organized with most general mount (like/
and/var/opt/gitlab/gitlab-rails/shared
) and progress to the most specific mounts such as:/var/opt/gitlab/gitlab-rails/shared/artifacts/2017_06
. Currently it doesn't seem to take that into account.The most notable problem is that
/var/opt/gitlab/gitlab-rails/shared
is mounted AFTER/var/opt/gitlab/gitlab-rails/shared/lfs-objects
.cc/ @jtevnan