file-81 data-disk mounted on /dev/sbc getting ignored by weight assigner

Summary

The data disk on file-81 is mounted on /dev/sdc on other file servers it's mounted on /dev/sdb.

steve@file-81-stor-gprd.c.gitlab-production.internal:~$ df -h | grep 'gitlab'
/dev/sdc         16T   14T  2.2T  87% /var/opt/gitlab

steve@file-89-stor-gprd.c.gitlab-production.internal:~$ df -h | grep 'gitlab'
/dev/sdb         16T  6.0T  9.6T  39% /var/opt/gitlab

This shouldn't matter but we use gitaly-shard-weights-assigner which runs a prometheus query that only looks at /dev/sdb. As we saw in production#8419 (closed) file-81 was still at 100 weight even though it was 90% full (it should have been 0)

Related Incident(s)

• production#8419 (closed)

Action Items

• Investigate why file-81 is special and has the data disk on /dev/sdc rather then /dev/sdb
• Update the weight assigner query to filter by mountpoint="/var/opt/gitlab" rather then device="/dev/sdb"
• Update the Production EXCLUDED_SHARDS variable to remove nfs-file-81