Pages shared secret mismatch between web-pages-01-sv-gstg and api-01-sv-gstg

Summary

With !3705 (merged) we added new shared secret file that will be used by Pages to authenticate requests to internal Gitlab API.

While working with @aamarsanaa on https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8941 we noticed that the content of .gitlab_pages_secret is different on web-pages-01-sv-gstg and api-01-sv-gstg - https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8941#note_281843237. This shared secret file should have the same content for all nodes, otherwise Pages will not be able to authenticate with the internal API.

The file that is on web-pages-01-sv-gstg is way older than the one on api-01-sv-gstg:

web-pages-01-sv-gstg
Oct 10  2018 /opt/gitlab/embedded/service/gitlab-rails/.gitlab_pages_secret

api-01-sv-gstg
Dec 11 15:38 /opt/gitlab/embedded/service/gitlab-rails/.gitlab_pages_secret

I suspect this may be some leftover from the admin API for Pages that we removed with !3669 (merged), mostly so that we can reuse the filename. There was some discussion about should we do some cleanup but we decided it's OK not to - !3669 (comment 229900553).

We should check do we have same issue on production and I already asked @aamarsanaa to have a look (https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8941#note_281910978) but I wonder

  1. Can we find how did this happen?
  2. What is the best way to fix this for staging (also for production and self-managed if they have the same issue)?

This internal API and the functionality that need it are not in used yet, I wonder if the best way is to have new MR for Omnibus that will 1) cleanup .gitlab_pages_secret and 2) use new name for this file so we can have it generated on all nodes.