Secret mismatch on multi-node Incoming Mail/Service Desk

Summary

When configuring Incoming Mail or Service Desk on a multi-node environment, GitLab Rails nodes must contain the same secret for authenticating requests between them.

This secret is set on each nodes' configuration file via the following setting:

gitlab_rails[incoming_mail_secret_file] 

This setting however is being ignored upon reconfigure, and nodes are reading the auto-generated secret which prevents requests from passing authentication.

Steps to reproduce

Follow the docs to enable multi-node Incoming Mail.

What is the current bug behavior?

Requests sent between nodes are rejected as they fail authentication.

What is the expected correct behavior?

Requests are accepted and processed as all nodes use the same secret for authentication.

Relevant logs and/or screenshots

The API requests between rails nodes are rejected (401) as follows:

"severity":"INFO","duration_s":0.13133,"db_duration_s":0.00062,"view_duration_s":0.13071,"status":401,"method":"POST","path":"/api/v4/internal/mail_room/incoming_email","params":[],"host":<HOST>,"remote_ip":"<REMOTE_IP>","ua":"Faraday v2.11.0","route":"/api/:version/internal/mail_room/*mailbox_type",...

Possible fixes

It's possible the gitlab_rails[incoming_mail_secret_file] is being ignored because currently our gitlab.yml.erb template for Omnibus installations is hardcoding this value instead using a variable and reading from the gitlab.rb file:

secret_file: .gitlab_incoming_email_secret
.
.
.
secret_file: .gitlab_service_desk_email_secret