<!--- Please read this! Before opening a new issue, make sure to search for keywords in the issues filtered by the "regression" or "type::bug" label: - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=type::bug and verify the issue you're about to submit isn't a duplicate. ---> ### Summary Related to https://gitlab.com/gitlab-org/gitlab/-/issues/358924 and https://gitlab.com/gitlab-org/gitlab/-/issues/5795 based on the analysis [in a customer ticket](https://gitlab.zendesk.com/agent/tickets/289905) (internal link) the NTP check will be run against the default NTP server of pool.ntp.org as we can see in [the net-ntp gem source](https://github.com/zencoder/net-ntp/blob/master/lib/net/ntp/ntp.rb#L66). Customers in air gapped environments or many on-premises datacentres may be able to resolve `pool.ntp.org` but won't be able to make arbitrary outgoing connections to the internet. This has come again in another [customer ticket](https://gitlab.zendesk.com/agent/tickets/340772) (internal link) and we need a workaround so customers can redirect the query to a valid local NTP time source. As it appears to be hard coded up stream, this is going to be a lot messier than putting something in `/etc/gitlab/gitlab.rb`. ### Workaround - Define `pool.ntp.org` in /etc/hosts - It'll be necessary to define IP addresses, but to partly mitigate this, it should be possible to define multiple entries, in the same way that the NTP client would use multiple time sources. - If one of the NTP servers is shut down, the check is likely to fail intermittently. ### Steps to reproduce Configure /etc/hosts to point pool.ntp.org to a local IP address that doesn't serve NTP: ```plaintext # sudo gitlab-rake gitlab:geo:check Checking Geo ... GitLab Geo secondary database is correctly configured ... not a secondary node Database replication enabled? ... not a secondary node Database replication working? ... not a secondary node GitLab Geo HTTP(S) connectivity ... not a secondary node GitLab Geo is available ... GitLab Geo is enabled ... yes This machine's Geo node name matches a database record ... no [snip] HTTP/HTTPS repository cloning is enabled ... yes Machine clock is synchronized ... Exception: No route to host - recvfrom(2) [output continues] ``` .. or doesn't respond: ```plaintext # sudo gitlab-rake gitlab:geo:check Checking Geo ... GitLab Geo secondary database is correctly configured ... not a secondary node Database replication enabled? ... not a secondary node Database replication working? ... not a secondary node GitLab Geo HTTP(S) connectivity ... not a secondary node GitLab Geo is available ... GitLab Geo is enabled ... yes This machine's Geo node name matches a database record ... no [snip] HTTP/HTTPS repository cloning is enabled ... yes Machine clock is synchronized ... Exception: Timeout::Error [output continues] ``` ### Example Project <!-- If possible, please create an example project here on GitLab.com that exhibits the problematic behavior, and link to it here in the bug report. If you are using an older version of GitLab, this will also determine whether the bug is fixed in a more recent version. --> ### What is the current *bug* behavior? `Machine clock is synchronized` check fails ### What is the expected *correct* behavior? `Machine clock is synchronized` check can be made to work in a way that doesn't involve changing `/etc/hosts` We added this check because of the importance of time syncing to Geo. ### Relevant logs and/or screenshots <!-- Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise. --> ### Output of checks <!-- If you are reporting a bug on GitLab.com, uncomment below --> <!-- This bug happens on GitLab.com --> <!-- /label ~"reproduced on GitLab.com" --> #### Results of GitLab environment info <!-- Input any relevant GitLab environment information if needed. --> <details> <summary>Expand for output related to GitLab environment info</summary> <pre> (For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`) </pre> </details> #### Results of GitLab application Check <!-- Input any relevant GitLab application check information if needed. --> <details> <summary>Expand for output related to the GitLab application check</summary> <pre> (For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:check SANITIZE=true`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true`) (we will only investigate if the tests are passing) </pre> </details> ### Possible fixes <!-- If you can, link to the line of code that might be responsible for the problem. -->