Geo: Setting an internal URL for the primary breaks replication
Summary
When setting an Internal URL for the Geo primary, replication breaks. We see Error syncing repository: 2:Fetching remote geo failed: fatal: could not read Username for 'https://<external_url>:443': No such device or address
for the error message.
When a new project is created, the initial sync is successful. However, any subsequent pushes cause the above error.
Steps to reproduce
- Set or change the Internal URL on the primary node Geo admin area
- Restart GitLab on both the primary and secondary nodes (not sure if this is needed)
- See project synchronization failing
What is the current bug behavior?
Changing the Internal URL for the primary node breaks replication.
What is the expected correct behavior?
Changing the Internal URL should not break replication.
Relevant logs and/or screenshots
Also seeing this in the geo.log
:
{"severity":"ERROR","time":"2019-06-13T16:27:12.933Z","correlation_id":"2777f863-0a41-4f19-9360-529828a6aa18","class":"Geo::NodeStatusPostService","message":"Could not connect to Geo primary node - HTTP Status Code: 404 Not Found","job_id":"98de79c9d5f1d6820048ac39"}
gitlab-rake gitlab:geo:check
is unable to connect to the primary:
GitLab Geo is available ... yes
GitLab Geo is enabled ... yes
GitLab Geo secondary database is correctly configured ... yes
Database replication enabled? ... yes
Database replication working? ... yes
GitLab Geo tracking database is configured to use Foreign Data Wrapper? ... yes
GitLab Geo tracking database Foreign Data Wrapper schema is up-to-date? ... yes
GitLab Geo HTTP(S) connectivity ...
* Can connect to the primary node ... no
HTTP/HTTPS repository cloning is enabled ... yes
Machine clock is synchronized ... yes
Git user has default SSH configuration? ... yes
OpenSSH configured to use AuthorizedKeysCommand ... yes
GitLab configured to disable writing to authorized_keys file ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking Geo ... Finished
Howvever, the database seems to have the correct internal_url
. From the rails console on the secondary node:
irb(main):002:0> Gitlab::Geo.primary_node.internal_uri
=> #<URI::HTTP http://<internal_url>/>
irb(main):003:0> Gitlab::Geo.primary_node.internal_uri.host
=> "<internal_url>"
Also, the config
file from the repositories has the correct, updated internal URL for remote "geo"
.