Patroni replica consul conf.d json files should include a port attribute
Spun out of https://gitlab.com/gitlab-org/gitlab-ee/issues/13732#note_207174736
Currently our service descriptors for patroni do not include a port.
This means that SRV
records for Patroni Replicas have the wrong port (zero!)
$ dig @127.0.0.1 -p 8600 replica.patroni.service.consul. SRV
; <<>> DiG 9.10.3-P4-Ubuntu <<>> @127.0.0.1 -p 8600 replica.patroni.service.consul. SRV
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32091
;; flags: qr aa rd; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 13
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;replica.patroni.service.consul. IN SRV
;; ANSWER SECTION:
replica.patroni.service.consul. 0 IN SRV 1 1 0 patroni-08-db-gprd.node.east-us-2.consul.
replica.patroni.service.consul. 0 IN SRV 1 1 0 patroni-09-db-gprd.node.east-us-2.consul.
replica.patroni.service.consul. 0 IN SRV 1 1 0 patroni-03-db-gprd.node.east-us-2.consul.
replica.patroni.service.consul. 0 IN SRV 1 1 0 patroni-05-db-gprd.node.east-us-2.consul.
replica.patroni.service.consul. 0 IN SRV 1 1 0 patroni-06-db-gprd.node.east-us-2.consul.
replica.patroni.service.consul. 0 IN SRV 1 1 0 patroni-02-db-gprd.node.east-us-2.consul.
This is because the service descriptor in /etc/consul/conf.d/patroni.json
does not include a port:
{
"services": [
...,
{
"id": "patroni-replica",
"name": "patroni",
"tags": [
"replica"
],
"checks": [
{
"http": "http://0.0.0.0:8009/replica",
"interval": "2s"
},
{
"args": [
"systemctl",
"status",
"pgbouncer"
],
"interval": "10s"
}
]
}
]
}
It should look like this (notice the port):
{
"services": [
...,
{
"id": "patroni-replica",
"name": "patroni",
"port": 6432,
"tags": [
"replica"
],
"checks": [
{
"http": "http://0.0.0.0:8009/replica",
"interval": "2s"
},
{
"args": [
"systemctl",
"status",
"pgbouncer"
],
"interval": "10s"
}
]
}
]
}
Execution Plan
Taken from https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7656#note_207343241
https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1686
-
pgbouncer cookbook change: gitlab-cookbooks/gitlab-pgbouncer!15 (merged) - (delivery: soon) @mwasilewski-gitlab, @alejandro, @ahmadsherif -
Application changes for supporting additional records: https://gitlab.com/gitlab-org/gitlab-ee/issues/13858 + https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/15691 - (delivery: Friday 30th) @ahmadsherif @tkuah -
Patch staging with SRV changes - (delivery to Staging on Friday 30th) @ahmadsherif -
Patch production with SRV changes - (delivery to Production on Monday 2nd) @ahmadsherif -
chef-repo change for single pgbouncer consul service: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1686 (requires step 1) - (delivery to Staging on Friday 30th, Production Monday 2nd) @ahmadsherif -
Staging -
Production
-
-
chef-repo change for application gitlab.rb
to use new SRV records to new pgbouncer service instead of A records (requires 2, 5). (delivery to Staging on Friday 30th, Production Monday 2nd) - @ahmadsherif-
Staging -
Production
-
-
chef-repo change for second pgbouncer service on Patroni hosts. (delivery to Staging on Friday 30th or Monday 2nd, Production Monday 2nd or Tuesday 3rd) TBD. - @ahmadsherif -
Staging -
Production
-
-
consul cookbook change (in the future)... to remove the old patroni-replica
(port 0) service. TBD. Lets create a new issue for this
Edited by Michal Wasilewski