Unicorn/Workhorse K8s service port name issue with Istio ingress gateway
Summary
When using Istio (1.4.x) for ingress, requests to the unicorn service will fail when using http2 if the services' port name does not begin with http, (see istio protocol selection).
There is a helm option to control the service name for unicorn (gitlab.unicorn.service.name) but the same does not exist for workhorse as it is hardcoded in the service template.
To immediately correct, I modified the unicorn service as below:
apiVersion: v1
kind: Service
metadata:
labels:
app: unicorn
chart: unicorn-3.0.1
heritage: Tiller
release: gitlab
name: gitlab-unicorn
spec:
ports:
- name: http-unicorn
port: 8080
protocol: TCP
targetPort: 8080
- name: http-workhorse
port: 8181
protocol: TCP
targetPort: 8181
selector:
app: unicorn
release: gitlab
sessionAffinity: None
type: ClusterIP
Making the workhorse service port name configurable would suffice to resolve.
Steps to reproduce
- Deploy Istio and ingress gateway
- Do NOT use sidecar injection for the gitlab namespace
- Deploy Gitlab in the cluster not using nginx ingress
- Use provided VirtualService configuration for istio.
Configuration used
Virtual Service:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: gitlab-istio-vs
namespace: gitlab
spec:
hosts:
- "gitlab.domain.io"
gateways:
- ingressnamespace/ingressgateway
http:
- match:
- uri:
prefix: /
route:
- destination:
port:
number: 8181
host: gitlab-unicorn.gitlab.svc.cluster.local
headers:
response:
set:
Strict-Transport-Security: 'max-age=31536000'
- match:
- uri:
prefix: /admin/sidekiq
route:
- destination:
port:
number: 8080
host: gitlab-unicorn.gitlab.svc.cluster.local
headers:
response:
set:
Strict-Transport-Security: 'max-age=31536000'
Destination Rule:
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: gitlab-no-tls-dest-rule
namespace: gitlab
spec:
host: "*.gitlab.svc.cluster.local"
trafficPolicy:
loadBalancer:
simple: ROUND_ROBIN
tls:
mode: DISABLE
Current behavior
Using http 1.1, the requests succeed as expected. Using http2, a 503 is returned.
Expected behavior
The protocol between the ingress gateway and the service should be negotiated based on the services' port name, http1.1 and http2 requests should both succeed.
Versions
- Chart: 3.0.1
- appVersion: 12.7.2
- Platform:
- Cloud: EKS
- Kubernetes: (
kubectl version
)- Client: v1.14.9
- Server: v1.14.9
- Helm: (
helm version
)- Client: v2.16.1
- Server: NA (helm template/kubectl apply)
- Istio: 1.4.3
Relevant logs
Can be tested using curl or git client.
curl -v https://gitlab.domain.io/group/project.git --http2
and
curl -v https://gitlab.domain.io/group/project.git --http1.1
Ingress Gateway request logs:
Before changing the workhorse service port name:
{"bytes_sent":"95","upstream_cluster":"outbound|8181||gitlab-unicorn.gitlab.svc.cluster.local","downstream_remote_address":"10.0.42.30:11636","authority":"gitlab.domain.io","path":"group/project.git/info/refs?service=git-upload-pack","protocol":"HTTP/2","upstream_service_time":"-","upstream_local_address":"-","duration":"0","downstream_local_address":"10.0.21.192:443","upstream_transport_failure_reason":"-","route_name":"-","response_code":"503","user_agent":"git/2.22.2","response_flags":"UC","start_time":"2020-01-29T04:34:52.062Z","method":"GET","request_id":"a8c147df-14b5-9f11-9a23-365318428814","upstream_host":"10.0.41.146:8181","x_forwarded_for":"10.0.42.30","requested_server_name":"gitlab.domain.io","bytes_received":"0","istio_policy_status":"-"}
After changing the workhorse service port name:
{"bytes_sent":"26","upstream_cluster":"outbound|8181||gitlab-unicorn.gitlab.svc.cluster.local","downstream_remote_address":"10.0.41.51:30932","authority":"gitlab.domain.io","path":"group/project.git/info/refs?service=git-upload-pack","protocol":"HTTP/2","upstream_service_time":"238","upstream_local_address":"-","duration":"238","downstream_local_address":"10.0.21.192:443","upstream_transport_failure_reason":"-","route_name":"-","response_code":"401","user_agent":"git/2.22.2","response_flags":"-","start_time":"2020-01-29T04:45:04.452Z","method":"GET","request_id":"70702313-bc36-9f41-9ed9-90180b790040","upstream_host":"10.0.41.146:8181","x_forwarded_for":"10.0.41.51","requested_server_name":"gitlab.domain.io","bytes_received":"0","istio_policy_status":"-"}
{"requested_server_name":"gitlab.domain.io","bytes_received":"0","istio_policy_status":"-","bytes_sent":"257","upstream_cluster":"outbound|8181||gitlab-unicorn.gitlab.svc.cluster.local","downstream_remote_address":"10.0.41.51:30932","authority":"gitlab.domain.io","path":"group/project.git/info/refs?service=git-upload-pack","protocol":"HTTP/2","upstream_service_time":"127","upstream_local_address":"-","duration":"127","downstream_local_address":"10.0.21.192:443","upstream_transport_failure_reason":"-","route_name":"-","response_code":"200","user_agent":"git/2.22.2","response_flags":"-","start_time":"2020-01-29T04:45:04.692Z","method":"GET","request_id":"80fa7330-1272-9780-819d-2b55722d781e","upstream_host":"10.0.41.200:8181","x_forwarded_for":"10.0.41.51"}