EKS - Which load balancer backend protocol to use?
Summary
I get Bad protocol version identification in gitlab-shell logs and I can't clone using SSH.
Configuration used
I deployed the chart using classic load balancer (ELB) with following config:
global:
hosts:
domain: mydomain.com
ingress:
configureCertmanager: false
nginx-ingress:
controller:
service:
annotations:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: xxx
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "https"I tried to change nginx-ingress annotations but I can access webapp only with service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "https", otherwise I get:
400 Bad Request - The plain HTTP request was sent to HTTPS port
BTW, it would be great to add a section about AWS ingress config (with certificate info) in chart documentation... It would be very helpful :-)
Versions
- Chart: 4.4.3
- Platform:
- Cloud: EKS
- Kubernetes:
- Client: 1.19.2
- Server: 1.17.9
- Helm:
- Server: 3.3.4
Relevant logs
> kubectl logs gitlab-gitlab-shell-7588b8cf64-zjl4n
+ /scripts/set-config /etc/gitlab-shell /srv/gitlab-shell
Begin parsing .erb files from /etc/gitlab-shell
Writing /srv/gitlab-shell/config.yml
Copying other config files found in /etc/gitlab-shell
+ exec /bin/sh -c '"/scripts/process-wrapper"'
Using existing Host Keys
Starting OpenSSH
Tailing Logs
==> /var/log/gitlab-shell/gitlab-shell.log <==
==> /var/log/gitlab-shell/ssh.log <==
Server listening on 0.0.0.0 port 2222.
Server listening on :: port 2222.
Bad protocol version identification '\026\003\001' from 172.16.1.163 port 43956
Bad protocol version identification '\026\003\001' from 172.16.1.163 port 35924
Bad protocol version identification '\026\003\001' from 172.16.1.176 port 54962
Bad protocol version identification '\026\003\001' from 172.16.1.136 port 33522
Bad protocol version identification '\026\003\001' from 172.16.1.163 port 35982> kubectl describe svc gitlab-gitlab-shell
Name: gitlab-gitlab-shell
Namespace: default
Labels: app=gitlab-shell
app.kubernetes.io/managed-by=Helm
chart=gitlab-shell-4.4.3
heritage=Helm
release=gitlab
Annotations: meta.helm.sh/release-name: gitlab
meta.helm.sh/release-namespace: default
Selector: app=gitlab-shell,release=gitlab
Type: ClusterIP
IP: 10.100.61.19
Port: ssh 22/TCP
TargetPort: 2222/TCP
Endpoints: 172.16.1.104:2222,172.16.1.137:2222
Session Affinity: None
Events: <none>> kubectl describe svc gitlab-nginx-ingress-controller
Name: gitlab-nginx-ingress-controller
Namespace: default
Labels: app=nginx-ingress
app.kubernetes.io/managed-by=Helm
chart=nginx-ingress-0.30.0-1
component=controller
heritage=Helm
release=gitlab
Annotations: field.cattle.io/publicEndpoints:
[{"addresses":["xxx.elb.amazonaws.com"],"port":80,"protocol":"TCP","serviceName":"def...
meta.helm.sh/release-name: gitlab
meta.helm.sh/release-namespace: default
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:xxx
Selector: app=nginx-ingress,component=controller,release=gitlab
Type: LoadBalancer
IP: 10.100.64.39
LoadBalancer Ingress: xxx.elb.amazonaws.com
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 30778/TCP
Endpoints: 172.16.1.136:80,172.16.1.163:80,172.16.1.176:80
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 31355/TCP
Endpoints: 172.16.1.136:443,172.16.1.163:443,172.16.1.176:443
Port: gitlab-shell 22/TCP
TargetPort: gitlab-shell/TCP
NodePort: gitlab-shell 31304/TCP
Endpoints: 172.16.1.136:22,172.16.1.163:22,172.16.1.176:22
Session Affinity: None
External Traffic Policy: Local
HealthCheck NodePort: 32059
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadBalancer 8m16s (x8 over 25h) service-controller Ensuring load balancer
Normal EnsuredLoadBalancer 8m15s (x8 over 25h) service-controller Ensured load balancer> kubectl describe ingress gitlab-webservice
Name: gitlab-webservice
Namespace: default
Address: xxx.elb.amazonaws.com
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
gitlab-wildcard-tls terminates gitlab.mydomain.com
Rules:
Host Path Backends
---- ---- --------
gitlab.mydomain.com
/ gitlab-webservice:8181 (172.16.1.165:8181,172.16.1.181:8181)
/admin/sidekiq gitlab-webservice:8080 (172.16.1.165:8080,172.16.1.181:8080)
Annotations: field.cattle.io/publicEndpoints:
[{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"default:gitlab-webservice","ingressName":"default:gitlab-webservice","host...
kubernetes.io/ingress.class: gitlab-nginx
kubernetes.io/ingress.provider: nginx
meta.helm.sh/release-name: gitlab
meta.helm.sh/release-namespace: default
nginx.ingress.kubernetes.io/proxy-body-size: 512m
nginx.ingress.kubernetes.io/proxy-connect-timeout: 15
nginx.ingress.kubernetes.io/proxy-read-timeout: 600
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal UPDATE 53m (x9 over 25h) nginx-ingress-controller Ingress default/gitlab-webservice
Normal UPDATE 53m (x9 over 25h) nginx-ingress-controller Ingress default/gitlab-webservice
Normal UPDATE 53m (x9 over 25h) nginx-ingress-controller Ingress default/gitlab-webserviceEdited by Cedric Thiebault