Research Spike: Understanding How to implement traffic routing for Kubernetes
Problem to solve
In order to support advanced deployment strategies such as traffic vectoring or blue/green deployments, we need to implement a load balancer that can move the traffic between k8s clusters.
Intended users
Further details
Ingress (Probably the best option)
Ingress is actually NOT a type of service. Instead, it sits in front of multiple services and act as a “smart router” or entrypoint into your cluster.
You can do a lot of different things with an Ingress, and there are many types of Ingress controllers that have different capabilities.
The default GKE ingress controller will spin up a HTTP(S) Load Balancer for you. This will let you do both path based and subdomain based routing to backend services. For example, you can send everything on foo.yourdomain.com to the foo service, and everything under the yourdomain.com/bar/ path to the bar service.
The YAML for a Ingress object on GKE with a L7 HTTP Load Balancer might look like this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
spec:
backend:
serviceName: other
servicePort: 8080
rules:
- host: foo.mydomain.com
http:
paths:
- backend:
serviceName: foo
servicePort: 8080
- host: mydomain.com
http:
paths:
- path: /bar/*
backend:
serviceName: bar
servicePort: 8080Load Balancer
There are two methods supporting in Kubernetes to achieve the load balancer service:
Configuration file
To create an external load balancer, add the following line to your service configuration file:
type: LoadBalancer
Your configuration file might look like:
apiVersion: v1
kind: Service
metadata:
name: example-service
spec:
selector:
app: example
ports:
- port: 8765
targetPort: 9376
type: LoadBalancerUsing kubectl You can alternatively create the service with the kubectl expose command and its --type=LoadBalancer flag:
kubectl expose rc example --port=8765 --target-port=9376 \
--name=example-service --type=LoadBalancerAnother option may be (less recommended)
NodePort
A NodePort service is the most primitive way to get external traffic directly to your service. NodePort, as the name implies, opens a specific port on all the Nodes (the VMs), and any traffic that is sent to this port is forwarded to the service.
The YAML for a NodePort service looks like this:
apiVersion: v1
kind: Service
metadata:
name: my-nodeport-service
spec:
selector:
app: my-app
type: NodePort
ports:
- name: http
port: 80
targetPort: 80
nodePort: 30036
protocol: TCPProposal
There are 3 proposed approaches described above, please evaluate these approaches and provide a breakdown of pros and cons for each approach. If possible, also provide a rough estimate of effort involved (S, M, L, XL).
According to the recommendations below: Ingress (and Ingres Controller providers like NGINX, HAproxy) are probably your best bet moving forward. These are native mechanics of K8S, which we should leverage if at all possible.
Unless the consumer has a very good understanding of K8S as a platform, they absolutely don't want to be managing LoadBalancers providers themselves. Don't get me started on the networking requirements of NodePorts, and exposing VM nodes directly to the internet over random ports, across who knows which platform.
- An Ingres controller is deployed, providing a Service. This Service is generally of type LoadBalancer or NodePort. Traffic is routed from outside the cluster, into that Service by whichever means are deployed.
- The controller's Pods that back the Service receive the traffic for all Ingress's serviced by the controller (see
kubernetes.io/ingress.provider,kubernetes.io/ingress.classannotations) - That Pod then decide the appropriate destination for the traffic, based upon whatever logic the server binary inside makes use of.
- In the case of NGINX Ingress, there is a fully rendered
/etc/nginx/nginx.conf
- In the case of NGINX Ingress, there is a fully rendered
- That binary then forwards the traffic, as it deems fit, to the appropriate Services or Pods directly.
/etc/nginx/nginx.conf key sections (trimmmed)
upstream upstream_balancer {
server 0.0.0.1; # placeholder
balancer_by_lua_block {
balancer.balance()
}
keepalive 32;
keepalive_timeout 60s;
keepalive_requests 100;
}
## start server gitlab.separate-containers.party
server {
server_name gitlab.separate-containers.party ;
listen 80;
set $proxy_upstream_name "-";
listen 443 ssl ;
location /admin/sidekiq {
set $namespace "default";
set $ingress_name "demo-unicorn";
set $service_name "demo-unicorn";
set $service_port "8080";
set $location_path "/admin/sidekiq";
set $proxy_upstream_name "default-demo-unicorn-8080";
proxy_pass http://upstream_balancer;
}
location / {
set $namespace "default";
set $ingress_name "demo-unicorn";
set $service_name "demo-unicorn";
set $service_port "8181";
set $location_path "/";
set $proxy_upstream_name "default-demo-unicorn-8181";
proxy_pass http://upstream_balancer;
}
}
# TCP services
server {
preread_by_lua_block {
ngx.var.proxy_upstream_name="tcp-default-demo-gitlab-shell-22";
}
listen 22;
proxy_timeout 600s;
proxy_pass upstream_balancer;
}