README.md 7.31 KB
Newer Older
Mike Terhar's avatar
Mike Terhar committed
1
Run code-server on Kubernetes.... and load that image up with build tools! 
Mike Terhar's avatar
Mike Terhar committed
2
3
4
5

## Roll out your own

1.  Fork this repo 
Mike Terhar's avatar
Mike Terhar committed
6
7
8
9
10
1.  Pick a nice kubernetes cluster
1.  Make sure it has persistent storage (I like OpenEBS)
1.  Make sure it has ingress (I like Kubernetes distributed nginx)
1.  Manually deploy `kubectl apply -f code-server.yml` 
1.  Enjoy as pods, ingress, certificates all spring to life
Mike Terhar's avatar
Mike Terhar committed
11

Mike Terhar's avatar
Mike Terhar committed
12
### Kubernetes Stuff
Mike Terhar's avatar
Mike Terhar committed
13

Mike Terhar's avatar
Mike Terhar committed
14
Be a cluster admin with Kubectl.  I didn't use helm or any templating engines for this... 
Mike Terhar's avatar
Mike Terhar committed
15

Mike Terhar's avatar
Mike Terhar committed
16
17
18
1. An persistent volume that can be mounted in `/home/coder`
1. Go into the code-server.yml file and set the PASSWORD variable.
1. First time you start it, copy the `/tmp_coder/coder/` dotfiles to `~`
Mike Terhar's avatar
Mike Terhar committed
19

Mike Terhar's avatar
Mike Terhar committed
20
Make sure not to commit the secrets to your repo unless you treat it as a private/secret storage location.
Mike Terhar's avatar
Mike Terhar committed
21
22
23

## Using it

Mike Terhar's avatar
Mike Terhar committed
24
25
26
27
28
29
30
Open a browser, open the domain in the ingress. Use your password to get in. Then configure stuff:

*  VS Code has lots of editor preferences, enjoy customizing to your liking
*  Setup git using the `git config --global user.name "John Doe"` and `git config --global user.email johndoe@example.com`
*  Validate that the `/home/coder` directory is in a persistent volume so that configuration and extensions don't get lost
*  The uid and gid are 1000 if that needs to be adjusted in the persistent volume, though `securityContext` is set in the spec
*  I clone repos directly into /home/coder/ and then use the `file > open` to focus on a project while working on it. 
Mike Terhar's avatar
Mike Terhar committed
31
*  To make more pods in the cluster, use the service account called `developer`
Mike Terhar's avatar
Mike Terhar committed
32
33
34
35
36
37

## Build tools

It already has: 

1. Python 3
38
39
40
41
1. Go 1.15
1. Hugo 0.74
1. kubectl 1.19
1. helm 3.3
Mike Terhar's avatar
Mike Terhar committed
42
43
44

### Adding more

Mike Terhar's avatar
Mike Terhar committed
45
46
47
48
49
Easiest way is to use the linux install scripts to pull binaries, throw them in `/usr/local/bin` during the docker build. 

After a new container is built, kill the pod using `kubectl scale deploy code-server --replicas=0 -n=code-server`. After the pod is unscheduled, set replicas back to 1 and it'll pull the fresh container.  

If you want to be more deliberate about what version is pulled (to allow for roll-backs), you can tag the images and update the `image` in the spec. 
Mike Terhar's avatar
Mike Terhar committed
50

Mike Terhar's avatar
Mike Terhar committed
51
## Deficiencies with Kubernetes
Mike Terhar's avatar
Mike Terhar committed
52

Mike Terhar's avatar
Mike Terhar committed
53
54
1.  Docker-in-docker builds aren't an option without severely impacting security
1.  File storage is loosely coupled so access tokens, ssh keys, etc all end up there
Mike Terhar's avatar
Mike Terhar committed
55
56
57
58
59
60

## Notify yourself

I use a homelab repository for storing my kubespray and kubernetes application specs. The [lab assistant](https://gitlab.com/brownfield-dev/public/lab-assistant) job in this repo makes an MR in my homelab repo to update the spec to the newest tag as new tags are added. 

To use a similar job, add a project CI variable with your Personal Access Token called "GL_TOKEN". This needs API access to create a branch and create an MR in the target project.  Then get the integer identifiers for your user account and the target project and put them in the `.gitlab-ci.yml` in the arguments. 
Mike Terhar's avatar
Mike Terhar committed
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133

## Make more pods

Apply the code-server.yml file and then get a cert and token. 

```bash
kubectl describe sa developer -n development
kubectl get secret developer-token-xxxxx -n development -o "jsonpath={.data.token}" | base64 -D > developer-token.txt
kubectl get secret developer-token-xxxxx -n development -o "jsonpath={.data['ca\.crt']}" > developer-cert.crt
```

Take those token and cert files and make a kubeconfig out of them. 

```plaintext
apiVersion: v1
kind: Config
preferences: {}

clusters:
- cluster:
    certificate-authority-data: [[CERTIFICATE]]
    server: https://[[IP_ADDRESS]]:6443
  name: homelab
users:
- name: developer
  user:
    as-user-extra: {}
    client-key-data: [[CERTIFICATE AGAIN]]
    token: [[TOKEN]]
contexts:
- context: 
    cluster: homelab
    namespace: development
    user: developer
  name: development-only

current-context: development-only
```

Put this configuration file in the code-server ~/.kube/config and you can start issuing kubectl and helm commands against your own running cluster in the same namespace. 

## Namespace isolation

The spec currently includes a service account and role binding to make the code-server pod capable of adding more pods to its own namespace.  This should also allow creation and removal of services and things like that.  It definitely allows code-server to ruin itself so be careful. 

The isolation currently hasn't been stress tested, but worked about like I expected: 

```plaintext
coder@code-server-598cb4c9d-jdnfm:~/code-server-buildtools$ kubectl get all
NAME                              READY   STATUS    RESTARTS   AGE
pod/code-server-598cb4c9d-jdnfm   1/1     Running   0          6m3s

NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/code-server        ClusterIP   10.233.38.239   <none>        80/TCP    23d
service/code-server-hugo   ClusterIP   10.233.59.166   <none>        80/TCP    23d

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/code-server   1/1     1            1           23d

NAME                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/code-server-598cb4c9d   1         1         1       6m3s
Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User "system:serviceaccount:development:developer" cannot list resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace "development"
coder@code-server-598cb4c9d-jdnfm:~/code-server-buildtools$ kubectl get all -n default
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:development:developer" cannot list resource "pods" in API group "" in the namespace "default"
Error from server (Forbidden): replicationcontrollers is forbidden: User "system:serviceaccount:development:developer" cannot list resource "replicationcontrollers" in API group "" in the namespace "default"
Error from server (Forbidden): services is forbidden: User "system:serviceaccount:development:developer" cannot list resource "services" in API group "" in the namespace "default"
Error from server (Forbidden): daemonsets.apps is forbidden: User "system:serviceaccount:development:developer" cannot list resource "daemonsets" in API group "apps" in the namespace "default"
Error from server (Forbidden): deployments.apps is forbidden: User "system:serviceaccount:development:developer" cannot list resource "deployments" in API group "apps" in the namespace "default"
Error from server (Forbidden): replicasets.apps is forbidden: User "system:serviceaccount:development:developer" cannot list resource "replicasets" in API group "apps" in the namespace "default"
Error from server (Forbidden): statefulsets.apps is forbidden: User "system:serviceaccount:development:developer" cannot list resource "statefulsets" in API group "apps" in the namespace "default"
Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User "system:serviceaccount:development:developer" cannot list resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace "default"
Error from server (Forbidden): jobs.batch is forbidden: User "system:serviceaccount:development:developer" cannot list resource "jobs" in API group "batch" in the namespace "default"
Error from server (Forbidden): cronjobs.batch is forbidden: User "system:serviceaccount:development:developer" cannot list resource "cronjobs" in API group "batch" in the namespace "default"
134
```