WIP: Knative natively in GitLab

What does this MR do?

This MR is yet another PoC how we can integrate with Serverless, in this example Knative to make GitLab aware of functions, routes, and be able to control them.

This MR provides:

• a way to install Knative into existing RBAC cluster,
• a simple interface to present Knative Services with domain,
• a simple way to add Functions from GitLab, it uses runtimes from https://gitlab.com/ayufan/serverless-functions,
• added functions are automatically run as Service and exposed under a domain,
• it allows deploying more complex services/functions from CI as described below,

How to use it?

• Provision cluster,
• Install Knative,
• Add function via Functions tab,
• Push CI repository (optionally)

Serverless tab

1. Choose a registry.gitlab.com/ayufan/serverless-functions/functions/go:1.11.0,
2. Put a function into text field:

import "net/http"

func process(w http.ResponseWriter, r *http.Request) {
  w.Write([]byte("Hello World from GitLab Serverless!"))
}

During the function run, the function code will be assigned to FUNCTION variable and be processed by container image build from https://gitlab.com/ayufan/serverless-functions/tree/master/functions/go/1.11.0.

CI usage

1. service.yaml:

apiVersion: serving.knative.dev/v1alpha1 # Current version of Knative
kind: Service
metadata:
  name: helloworld-go # The name of the app
  namespace: default # The namespace the app will use
spec:
  runLatest:
    configuration:
      revisionTemplate:
        spec:
          container:
            image: gcr.io/knative-samples/helloworld-go # The URL to the image of the app
            env:
            - name: TARGET # The environment variable printed out by the sample app
              value: "Go Sample v1"

2. And .gitlab-ci.yml:

production functions:
  before_script: (install kubectl)
  script: kubectl apply -f service.yaml
  environment: production

Screenshots

Screencast

Functions as convention

We could iterate repository during deployment and:

1. Look for /functions/*/ and perform an automated build, the same way as we do for Auto DevOps,
2. Secondly, if we have automated build image (extension of Kaniko) that can understand a way to build a self-contained source code, we could push above manifest with the request: to clone sources, push to Container Registry (registry.gitlab.com/group/project/functions/function-name:branch), deploy the service with that function using Registry deploy credentials.

Functions as code

The second idea is that we allow a user to create functions from Web Interface, define the language and deploy it for them.

We control Container Registry, and we can perform https://github.com/knative/docs/blob/master/serving/samples/source-to-url-go/service.yaml where we inject code, build the image, push the image, deploy service.

This is done by existing API and using Kubernetes to schedule build and service creation. We can then look out for Services and cross-match them with Functions from GitLab.

Service could be code-provided then (file name from a repository), or user-provided in form of the code copied into text field That way we fully control the function lifecycle.

Currently it is implemented as adding function via Web and function being evaluated dynamically.

Use of functions for modeling complex behavior

Exposing functions as a code allows us to make GitLab aware of the functions and have automated webhooks to created functions from GitLab. This allows scripting how GitLab behaves.

Monitoring of Functions

Since we use Prometheus, the functions could be extended to monitored, ex. a number of replicas, versions running, CPU/memory usage, similar to environments and deployments.

Dealing with Knative

It seems to be quite easy. However, I don't understand all limitations yet. We can easily request all services, routes, we can make modifications for it. We likely can easily build everything too and watch for the status of the build. The API is not yet stable. The building glue if we know language is quite straightforward. The building glue if we don't know language is the same as we have in Auto DevOps today.

Missing pieces and shortcuts

1. functions are dynamically run today, they are not built and pushed to container registry, but rather "run" at runtime, likely we should make a conscious decision when it makes sense: sometimes if we don't have dependencies it is just faster, than building and deploying container, but likely we want to build image always,
2. it implements UI as for purpose of demo presentation, likely it should have a better format for describing functions, more code oriented, maybe similar or equivalent of serverless.yml,
3. does not allow to control domain (the default is being used),
4. it does not deploy under URL, but rather full domain (yet to verify what workflow do Knative support),
5. as stated in 1., it does not use knative-build, so it does not build/push and deploy a function specific image, but rather generic, this is done on purpose, as simpler to achieve and offering for demo purposes the same benefit. To achieve that it also mises read-write project-level docker push credentials (we only offer read-only for now),
6. created function automatically creates knative service, which is represented as a trigger, showing 1-to-1 relocation. Ideally we should have a different data model for Triggers. Create trigger, choose type of trigger (http, pubsub, cronjob, anything else?), and assign function to trigger. Trigger should be then deployed to provider (knative) for example,
7. this does not use Eventing of knative, it seems to be out of scope, but would be natural extension,
8. everything likely should be serverless-as-a-code, or functions-as-a-code,