Crossplane as a GitLab-managed-app
## Problem to solve
GitLab supports [one-click deployment of helpful services](https://gitlab.com/help/user/project/clusters/eks_and_gitlab/index.html#deploy-services-to-the-cluster) to any connected Kubernetes cluster, many of which support Auto DevOps. To support apps that require a database, an [in-cluster PostgreSQL database](https://docs.gitlab.com/ee/topics/autodevops/#postgresql-database-support) is provisioned by default configured [via environment variables](https://docs.gitlab.com/ee/topics/autodevops/#database). Auto DevOps provides functionality that is often included in a PaaS, but goes beyond this in multiple ways [including support for any Kubernetes cluster on any public cloud](https://docs.gitlab.com/ee/topics/autodevops/#comparison-to-application-platforms-and-paas).
Many applications however, including the [GitLab helm chart](https://docs.gitlab.com/charts/installation/deployment.html#postgresql), do not recommend an in-cluster database for production use and also rely on additional app infrastructure services like Redis and Buckets that must be separately provisioned and securely connected to a Kubernetes application.
Solutions like the GCP implementation of Open Service Broker [have been deprecated](https://cloud.google.com/kubernetes-engine/docs/concepts/google-cloud-platform-service-broker) in favor of a Kubernetes-native solution, but one that is Google-specific and closed source.
The goal of this issue is to enhance Auto DevOps to support an open service model for managed service provisioning (DBaaS, Cache, Buckets, etc.) that can securely connect cloud service instances to the deployments/pods in a connected Kubernetes cluster.
## Target audience
This would address a number of target audiences:
* Developers who want to build and deploy apps with Auto DevOps support for provisioning managed cloud services (DBaaS, Cache, Buckets, etc.) into their choice of cloud, managed k8s provider, and independent cloud services.
* CIO responsible for finding a cloud-native DevOps solution that provides a great Kubernetes application developer experience for managed service provisioning.
## Further details
### Goals
The open services model should:
* enable developer choice of app infrastructure:
- Cloud provider: (AWS, GCP, Azure, DO) with managed Kubernetes, networking, etc.
- Independent managed Kubernetes provider: (Giant Swarm, Loodse, PKS)
- Independent managed service provider: (Elastic, MongoDB, CockroachDB)
* enable developer choice between provider-specific and portable services
- provider-specific services: RDS, CloudSQL, BigTable, DynamoDB, SNS
- portable services: PostgreSQL, MySQL, Redis
* support full expressiveness for configuring provider-specific aspects
* provide a Kubernetes-native experience familiar to app developers
* support public cloud providers and in-cluster providers like [Rook](https://rook.io/)
* support running in a single cloud region in a connected Kubernetes cluster
* be open source with a permissive free license, so it can be extended by the community
The Auto DevOps integration should:
* make it easy to add app infrastructure services to your deployment pipeline
* support progressive adoption of advanced functionality
* support real-world apps in production
### Proposal
The [open source Crossplane project](https://github.com/crossplaneio/crossplane) has rapidly evolved over the past months, incorporating the learnings from [deploying the full GitLab stack into multiple clouds using kubectl](https://about.gitlab.com/2019/05/20/gitlab-first-deployed-kubernetes-api-to-multiple-clouds/), including fully-managed services it depends on: PostgreSQL, Redis and Buckets.
Crossplane exposes managed services from multiple cloud providers (AWS, GCP, Azure) as Kubernetes objects, for declarative provisioning and management alongside app deployments/pods using kubectl.
Crossplane runs inside your Kubernetes cluster and provides a [Stack Manager](https://github.com/crossplaneio/crossplane/blob/50356f0d4bac2efe61c0bc1558569997cfb0501a/design/design-doc-stacks.md) for installing Kubernetes extensions via the Kubernetes API. In the current [v0.3 milestone](https://github.com/crossplaneio/crossplane/blob/master/ROADMAP.md#v03---enable-community-to-build-infra-stacks), Crossplane Stacks for [GCP](https://github.com/crossplaneio/crossplane/issues/615), [AWS](https://github.com/crossplaneio/crossplane/issues/616), [Azure](https://github.com/crossplaneio/crossplane/issues/617) are being enhanced use the best-practices embodied in the [crossplane-runtime](https://github.com/crossplaneio/crossplane-runtime), upgraded to support single-region secure connectivity, and moved out-of-tree where they can be extended or adapted by the community.
Some ideas for GitLab Auto DevOps integration include extending the [environment variables](https://docs.gitlab.com/ee/topics/autodevops/#database) approach or to provide a richer experience by clicking the “Manage” button on the Crossplane Managed App (screenshot below).
This issue proposes:
* Making Crossplane available as a [GitLab Managed App](https://docs.gitlab.com/ee/user/clusters/applications.html), that can be installed into a connected Kubernetes cluster alongside ingress, cert-manager, and prometheus controllers.
* Modifying AutoDevOps code to support provisioning fully-managed services from AWS, GCP, or Azure using Crossplane.
* Validating that real-world apps (e.g. [sock shop](https://github.com/microservices-demo/microservices-demo/blob/master/internal-docs/design.md)) can be deployed with the managed services they depend on.
## What does success look like, and how can we measure that?
* Apps deployed with Auto DevOps can provision the managed services they depend on.
* Opportunity to take advantage of new capabilities delivered by Crossplane, including the planned [Rook](https://rook.io/) integration.
* Support for additional clouds and independent managed service offerings.
## DRAFT: How could this be demonstrated?
### 1) Install Crossplane
* [Add a new GKE cluster or add an existing Kubernetes cluster](https://docs.gitlab.com/ee/user/project/clusters/#adding-and-removing-clusters)
* Add Helm as a [GitLab managed app](https://docs.gitlab.com/ee/user/clusters/applications.html)
* Add Crossplane as a [GitLab managed app](https://docs.gitlab.com/ee/user/clusters/applications.html) with an infrastructure stack for managed service provisioning:
- [GCP Stack](https://github.com/crossplaneio/stack-gcp)
- [AWS Stack](https://github.com/crossplaneio/stack-aws)
- [Azure Stack](https://github.com/crossplaneio/stack-azure)
* Allow the default Crossplane configuration files to be written to a `crossplane` folder in your GitLab project
* `crossplane/provider.yaml` - GCP project and service account secret reference
* `crossplane/classes` - populated with classes of service available to your GitLab project:
* postgres-standard.yaml, postgres-ha.yaml, redis-standard.yaml, etc.
### 2) Configure Crossplane for use with your cloud provider
* Create a cloud service account with roles for provisioning managed services.
* Create a Kubernetes `Secret` with service account credentials in the connected Kubernetes cluster.
* Configure the Crossplane `provider.yaml` to use that secret and set your project and region.
* Apply all yaml files in the `crossplane` folder using `kubectl apply` from your pipeline.
**Note:** Auto DevOps could automate much of this.
The following example shows using a hand-crafted pipeline.
**GCP example**
Create a GCP service account with the following roles in your project.
```
Cloud SQL Admin
Cloud Memorystore Redis Admin
Storage Admin
Service Account User
```
Create a [CI/CD environment variable](https://docs.gitlab.com/ee/ci/variables/#variable-types) with service account credentials: BASE64ENCODED_GCP_PROVIDER_CREDS.
```sh
base64 gcp-credentials.json | tr -d "\n"
```
Apply a Kubernetes `Secret` to the connected Kubernetes cluster from your pipeline.
```yaml
apiVersion: v1
kind: Secret
metadata:
name: example-provider-gcp
namespace: crossplane-system
type: Opaque
data:
credentials.json: <BASE64ENCODED_GCP_PROVIDER_CREDS>
```
Customize the `crossplane/provider.yaml` in your project, and `kubectl apply` it from your pipeline.
```yaml
apiVersion: gcp.crossplane.io/v1alpha1
kind: Provider
metadata:
name: example
namespace: crossplane-system
spec:
credentialsSecretRef:
name: example-provider-gcp
key: credentials.json
projectID: <PROJECT_ID>
```
Apply all `crossplane/classes` with `kubectl apply` from your pipeline, so they're available for provisioning managed service instances.
* postgres-standard.yaml, postgres-ha.yaml, redis-standard.yaml, etc.
### 3) Use Crossplane to provision a managed PostgreSQL instance
* Add a `crossplane/claims/postgres-instance.yaml` to your project and `kubectl apply` it from your pipeline.
```yaml
apiVersion: database.crossplane.io/v1alpha2
kind: PostgreSQLInstance
metadata:
name: postgres-instance
namespace: my-app
spec:
classRef:
kind: PostgresSQLInstanceClass
apiVersion: database.gcp.crossplane.io/v1alpha2
name: postgres-ha
namespace: my-app
writeConnectionSecretToRef:
name: postgres-instance
engineVersion: "9.6"
```
Use the `postgres-instance` from your application deployment.
## Links / references
cc @deuley
## Related issues
### GitLab CI pipeline - hand crafted:
- Install as a [GitLab managed app](https://docs.gitlab.com/ee/user/clusters/applications.html) into a connected Kubernetes cluster
- Select an Infrastructure Stack (GCP, AWS, or Azure) for use with the Crossplane [managed app](https://docs.gitlab.com/ee/user/clusters/applications.html)
- Accept default Crossplane [managed app](https://docs.gitlab.com/ee/user/clusters/applications.html) configuration to be written to the `crossplane` folder in the GitLab project
- provider.yaml, resource class templates, etc. for one of: GCP, AWS, or Azure
- Configure Crossplane for use with GCP, AWS, Azure for managed service provisioning from the Kubernetes API
- configure provider credentials, region, location, secure connectivity, etc.
- Document pipeline example to Provision a managed PostgreSQL instance using Crossplane Crossplane [managed app](https://docs.gitlab.com/ee/user/clusters/applications.html)
- create `crossplane/claims/postgres-instance.yaml`
- `kubectl apply` from GitLab CI pipeline
- securely use the provisioned PostgreSQL instance from deployed application using populated Kubernetes `Secret`
### GitLab Auto DevOps - auto generated pipeline:
**Auto-configure Crossplane to work with your cloud provider**
- configure via GUI, but writes config to version controlled project
- provider.yaml populated as part of GKE cluster creation flow, etc.
- auto-detect region
- auto configure resource classes and environmental info to work with target cluster
**Deploy managed PostgreSQL using [Auto DevOps database environment variables](https://docs.gitlab.com/ee/topics/autodevops/#database)**
- Provision managed PostgreSQL instance via Crossplane using POSTGRES_MANAGED [environment variable](https://docs.gitlab.com/ee/topics/autodevops/#database)
- auto-creates a `crossplane/claims/postgres-instance.yaml` and applies it as part of your pipeline
- managed alternative to the unmanaged in-cluster PostgreSQL container, more suitable to production workloads
**Deploy managed Redis using [Auto DevOps environment variables](https://docs.gitlab.com/ee/topics/autodevops/#database)**
- Provision managed PostgreSQL instance via Crossplane using REDIS_MANAGED, new REDIS_* [environment variables](https://docs.gitlab.com/ee/topics/autodevops/#database)
- auto-creates a `crossplane/claims/postgres-instance.yaml` and applies it as part of your pipeline
- managed alternative to the unmanaged in-cluster PostgreSQL container, more suitable to production workloads
**Configure GitLab auto-devops managed services via Configure button**
- Provision managed services by configuring Crossplane [managed app](https://docs.gitlab.com/ee/user/clusters/applications.html)
- configure via GUI, but writes config to version controlled project
- auto-creates a `crossplane/claims/postgres-instance.yaml` and applies it as part of your pipeline
- managed alternative to the unmanaged in-cluster PostgreSQL container, more suitable to production workloads
epic