Epic: Auto DevOps support for managed service provisioning

Problem to solve

GitLab supports one-click deployment of helpful services to any connected Kubernetes cluster, many of which support Auto DevOps. To support apps that require a database, an in-cluster PostgreSQL database is provisioned by default configured via environment variables. Auto DevOps provides functionality that is often included in a PaaS, but goes beyond this in multiple ways including support for any Kubernetes cluster on any public cloud.

Many applications however, including the GitLab helm chart, do not recommend an in-cluster database for production use and also rely on additional app infrastructure services like Redis and Buckets that must be separately provisioned and securely connected to a Kubernetes application.

Solutions like the GCP implementation of Open Service Broker have been deprecated in favor of a Kubernetes-native solution, but one that is Google-specific and closed source.

The goal of this issue is to enhance Auto DevOps to support an open service model for managed service provisioning (DBaaS, Cache, Buckets, etc.) that can securely connect cloud service instances to the deployments/pods in a connected Kubernetes cluster.

Target audience

This would address a number of target audiences:

  • Developers who want to build and deploy apps with Auto DevOps support for provisioning managed cloud services (DBaaS, Cache, Buckets, etc.) into their choice of cloud, managed k8s provider, and independent cloud services.
  • CIO responsible for finding a cloud-native DevOps solution that provides a great Kubernetes application developer experience for managed service provisioning.

Further details

Goals

The open services model should:

  • enable developer choice of app infrastructure:
    • Cloud provider: (AWS, GCP, Azure, DO) with managed Kubernetes, networking, etc.
    • Independent managed Kubernetes provider: (Giant Swarm, Loodse, PKS)
    • Independent managed service provider: (Elastic, MongoDB, CockroachDB)
  • enable developer choice between provider-specific and portable services
    • provider-specific services: RDS, CloudSQL, BigTable, DynamoDB, SNS
    • portable services: PostgreSQL, MySQL, Redis
  • support full expressiveness for configuring provider-specific aspects
  • provide a Kubernetes-native experience familiar to app developers
  • support public cloud providers and in-cluster providers like Rook
  • support running in a single cloud region in a connected Kubernetes cluster
  • be open source with a permissive free license, so it can be extended by the community

The Auto DevOps integration should:

  • make it easy to add app infrastructure services to your deployment pipeline
  • support progressive adoption of advanced functionality
  • support real-world apps in production

Proposal

The open source Crossplane project has rapidly evolved over the past months, incorporating the learnings from deploying the full GitLab stack into multiple clouds using kubectl, including fully-managed services it depends on: PostgreSQL, Redis and Buckets.

Crossplane exposes managed services from multiple cloud providers (AWS, GCP, Azure) as Kubernetes objects, for declarative provisioning and management alongside app deployments/pods using kubectl.

Crossplane runs inside your Kubernetes cluster and provides a Stack Manager for installing Kubernetes extensions via the Kubernetes API. In the current v0.3 milestone, Crossplane Stacks for GCP, AWS, Azure are being enhanced use the best-practices embodied in the crossplane-runtime, upgraded to support single-region secure connectivity, and moved out-of-tree where they can be extended or adapted by the community.

Some ideas for GitLab Auto DevOps integration include extending the environment variables approach or to provide a richer experience by clicking the “Manage” button on the Crossplane Managed App (screenshot below).

This issue proposes:

  • Making Crossplane available as a GitLab Managed App, that can be installed into a connected Kubernetes cluster alongside ingress, cert-manager, and prometheus controllers.
  • Modifying AutoDevOps code to support provisioning fully-managed services from AWS, GCP, or Azure using Crossplane.
  • Validating that real-world apps (e.g. sock shop) can be deployed with the managed services they depend on.

Crossplane support

What does success look like, and how can we measure that?

  • Apps deployed with Auto DevOps can provision the managed services they depend on.
  • Opportunity to take advantage of new capabilities delivered by Crossplane, including the planned Rook integration.
  • Support for additional clouds and independent managed service offerings.

DRAFT: How could this be demonstrated?

1) Install Crossplane

2) Configure Crossplane for use with your cloud provider

  • Create a cloud service account with roles for provisioning managed services.
  • Create a Kubernetes Secret with service account credentials in the connected Kubernetes cluster.
  • Configure the Crossplane provider.yaml to use that secret and set your project and region.
  • Apply all yaml files in the crossplane folder using kubectl apply from your pipeline.

Note: Auto DevOps could automate much of this.

The following example shows using a hand-crafted pipeline.

GCP example

Create a GCP service account with the following roles in your project.

Cloud SQL Admin
Cloud Memorystore Redis Admin
Storage Admin
Service Account User

Create a CI/CD environment variable with service account credentials: BASE64ENCODED_GCP_PROVIDER_CREDS.

base64 gcp-credentials.json | tr -d "\n"

Apply a Kubernetes Secret to the connected Kubernetes cluster from your pipeline.

apiVersion: v1
kind: Secret
metadata:
  name: example-provider-gcp
  namespace: crossplane-system
type: Opaque
data:
  credentials.json: <BASE64ENCODED_GCP_PROVIDER_CREDS>

Customize the crossplane/provider.yaml in your project, and kubectl apply it from your pipeline.

apiVersion: gcp.crossplane.io/v1alpha1
kind: Provider
metadata:
  name: example
  namespace: crossplane-system
spec:
  credentialsSecretRef:
    name: example-provider-gcp
    key: credentials.json
  projectID: <PROJECT_ID>

Apply all crossplane/classes with kubectl apply from your pipeline, so they're available for provisioning managed service instances.

  • postgres-standard.yaml, postgres-ha.yaml, redis-standard.yaml, etc.

3) Use Crossplane to provision a managed PostgreSQL instance

  • Add a crossplane/claims/postgres-instance.yaml to your project and kubectl apply it from your pipeline.
apiVersion: database.crossplane.io/v1alpha2
kind: PostgreSQLInstance
metadata:
  name: postgres-instance
  namespace: my-app
spec:
  classRef:
    kind: PostgresSQLInstanceClass
    apiVersion: database.gcp.crossplane.io/v1alpha2
    name: postgres-ha
    namespace: my-app
  writeConnectionSecretToRef:
    name: postgres-instance
  engineVersion: "9.6"

Use the postgres-instance from your application deployment.

Links / references

cc @deuley

Related issues

GitLab CI pipeline - hand crafted:

Install and Configure Crossplane as a GitLab managed app

  • Install Crossplane as a managed app into a connected Kubernetes cluster
  • Select an Infrastructure Stack (GCP, AWS, or Azure) for use with the Crossplane managed app
  • Accept default Crossplane managed app configuration to be written to the crossplane folder in the GitLab project
    • provider.yaml, resource class templates, etc. for one of: GCP, AWS, or Azure
  • Configure Crossplane for use with GCP, AWS, Azure for managed service provisioning from the Kubernetes API
    • configure provider credentials, region, location, secure connectivity, etc.

Deploy a managed PostgreSQL instance from a GitLab pipeline using Crossplane

  • Document pipeline example to Provision a managed PostgreSQL instance using Crossplane managed app
    • create crossplane/claims/postgres-instance.yaml
    • kubectl apply from GitLab CI pipeline
    • securely use the provisioned PostgreSQL instance from deployed application using populated Kubernetes Secret

GitLab Auto DevOps - auto generated pipeline:

Auto-configure Crossplane to work with your cloud provider

  • configure via GUI, but writes config to version controlled project
  • provider.yaml populated as part of GKE cluster creation flow, etc.
  • auto-detect region
  • auto configure resource classes and environmental info to work with target cluster

Deploy managed PostgreSQL using Auto DevOps database environment variables

  • Provision managed PostgreSQL instance via Crossplane using POSTGRES_MANAGED environment variable
    • auto-creates a crossplane/claims/postgres-instance.yaml and applies it as part of your pipeline
    • managed alternative to the unmanaged in-cluster PostgreSQL container, more suitable to production workloads

Deploy managed Redis using Auto DevOps environment variables

  • Provision managed PostgreSQL instance via Crossplane using REDIS_MANAGED, new REDIS_* environment variables
    • auto-creates a crossplane/claims/postgres-instance.yaml and applies it as part of your pipeline
    • managed alternative to the unmanaged in-cluster PostgreSQL container, more suitable to production workloads

Configure GitLab auto-devops managed services via Configure button

  • Provision managed services by configuring Crossplane managed app
    • configure via GUI, but writes config to version controlled project
    • auto-creates a crossplane/claims/postgres-instance.yaml and applies it as part of your pipeline
    • managed alternative to the unmanaged in-cluster PostgreSQL container, more suitable to production workloads
Edited by 🤖 GitLab Bot 🤖