## DRI @fforster ## Participants ## OKR gitlab-com/gitlab-OKRs#7690 ## Background For some services, Cloud Run is not an applicable runtime. This affects many services that must run as part of self-managed GitLab instances, which are sometimes even "air-gapped". The goal for this epic is to identify and curate non-Cloud-Run use-cases, identify a use-case with a favorable impact/effort ratio, and create a proposal how to bring platform engineering principles to such a use-case. This can mean extending Runway to also cover an additional runtime, or creating a Runway equivalent for such a runtime. <details> <summary>Previous status updates</summary> ## Status 2024-05-22 Not started ## Status 2024-06-05 Reviewing already documented use-cases to understand the addressable market. ## Status 2024-06-12 Set up a test GitLab instance with CI Runners on Kubernetes to better understand the perspective of self-managed customers. ## Status 2024-06-19 The [Runway for Auxiliary Services Vision](https://docs.google.com/document/d/11ChXEl1FosTINCgsPXCW3mK_1T8KD9UIYHJc22tPp_I/edit?usp=sharing) document has been shared with a small group for initial feedback. Next: share more widely and meet with strategic stakeholders to discuss the direction. ## Status 2024-06-26 The [vision document](https://docs.google.com/document/d/11ChXEl1FosTINCgsPXCW3mK_1T8KD9UIYHJc22tPp_I/edit?usp=sharing) has seen lively engagement; thanks for all the feedback! :tanuki-colored-heart: "Auxiliary Services" have been renamed to "Satellite Services". A [Runway for GKE Blueprint](https://docs.google.com/document/d/1Q25BxXJU1CQaez8sl0emcMKgwMgYKSC7km6YC4wVklU/edit?usp=sharing) has been shared with the Runway team. The current prototype has been [demoed to the Scalability Group](https://drive.google.com/drive/folders/1CLmWCkH3rnQ-ZCth_D_NsHxFEOHrStOo) (skip to 3:50 for the demo). ## Status 2024-07-03 An initial draft of a `runway-base` Helm chart has been merged to the `runwayctl` repository, and we have successfully deployed the example service to our dev cluster with it. Creation of shared GKE clusters for small and test services is currently in review. ## Status 2024-07-10 The [Runway for GKE Blueprint](https://docs.google.com/document/d/1Q25BxXJU1CQaez8sl0emcMKgwMgYKSC7km6YC4wVklU/edit?usp=sharing) has been improved with more technical details and separating more clearly between the sub-objectives "create sharable Helm Chart" and "manage GitLab-operated workloads". Work started to separate GitLab/GKE specific concerns into a separate chart that will _not_ be shared with customers. A CI job in the "example-service" builds a Helm Chart that is theoretically sharable. We successfully deployed an example front-end and a back-end to our prototype cluster, demonstrating intra-cluster service resolution and communication using a service mesh. This work was [demoed to the Scalability Group](https://drive.google.com/drive/folders/1MFKryYkz-daqZ1JBNcnsoxYKdmUvJMBR) (timestamps: 35:40 – outline of Q3 work; 48:30 – demo). ## Status 2024-07-17 Helm chart tooling has been merged into `runwayctl`, along with a CI template that will create and push a Helm chart as part of the build/release process. This is still a fast moving target and not yet ready for customer use. (Reference: gitlab-com/gl-infra/platform/runway/team#236) Next: create shared GKE clusters (gitlab-com/gl-infra/platform/runway/team#287) and productionize the `runway-base` Helm chart (gitlab-com/gl-infra/platform/runway/team#283) ## Status 2024-07-24 Work is underway to transition the project from "build a prototype" (Q2 goal) to "build an MVP and deploy a pilot" (Q3 goal): we created issues for concrete next steps, allowing other engineers to contribute. Shared Runway GKE clusters have been created and we're in the process of setting up management infrastructure for them. Product and engineering are collaborating on identifying a suitable pilot use-case/customer. We successfully deployed the "example service" to our prototype cluster using the split `runway-base` / `runway-gke` Helm charts. This proofs the concept of provider-agnostic and publicly sharable base chart, with GitLab/GKE specific "mixin" chart. We also proofed a concept of generating Flux configuration for Runway services using our prototype. Both remove "known unknowns" and solidify our timeline. Next: work with engineers that have indicated that they have spare cycles, to ensure a good ramp-up experience. Transfer (some of) the concepts tested with the prototype cluster to the shared GKE clusters. ## Status 2024-07-31 We're working towards deploying the example service on the shared Runway GKE clusters. This surfaces gaps that we need to address before a pilot onboarding. After consulting with the Foundation group we changed our deployment approach to using fully denormalized Kubernetes resource manifests. This keeps the system simple, simplifies rollbacks, and avoid shortcomings of Flux' `HelmRelease` implementation. </details> ## Status 2024-08-07 **🤔 Problem description:** Figure out how GitLab's internal developer platform, Runway, can be evolved to meet the needs of self-managed customers, and build a prototype. Runway is an internal developer platform allowing teams at GitLab to easily run services on GCP's Cloud Run. However, some large self-managed customers are unable to use SaaS-style services, e.g. due to regulatory requirements, and are asking to run AI Gateway themselves. The goal is to create a sharable artifact, e.g. a Helm chart, while maintaining the convenience and safety for deployments operated by GitLab. **🏗️ What we did:** * Turned up a GitLab instance "by hand", including runners on a Kubernetes cluster and the GitLab agent for Kubernetes, to better understand the end-user perspective. * Published [Runway for Satellite Services Vision](https://docs.runway.gitlab.com/reference/blueprints/satellite-services-vision/), outlining our long term vision and the value provided by increments on the path towards that vision. * Published [Runway for GKE Blueprint](https://docs.google.com/document/d/1Q25BxXJU1CQaez8sl0emcMKgwMgYKSC7km6YC4wVklU/edit?usp=sharing), outlining an implementation of the first iteration described in the vision document. * Built a prototype using the self-managed GitLab test instance and GKE cluster, and Flux. * Contributed some [Flux fixes](https://github.com/fluxcd/source-controller/pull/1529) upstream. * Added common CI tasks, allowing Runway services to easily generate a Helm chart. The generated Helm chart is based on `runway-base`, providing the common structure for all Runway-mangaed services. * Added deployment CD jobs for generating Kubernetes resource manifests from the Helm chart and uploading them to an OCI repository. * Created shared Runway GKE clusters using the Runway Provisioner. **💡 Findings:** * There is ample appetite in GitLab for a platform giving teams the option to run services instead of adding to the monolith. * The KISS principle requires a conscious effort in this space. Complexity creeps in with every step. * Helm templates are horrible to debug. * Helm charts don't handle custom resource definitions well. * The multi-level kustomization scheme proposed by https://github.com/fluxcd/flux2-multi-tenancy is impossible to reason about, making it inoperable. **🔜 Next steps:** (see gitlab-com/gl-infra/platform/runway&7) * Identify an (internal) customer with which to build a pilot. * Implement the identified gaps to deploy the "example service" to GKE via Runway. * Identify and close gaps to unblock adoption by the pilot customer. * Time permitting, work on first GA features (progressive deployment, monitoring, automatic rollbacks, …) **:tanuki-colored-heart: Thanks** * @schin1 for the many, many code reviews and great questions * @pguinoiseau for his help in creating the shared Runway GKE clusters * @f_santos for sharing his insight, particularly on Helm releases, Flux, and their shortcomings * @andrewn for his guidance and encouragement * @swiskow for the product side support * @igorwwwwwwwwwwwwwwwwwwww and @gsgl for their input on the blueprint **:checkered_flag: Grand review:** please do the honors of closing this epic. ## Issue relationships <gldeps> ```mermaid graph TD issue315["✅ #315"] click issue315 "https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/315" "Bootstrap Flux in the Runway GKE clusters" issue315 --> issue316 issue314["✅ #314"] click issue314 "https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/314" "Emit resource manifests from the deployment repository." issue313 --> issue314 issue314 --> issue288 issue313["✅ #313"] click issue313 "https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/313" "Create a deployment Helm chart in the deployment project" issue290["✅ #290"] click issue290 "https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/290" "Configure a global load balancer for the example service." issue280 --> issue290 issue287["✅ #287"] click issue287 "https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/287" "Create a shared Runway GKE cluster using the `config-mgmt` repository." issue280 --> issue287 issue287 --> issue288 issue287 --> issue289 issue286["✅ #286"] click issue286 "https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/286" "Add the ability to generate Kubernetes configuration from a Runway manifest to `runwayctl`." issue283 --> issue286 issue286 --> issue288 issue284["✅ #284"] click issue284 "https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/284" "Update the deployment to instantiate the service's Helm chart." issue283 --> issue284 issue283["✅ #283"] click issue283 "https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/283" "Distill the Kubernetes configuration into a Helm chart." issue281 --> issue283 issue282["✅ #282"] click issue282 "https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/282" "Configure a global load balancer for the example service" issue281 --> issue282 issue281["✅ #281"] click issue281 "https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/281" "Deploy the example service to the prototype cluster" issue280 --> issue281 issue280["✅ #280"] click issue280 "https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/280" "Build a prototype GitLab→GKE deployment. Expect everything to be torn down again." issue218["✅ #218"] click issue218 "https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/218" "Runway Runtime Expansion" issue316["❌ #316"] click issue316 "https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/316" "Generate `OCIRepository` resources for each deployment project" issue288["❌ #288"] click issue288 "https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/288" "Deploy the example service to the shared GKE cluster, using the Runway manifest + Kubernetes config generation." issue289["❌ #289"] click issue289 "https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/289" "Deploy a metric exporter into the cluster. Ensure it can write metrics to Mimir." ``` </gldeps>