**Engineering DRI:** @tkuah Most discussion happened in https://gitlab.com/gitlab-org/configure/general/-/issues/97 ## Why are we deprecating GitLab Managed Apps? Today we have two versions of GitLab Managed Apps. Version 1, or one click install apps are the applications you can see on the GitLab UI under Operations / Clusters. V2 or CI/CD based GitLab Managed Apps are a cloud-native friendly approach to installing applications where applications are installed with GitLab CI pipelines. ### Our direction with respect to GitLab Managed Apps - We want to allow cluster operators to bring their own installations of ingress, prometheus, cilium, fluentd to integrate with GitLab. Moreover, we have a few principles that assure the managed app framework is not hijacked by 3rd party applications and low quality integrations. - Each GitLab Managed App should provide sane, production-ready default configurations. - Each GitLab Managed App should be a requirement for current or planned GitLab functionality or it should integrate deeply with GitLab. - Each GitLab Managed App should have a clear support path, possibly outside of GitLab. As a result of these directions, we are going to remove one click install GitLab Managed Apps, and also drop future support for instead recommending the use of the CI/CD based approach or we might just point them to our documentation and the installation instructions needed for various applications we integrate with. This means that we see much value in the features that are built inside GitLab on top of these apps. I'm sure that the respective product managers made the necessary problem validations, and together with their teams they came up with solutions that our customers want, and we want to support their efforts and integrations. [In 13.7 we deprecated](https://about.gitlab.com/releases/2020/12/22/gitlab-13-7-released/#deprecate-one-click-gitlab-managed-apps-in-gitlab-14.0) version 1 applications. They've served us well to learn what our users want, and the current solution is not flexible enough for the majority of production use cases, and some of the apps put much support burden on us too. ## GMA v2 (CI/CD) With respect to CI/CD based apps, we will deprecate it in %"13.12". GMA v2 brought the advent of managing Helm releases through pipeline via Helmfiles, which brought much more flexibility to the GMA solution as GMAv1 as users were able to specify their chart values, and also brought better log surfacing capabilities, as users would see the result of Helm commands in a `tiller.log` artifact and also the full job execution output. Although, this solution still limits the users in someways. For instance: - The Helmfile templates are kept inside the a docker image maintained by GitLab which was used as the base image of the [`Managed-Cluster-Applications.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/c65eb7dc3a0e6266b428572cef1a20456f181f2c/lib/gitlab/ci/templates/Managed-Cluster-Applications.gitlab-ci.yml#L3). This means that, the users didn't have direct customization access to those templates. So, in many cases, this limitation was a problem to the users and would require the templates to eventually need new contributions to comply with different user's need. A good example of it is the [lack of DNS01 challenge support](https://gitlab.com/gitlab-org/gitlab/-/issues/27148) in the CertManager template. Some users need this to be able to generate wildcard DNS certificates. The fact is that a new contribution would always be needed in the future to keep these templates compliant with upcoming versions of those charts. **The best here is if users had full control of their own Helmfiles, not only their chart values.** - The upgrade workflow was unpredictable. GitLab was responsible for bumping the charts version whenever new releases would come out. This is risky, since the users could be running their pipelines to update one app's configuration while at the same time they could get another app's upgrade without expecting it to happen. **The best here is for the users to have full control of when they want to upgrade their apps**. - Upgrades were not guaranteed. For upgrades to happen, the user would need to manually run a pipeline. So even after GitLab upgrades those upstream chart dependencies, the users would still need to manually check the repository to find out that new versions are available and the manually trigger a new pipeline. So there was no "upgrade available" notification feature. - There was no trackbility of a user's version. So, if GitLab bumps a version of chart, depending on the user's version at the time they run the upgrade, it could be a breaking change. **So for this bullet and the above, the best person to acknowledge which version to upgrade to and when to upgrade to, is actually the user.** ## How to deprecate it by providing value? - The v1 deprecation was announced in the release notes of 13.7. - We want to provide an API endpoint where users can register their own installations of Nginx, Cilium, etc - We're planing on providing a Project Template in %"14.0". See below Why a Project Template. To address the problems we had with v2, we've come to the conclusion that, regarding managing applications, the best scenario is to allow the users to have full flexibility to chart customization and also full control of application upgrades. We'll still build features that integrate with those other applications that our users love, but GitLab shouldn't limit how the users install/configure them. Therefore, a Project Template would allow users to have a quick start installing and integrating those applications with GitLab. This template would have all the Helmfiles and configuration values directly in the users repository, so 100% flexibility. And with that, the users would then decide by themselves and have full control of when and how to upgrade their own apps. Like explained above, this is something that GitLab was providing to the users, but it also had it's downsides. ## Current state and expected state after the changes ### Current applications and dependencies in GitLab * Clusters::Applications::Helm - Nothing, deprecated for tillerless helm * Clusters::Applications::Ingress - Only required for `KUBE_INGRESS_BASE_DOMAIN` * Clusters::Applications::CertManager - Nothing * Clusters::Applications::Crossplane - Nothing * Clusters::Applications::Prometheus - Making PromQL queries for Metrics and Alerts, see also https://docs.gitlab.com/ee/user/project/integrations/prometheus.html#manual-configuration-of-prometheus * Clusters::Applications::Runner - Nothing * Clusters::Applications::Jupyter - Nothing * Clusters::Applications::Knative - Serverless pages domain, deprecated https://gitlab.com/gitlab-org/gitlab-pages/-/issues/467 * Clusters::Applications::ElasticStack - Making elastic queries for Logs * Clusters::Applications::Fluentd - Nothing, though it looks like it dynamic sets up log forwarding for WAF and Cilium logs. * Clusters::Applications::Cilium - GMA v2 only, https://docs.gitlab.com/ee/user/project/clusters/protect/container_network_security/#supported-networkpolicies ### Proposal https://gitlab.com/gitlab-org/gitlab/-/issues/292460 - %13.11 * Clusters::Applications::Prometheus - Have an integration setting for the cluster, tells GitLab that Prometheus is installed on the cluster. ---- https://gitlab.com/gitlab-org/gitlab/-/issues/327803 - %13.12 * Write docs on how to take over any previously installed GMA v1 applications ---- https://gitlab.com/gitlab-org/gitlab/-/issues/327655 - %"13.12" Migrate legacy Prometheus to new ---- https://gitlab.com/gitlab-org/gitlab/-/issues/326560 - %13.12 * Clusters::Applications::ElasticStack - Have an integration setting for the cluster, tells GitLab that ElasticStack is installed on the cluster. ---- https://gitlab.com/gitlab-org/gitlab/-/issues/321509 - %"14.0" * Clusters::Applications::Ingress - [Document the recommend way to install Nginx-ingress ](https://gitlab.com/gitlab-org/gitlab/-/issues/321509) ---- https://gitlab.com/gitlab-org/gitlab/-/issues/292461 - %"14.0" * Clusters::Applications::Runner - Document the recommended way to install the Runner ---- https://gitlab.com/gitlab-org/gitlab/-/issues/293583 - %14.0 * Remove GMAv1 interface * Clusters::Applications::CertManager - Deprecate and remove in 14.0 * Clusters::Applications::Crossplane - Deprecate and remove in 14.0 * Clusters::Applications::Jupyter - Deprecate and remove in 14.0 * Clusters::Applications::Knative - Deprecate and remove in 14.0 https://gitlab.com/gitlab-org/gitlab/-/issues/326565 - unscheduled * REST API to enable / disable cluster integration ---- (no issue yet) * Clusters::Applications::Cilium - Have an integration setting for the cluster, tells GitLab that ElasticStack is installed on the cluster. (related: https://docs.gitlab.com/ee/user/project/clusters/protect/container_network_security/#supported-networkpolicies) ---- https://gitlab.com/gitlab-org/gitlab/-/issues/327908 * Move all responsibilities of cluster applications into Project Template * ~~CI/CD GMA automatically enable / disable integration for Prometheus, ElasticStack, and Cilium~~ * ~~Consolidate CI/CD based GitLab-managed apps~~ * Consider deprecating, and removing these apps that is less commonly installed * Clusters::Applications::Crossplane * Clusters::Applications::Jupyter * Clusters::Applications::Knative * Posthog ---- https://gitlab.com/gitlab-org/gitlab/-/issues/329944 * Clusters::Applications::Fluentd - Enable CI/CD fluentd to setup log forwarding for cilium (https://gitlab.com/gitlab-org/gitlab/-/blob/4ca7538c9bfd1b9ff811eb09ae18cf2d72f4636f/app/models/clusters/applications/fluentd.rb#L113-118)