# Problem Statement Deployment of GitLab to OpenShift is problematic. OpenShift is growing as a product, and many of our customers are asking for support with it. Currently in our strategy we treat OpenShift as it is the same as Kubernetes. It's not, there are many differences primarily to do with security between Kubernetes and OpenShift. With OpenShift 4.x this gap widens even further. Red Hat is putting large efforts behind OpenShift and trying to expand it. Recently SUSE stopped supporting OpenStack. Red Hat has no plans to stop supporting OpenStack, However they have largely shuffled support from OpenStack to behind OpenShift. # Current State High Level Summary * [x] Gitlab deploy an app to OpenShift. * [ ] Gitlab can use kubernetes integration to manage a cluster. * [x] Gitlab runner can run in openshift. * [ ] [Gitlab server runs in openshift.](https://gitlab.com/groups/gitlab-org/-/epics/3444 "Support deploying GitLab on OpenShift - Beta (MVP)") # Technical Challenges 1. GitLab after switching from Omnibus Docker Containers has created numerous scalable containers. Those containers do not run on OpenShift natively due to two issues, running as a User ID, and running as root. Some of our partner containers, such as Postgres and Redis are beyond our control. However Red Hat provides and supports an OpenShift Compatible container that we can use. Without these changes, we require OpenShift security controls to be disabled. Which not only makes OpenShift more vulnerable; It makes GitLab CI an attack surface into OpenShift. **(This is really bad on OpenShift 3.x. Since 4.0x, OpenShift has made better strides to support this. GitLab as is can run with anyuid in 4.0. However the Runner should not run as anyuid, Because then a CI/CD Job can run as root.)** 2. GitLab's Helm Charts do not provision OpenShift's Persistent Storage, nor do they setup ingress routes or database storage. **(This applies to OpenShift 3.x, Not 4.0x - Since 4.0x OpenShift has support for Ingress Routes, Database Storage and More. )** 3. GitLab's Registry does not run on OpenShift. 4. GitLab when installed onto OpenShift does not support SSH Pushes. This is because the OpenShift's Router (HAProxy) cannot proxy the SSH Traffic to individual containers. This requires that GitLab's workhorse containers be installed on a specific node. Then the firewall of that node be configured to route port 22 traffic to GitLab. This is a setup process that cannot be done inside of Helm and must be done manually. In addition to this, the domain must point to the infrastructure node. So upstream load balancer changes may be required. **(This may no longer be a concern in OpenShift 4.0 - With the advent of native ingress support.)** 5. GitLab's Kubernetes AutoDevOps functionality tries to control the cluster through RBAC and Namespace creation. This requires access to the Kubernetes system. OpenShift actively prevents this as well as any changes to the `kubernetes` namespace. 6. GitLab's AutoDevOps deploys containers into a `gitlab-managed-apps` namespace. This means that if two projects on GitLab try to connect too OpenShift. They will collide and possibly break. 7. GitLab's Kubernetes Integration requires Helm - Which some OpenShift customers have prohibited or don't have access to install. 8. Some of GitLab's AutoDevOps Tools, such as Security Scanning and License Management require Docker in Docker. In OpenShift 4.x - Docker in Docker is not possible. These tools will not run. GitLab has made strides to remove Docker in Docker from many tools and in GitLab 13 few remain. 9. Many of GitLab's Components use S3 Backed Storage. Minio should be present and installed on OpenShift. # Target Audience 1. The larger OpenShift Community. 2. Many of our Professional Services and GitLab Premium/Ultimate Customers. 3. Our Red Hat Partners. # Proposal To address our OpenShift shortcomings, I think we need to have a discussion about our OpenShift support. Currently we offer guides and tools to deploy GitLab on OpenShift that open security risks at worse; at best create a bad user experience. If we're going to support OpenShift, I feel we need to support it as a first class customer and put efforts and resources behind it. If we cannot or should not support OpenShift, we should let the community know and not provide documentation with workarounds that contain security risks and a bad user experience. # Success Criteria Previous Success Criteria was ill-defined. Our new path forward to success is via using an OpenShift Operator to install, configure, and monitor GitLab in OpenShift. #### To install GitLab Runner on OpenShift - [x] GitLab Runner supported on OpenShift Via Operator - [x] GitLab Runner doesn't require anyuid. - [ ] GitLab Runner needs object storage. This could be a Minio Operator being deployed. But we need to deploy S3 capable storage with the Runner. This is a nice to have. _(Avoiding ReadWriteMany mounts or NFS)_ - [ ] _(If Minio)_ End to end test of GitLab Runner using Minio for S3 Storage. #### To install GitLab Application on OpenShift. We are currently planning to GA the GitLab Operator (different from the Runner Operator) in 14.3 and announce GA shortly after. This will allow users to install GitLab on OpenShift. - [x] GitLab Containers run UBI RHEL. - [x] GitLab Containers don't run as root. - [ ] GitLab Containers run without anyuid -or- Operator protections in place to overcome. - [ ] GitLab can be accessed via port 22 for SSH Push/Pull. - [ ] GitLab needs object storage. This could be a Minio Operator being deployed. But we need to deploy s3 capable storage with the Runner. This is a requirement. (Avoiding ReadWriteMany mounts or NFS) - [ ] _(If Minio)_ End to end test of GitLab Application using Minio for S3 Storage. #### CI/CD Workloads on OpenShift - [ ] OpenShift + GitLab Training Materials around Release Management and CI/CD Processes. - [ ] AutoDevOps does not require Docker in Docker - https://gitlab.com/gitlab-org/gitlab/-/issues/29039 - [x] Confirm whether we intend to support OpenShift for Kubernetes cluster management - [ ] Security Tools do not require Docker in Docker (SAST Works Now / More in 13.0) - [x] License Management tool do not require Docker in Docker - [ ] Assessment of AutoDevOps Capabilities - https://gitlab.com/gitlab-org/gitlab/-/issues/332560 #### Desirables. - [ ] We develop an OpenShift Operator to install GitLab and GitLab Runner on OpenShift - [x] (Become Red Hat Container Certified.)\[https://access.redhat.com/containers/#/registry.connect.redhat.com/gitlab/gitlab-runner\] https://connect.redhat.com/explore/red-hat-container-certification - [ ] We document and explain the difficulties of running Postgres/Redis on OpenShift. Explaining best practices for Databases on OpenShift. (or point to Red Hat Documentation) - [ ] We document and explain the difficulties of running [MinIO](https://min.io/) on OpenShift. (or point to Red Hat Documentation) # Related Issues - https://gitlab.com/gitlab-org/charts/gitlab/issues/639 - https://gitlab.com/gitlab-org/charts/gitlab/issues/895 - https://gitlab.com/gitlab-org/charts/gitlab/issues/1378 - https://gitlab.com/gitlab-org/charts/gitlab/issues/1274 - https://gitlab.com/gitlab-org/charts/gitlab/issues/1192 - https://gitlab.com/gitlab-org/charts/gitlab/issues/1390 - https://gitlab.com/gitlab-org/charts/gitlab/issues/1098 - https://gitlab.com/gitlab-org/charts/gitlab/issues/893 - https://gitlab.com/gitlab-org/charts/gitlab/issues/892 - https://gitlab.com/gitlab-org/gitlab-foss/issues/52494 - https://gitlab.com/gitlab-org/charts/gitlab/issues/1156 - https://gitlab.com/gitlab-org/charts/gitlab/issues/1448 - https://gitlab.com/gitlab-org/charts/gitlab/issues/1381 - https://gitlab.com/gitlab-org/charts/gitlab/issues/1246 - https://gitlab.com/gitlab-org/charts/gitlab/issues/1069 - https://gitlab.com/gitlab-org/charts/gitlab/issues/894 - https://gitlab.com/gitlab-org/charts/gitlab/issues/752 - https://gitlab.com/gitlab-org/ci-cd/docker-machine/-/issues/14 - https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26632 # Opportunities that need this support - https://gitlab.my.salesforce.com/0064M00000WuQcg - https://gitlab.my.salesforce.com/0066100000LuRZg - https://gitlab.my.salesforce.com/0016100001EoXAV - https://gitlab.my.salesforce.com/0066100000QcAyG - https://gitlab.my.salesforce.com/0064M00000XYSJb - https://gitlab.my.salesforce.com/0064M00000Ve3qJ - https://gitlab.my.salesforce.com/0064M00000Wv7Me - https://gitlab.my.salesforce.com/0064M00000XZH1t - https://gitlab.my.salesforce.com/0064M00000Xa4DD - https://gitlab.my.salesforce.com/0064M00000Xa2Ir - https://gitlab.my.salesforce.com/0064M00000XaDKV - https://gitlab.my.salesforce.com/0064M00000XaEv8 - https://gitlab.my.salesforce.com/0064M00000XaEvD - https://gitlab.my.salesforce.com/0064M00000XZyTz - https://gitlab.my.salesforce.com/0064M00000VMvrJ - https://gitlab.my.salesforce.com/0064M00000XYwWE - https://gitlab.my.salesforce.com/0064M00000XaEwB - https://gitlab.my.salesforce.com/0064M00000XaEwL - https://gitlab.my.salesforce.com/0064M00000XaD4n - https://gitlab.my.salesforce.com/0064M00000XaccT - https://gitlab.my.salesforce.com/0064M00000XaG8o - https://gitlab.my.salesforce.com/0064M00000XaG93 - https://gitlab.my.salesforce.com/0064M00000XaG9I - https://gitlab.my.salesforce.com/0064M00000XaG4F - https://gitlab.my.salesforce.com/0064M00000XaG4Q - https://gitlab.my.salesforce.com/00161000003NCNP - https://gitlab.my.salesforce.com/0014M00001lbj90 - https://gitlab.my.salesforce.com/0064M00000YMcq5QAD - https://gitlab.my.salesforce.com/0064M00000YMwsd - https://gitlab.my.salesforce.com/0064M00000YOqAu - https://gitlab.my.salesforce.com/0016100001amRm6 - https://gitlab.my.salesforce.com/0064M00000XZ2tQ - https://gitlab.my.salesforce.com/006PL000004N5Kv - https://gitlab.my.salesforce.com/0068X00001Hk91J - https://gitlab.my.salesforce.com/006PL000001zjno - https://gitlab.my.salesforce.com/006PL000002VsNy