[WIP] Follow GitLab Upstream on GKE Marketplace

- Check out GitLab submodule from upstream repository instead of special
  forked repository
- Create a wrapper chart for the Google GKE Marketplace named gitlab-mp
  allowing special configuration unique to the marketplace to be passed
  in without having to change the upstream chart
- Add env-doctor script allowing users to quickly verify that their
  environment is ready to produce a GKE Marketplace update
- Fortify list-helm-images, update-schema-yml, mirror-helm scripts to
  work when invoked from any path
- Remove a deprecated script 'build.sh'
- Move the Dockerfile definition to the top level directory and simplify
  to use the GKE Marketplace deployer_helm_tiller build container
- Move schema.yaml to schema.yaml.template and have the build script
  make a copy and run update-schema-yml against the copy instead of the
  original to prevent unwanted modifications from being committed to the
  repository
- Add build-marketplace-release.sh script to build the deployer
  container from the configured environment
- Added tooling to push builds to the GKE Marketplace and build and
  tear down a test environment
- Modify update-schema-yaml.sh to also insert RBAC information
- Updated documentation; moved original README to doc/installing and
  split documentation into user versus developer

Resolves: charts/gitlab#1041

Signed-off-by: Robert Marshall <rmarshall@gitlab.com>

.gitignore

@@ -2,3 +2,5 @@ build-scripts/rbac-exporter/bin/*
 build-scripts/rbac-exporter/.ruby-version
 build-scripts/rbac-exporter/.bundle/
 build-scripts/rbac-exporter/.rspec
+schema.yaml
+schema.yaml-e

.gitlab_gke_marketplace_build_env

+# vim: set filetype=bash:
+#!/usr/bin/env bash
+
+if [ "${GL_MP_DEBUG}" = "yes" ]; then
+    set -x
+fi
+
+# This is the release of GitLab to build
+GL_RELEASE_VERSION=${GL_RELEASE_VERSION:-"1.5.0"}
+
+# Define Paths to directories and binaries
+GL_MP_REPO="$(git rev-parse --show-toplevel)"
+GL_MP_SCRIPTS="${GL_MP_REPO}/build-scripts"
+GL_MP_SCRIPT_LIBS="${GL_MP_SCRIPTS}/lib"
+GL_MP_CHART="${GL_MP_REPO}/chart/gitlab-mp"
+GL_MP_SCRATCH="${GL_MP_REPO}/.scratch"
+GL_CHART_REPO="${GL_MP_SCRATCH}/gitlab"
+GL_MP_RBAC_EXPORTER="${GL_MP_SCRIPTS}/rbac-exporter/convert_chart_roles_to_schema.rb"
+GL_UPSTREAM="https://gitlab.com/charts/gitlab.git"
+
+# Define the Registry
+REGISTRY="gcr.io/top-chain-204115"
+APP_NAME="gitlab"
+GCR_REGISTRY="${REGISTRY}/${APP_NAME}"
+
+# Define Values Needed during Testing
+GL_MP_TEST_NAMESPACE="mkt-ns"
+GL_MP_TEST_APP_INSTANCE_NAME="gl-mp-test"
+GL_MP_TEST_DOMAIN="cloud-native.win"
+
+# function to display progress on stderror so it doesn't interfere with
+# scripts that rely on reading output from stdout
+display_progress() {
+    msg=$1
+
+    if [ -n "${msg}" ]; then
+        echo "PROGRESS: ${msg}" >&2
+    fi
+}
+
+display_failure() {
+    msg=$1
+    if [ -n "${msg}" ]; then
+        display_progress "${msg}"
+        exit 1
+    fi
+}

.gitmodules

 [submodule "gitlab"]
 	path = gitlab
-	url = https://gitlab.com/charts/gitlab-marketplace-app.git
+	url = https://gitlab.com/charts/gitlab.git
 [submodule "deployer/source/s3cmd"]
 	path = deployer/source/s3cmd
 	url = https://github.com/s3tools/s3cmd.git

Dockerfile

+FROM gcr.io/cloud-marketplace-tools/k8s/deployer_helm_tiller/onbuild:0.7.4

README.md

-# Deploying GitLab to GKE via Google Cloud Marketplace
+# GitLab Chart for Google GKE Marketplace
 
-[GitLab](https://about.gitlab.com) is a single application for the complete DevOps lifecycle from project planning and source code management to CI/CD and monitoring.
+## For Developers
 
-The [Google Cloud Marketplace](https://cloud.google.com/launcher/) is a easy way to deploy apps like GitLab to a [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/) cluster, with just a few clicks.
+Developers should consult
+[the instructions for building and testing](doc/building-deployer.md) for
+more information about how to use the included build tooling.
 
-> **Note:** For production deployments, we recommend using the [`gitlab` Helm chart](https://docs.gitlab.com/ee/install/kubernetes/gitlab_chart.html) and configuring [external PostgreSQL, Redis, and object storage services](https://gitlab.com/charts/gitlab/tree/master/doc/advanced).
+## For Users and Administrators
 
-# Installation
-
-## Quick install with Google Cloud Marketplace
-
-Deploy GitLab to Google Kubernetes Engine using Google Cloud Marketplace, by following the [on-screen instructions](https://console.cloud.google.com/marketplace/details/gitlab-public/gitlab).
-
-## Command line instructions
-
-### Prerequisites
-
-#### Set up command-line tools
-
-You'll need the following tools in your development environment:
-- [gcloud](https://cloud.google.com/sdk/gcloud/)
-- [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/)
-- [Helm](https://github.com/kubernetes/helm/blob/master/docs/install.md)
-- [docker](https://docs.docker.com/install/)
-- [git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
-
-Configure `gcloud` as a Docker credential helper:
-
-```shell
-gcloud auth configure-docker
-```
-
-#### Create a Google Kubernetes Engine cluster
-
-Create a new cluster from the command-line.
-
-```shell
-export CLUSTER=marketplace-cluster
-export ZONE=us-west1-a
-
-gcloud container clusters create "$CLUSTER" --zone "$ZONE"
-```
-
-Configure `kubectl` to talk to the new cluster.
-
-```shell
-gcloud container clusters get-credentials "$CLUSTER" --zone "$ZONE"
-```
-
-#### Clone this repo
-
-Clone this repo and the associated tools repo.
-
-```shell
-git clone --recurse-submodules https://gitlab.com/charts/deploy-image-helm-base.git
-gcloud source repos clone google-marketplace-k8s-app-tools --project=k8s-marketplace-eap
-```
-
-#### Install the Application resource definition
-
-Do a one-time setup for your cluster to understand Application resources.
-
-```shell
-kubectl apply -f google-marketplace-k8s-app-tools/crd/*
-```
-
-The Application resource is defined by the
-[Kubernetes SIG-apps](https://github.com/kubernetes/community/tree/master/sig-apps)
-community. The source code can be found on
-[github.com/kubernetes-sigs/application](https://github.com/kubernetes-sigs/application).
-
-#### Configure the app
-
-Open and edit `deploy-image-helm-base/gitlab/values.yaml` to customize the settings and desired container images. Additional information on the available settings is available in the `deploy-image-helm-base/gitlab/docs/` folder.
-
-#### Expand the manifest template and apply to Kubernetes
-
-```shell
-helm template gitlab --set APP_INSTANCE_NAME=$APP_INSTANCE_NAME,NAMESPACE=$NAMESPACE > expanded.yaml
-kubectl apply -f expanded.yaml
-```
-
-### Setting up DNS
-
-Retrieve the IP address GitLab is available at, note it may take a few minutes for the IP address to populate:
-
-```shell
-kubectl get \
---namespace <namespace> \
-ing <name>-unicorn \
--o jsonpath='{.status.loadBalancer.ingress[0].ip}'
-```
-
-Then configure a DNS record for the domain you provided during installation, resolving to the IP address you retrieved above.
-
-### Signing in
-
-Browse to https://`gitlab.<yourdomain>`.
-
-GitLab is provisioned with a randomly generated administrator password. To retrieve it: 
-
-```shell
-# specify the variables values matching your installation:
-export APP_INSTANCE_NAME=gitlab-1
-export NAMESPACE=default
-
-kubectl get secret -n $NAMESPACE $APP_INSTANCE_NAME-gitlab-initial-root-password -ojsonpath={.data.password} | base64 --decode
-```
-
-# Administration of GitLab
-
-GitLab offers a number of different options to customize the behavior to your needs. More information is available in our [administration documentation](https://docs.gitlab.com/ee/administration/index.html#configuring-gitlab).
-
-## Configuring a valid TLS certificate
-
-By default GitLab will utilize self-signed certificates. To utilize your own certificate:
-
-```shell
-# specify the variables values matching your installation:
-export APP_INSTANCE_NAME=gitlab-1
-export NAMESPACE=default
-
-kubectl create secret tls -n $NAMESPACE $APP_INSTANCE_NAME-wildcard-tls --cert=<path/to-full-chain.crt> --key=<path/to.key> --dry-run -o json | kubectl apply -f -
-```
-
-## Update GitLab
-
-GitLab is made up of multiple containers, each with their own images. These individual containers should be updated together, to ensure proper functionality. Database migrations also need to be run, to update the schema and any required data.
-
-Because of this, the best way to perform an upgrade is to clone this repo: 
-
-```shell
-# specify the variables values matching your installation:
-export APP_INSTANCE_NAME=gitlab-1
-export NAMESPACE=default
-
-git clone --recurse-submodules https://gitlab.com/charts/deploy-image-helm-base.git
-```
-
-Next, edit `gitlab/values.yaml` and be sure that any changes made during installation, or after, have been applied. Once complete, expand the Helm chart and apply it:
-
-```shell
-helm template gitlab --set APP_INSTANCE_NAME=$APP_INSTANCE_NAME,NAMESPACE=$NAMESPACE > expanded.yaml
-kubectl apply -f expanded.yaml
-```
-
-## Backup and Restore
-
-Detailed documentation on backup and restore is available [here](https://gitlab.com/charts/gitlab/tree/master/doc/backup-restore).
-
-## Scaling
-
-To make it easier to scale GitLab we include [horizontal pod autoscalers](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/), which add additional replicas as load increases.
-
-By default, these are limited to single replica. To view the current scaling status:
-
-```shell
-# specify the variables values matching your installation:
-export APP_INSTANCE_NAME=gitlab-1
-export NAMESPACE=default
-
-kubectl get hpa \
-  --namespace $NAMESPACE
-  --selector app.kubernetes.io/name=$APP_INSTANCE_NAME
-```
-
-You can then change the parameters by deleting the autoscaler and recreating it. For example, to change the minimum pods to 2 and maximum pods to 10 for `gitlab-unicorn`:
-
-```shell
-kubectl patch hpa -n $NAMESPACE $APP_INSTANCE_NAME-unicorn --patch '{"spec":{"maxReplicas":10}}'
-kubectl patch hpa -n $NAMESPACE $APP_INSTANCE_NAME-unicorn --patch '{"spec":{"minReplicas":2}}'
-```
-
-# Uninstall the Application
-
-## Using GKE UI
-
-Navigate to `GKE > Applications` in GCP console. From the list of applications, click on the one that you wish to uninstall.
-
-On the new screen, click on the `Delete` button located in the top menu. It will remove
-the resources attached to this application.
-
-## Using the command line
-
-### Prepare the environment
-
-Set your installation name and Kubernetes namespace:
-
-```shell
-export APP_INSTANCE_NAME=gitlab-1
-export NAMESPACE=default
-```
-
-### Delete the resources
-
-> **NOTE:** Please keep in mind that `kubectl` guarantees support for Kubernetes server in +/- 1 versions.
-> It means that for instance if you have `kubectl` in version 1.10.&ast; and Kubernetes 1.8.&ast;,
-> you may experience incompatibility issues, like not removing the StatefulSets with
-> apiVersion of apps/v1beta2.
-
-If you still have the expanded manifest file used for the installation, you can use it to delete the resources.
-Run `kubectl` on expanded manifest file matching your installation:
-
-```shell
-kubectl delete -f $APP_INSTANCE_NAME_manifest.yaml --namespace $NAMESPACE
-```
-
-Otherwise, delete the resources by indication of types and a label:
-
-```shell
-kubectl delete configmap,ingress,hpa,pdb,deployment,job,statefulset,secret,service \
-  --namespace $NAMESPACE \
-  --selector app.kubernetes.io/name=$APP_INSTANCE_NAME
-```
-
-### Delete the persistent volumes of your installation
-
-By design, removal of resources in Kubernetes does not remove the PersistentVolumeClaims that
-were attached to their Pods. It protects your installations from mistakenly deleting important data.
-
-If you wish to remove the PersistentVolumeClaims with their attached persistent disks, run the
-following `kubectl` commands:
-
-```shell
-# specify the variables values matching your installation:
-export APP_INSTANCE_NAME=gitlab-1
-export NAMESPACE=default
-
-kubectl delete persistentvolumeclaims \
-  --namespace $NAMESPACE
-  --selector app.kubernetes.io/name=$APP_INSTANCE_NAME
-```
+Consult the [documentation about installing GitLab from the
+Marketplace](doc/installing.md).
\ No newline at end of file

build-scripts/build-marketplace-release.sh

+#!/usr/bin/env bash
+# build-release.sh
+#
+# Builds a release for GKE Marketplace
+# expects no arguments
+
+# Import build environment variables and shared functions
+GL_MP_ENV="$(git rev-parse --show-toplevel)"
+GL_MP_ENV_CONFIG="${GL_MP_ENV}/.gitlab_gke_marketplace_build_env"
+
+[ ! -f "${GL_MP_ENV_CONFIG}" ] && echo "Missing ${GL_MP_ENV_CONFIG}" && exit 1
+
+. "${GL_MP_ENV_CONFIG}"
+
+# static variables
+SCHEMA_TEMPLATE="${GL_MP_ENV}/schema.yaml.template"
+SCHEMA_FILE="${GL_MP_ENV}/schema.yaml"
+
+display_progress "Copying template to operational schema file"
+if cp -a "${SCHEMA_TEMPLATE}" "${SCHEMA_FILE}"; then
+    display_progress "...completed"
+else
+    display_failure "...failed"
+fi
+
+display_progress "Updating image tags and RBAC configurations in
+schema.yaml"
+if "${GL_MP_SCRIPT_LIBS}"/update-schema-yml.sh; then
+    display_progress "...schema.yaml update complete"
+else
+    display_failure "...schema.yaml updating failed"
+fi
+
+display_progress "Updating GitLab Marketplace helm chart dependencies"
+if helm dependency build "${GL_MP_CHART}"; then
+    display_progress "...complete"
+else
+    display_failure "...failed"
+fi
+
+# We are not using $TAG and $REGISTRY in the same way as defined in the
+# Google GKE Marketplace documentation so we don't pass it here. If we
+# were to pass it, it would also have to be in the schema.yaml
+display_progress "Attempting to build GitLab Marketplace deployer container"
+if  docker build -t $REGISTRY/$APP_NAME/deployer "${GL_MP_ENV}"; then
+    display_progress "...deployer container build complete"
+else
+    display_failure "...deployer container build failed"
+fi

build-scripts/build.sh

-function major_version_number() {
-  echo "$1" | cut -d"." -f 1
-}
\ No newline at end of file

build-scripts/env-doctor.sh

+#!/usr/bin/env bash
+
+# Check the environment to be sure the user can actually build and use the
+# tooling included with this repository.
+
+display_error() {
+    echo "[PROBLEM] $1"
+    final_exit=1
+}
+
+final_exit=0
+
+echo "[START] Checking environment for tools necessary to build a GitLab GKE Marketplace release..."
+
+# git is required
+if ! command -v git > /dev/null 2>&1; then
+    display_error "A valid installation of git is required"
+    final_exit=1
+fi
+
+# environment configuration must exist
+GL_MP_ENV="$(git rev-parse --show-toplevel)"
+GL_MP_ENV_CONFIG="${GL_MP_ENV}/.gitlab_gke_marketplace_build_env"
+
+if [ ! -f "${GL_MP_ENV_CONFIG}" ]; then
+    display_error "Missing ${GL_MP_ENV_CONFIG}"
+fi
+
+if [ $final_exit = 0 ]; then
+    echo "[SUCCESS] Environment is ready to build GitLab GKE Marketplace"
+else
+    echo "[FAILURE] Resolve above issues to build GitLab GKE Marketplace"
+fi
+
+exit $final_exit

build-scripts/lib/checkout_upstream_chart.sh

+#!/usr/bin/env bash
+# Script to make sure the GitLab cloud native upstream repository is
+# available. If it is already checked out, try to check out the correct
+# tagged release version. It is smart enough to fetch from origin if the tag
+# doesn't exist in a pre-existing cloned repository.
+
+set -eo pipefail
+
+GL_MP_ENV="$(git rev-parse --show-toplevel)"
+GL_MP_ENV_CONFIG="${GL_MP_ENV}/.gitlab_gke_marketplace_build_env"
+
+[ ! -f "${GL_MP_ENV_CONFIG}" ] && echo "Missing ${GL_MP_ENV_CONFIG}" && exit 1
+
+# shellcheck source=/dev/null
+. "${GL_MP_ENV_CONFIG}"
+
+[ ! -d "${GL_MP_SCRATCH}" ] && mkdir -p "${GL_MP_SCRATCH}"
+
+needs_cloned="no"
+
+if [ -d "${GL_CHART_REPO}" ]; then
+    pushd "${GL_CHART_REPO}" > /dev/null
+    # no need to guard the next statement; if it fails origin_url is going
+    # to correct assume that the directory should be removed/replaced with
+    # an actual git repository
+    origin_url=$(git config --get remote.origin.url)
+    popd > /dev/null
+
+    if [ "${origin_url}" != "${GL_UPSTREAM}" ]; then
+        display_progress "Attempting to remove ${GL_CHART_REPO} [incorrect origin]"
+
+        # safe because we already know this directory exists
+        if rm -Rf "${GL_CHART_REPO}"; then
+            display_progress "...removal succeeded"
+        else
+            display_failure "...removal failed"
+        fi
+
+        needs_cloned="yes"
+    else
+        pushd "${GL_CHART_REPO}" > /dev/null
+
+        display_progress "Checking local GitLab Chart repository for v${GL_RELEASE_VERSION} tag"
+        if ! git tag |grep -q "^v${GL_RELEASE_VERSION}$"; then
+            display_progress ".. tag v${GL_RELEASE_VERSION} not present"
+            display_progress "...checking remote repository for updates"
+
+            if git fetch origin; then
+                display_progress "...local repository updated successfully"
+            else
+                display_failure "...failed to retrieve updates from ${GL_UPSTREAM}"
+            fi
+        else
+                display_progress "...found tag v${GL_RELEASE_VERSION} in local repository"
+        fi
+
+        display_progress "Attempting to checkout tag v${GL_RELEASE_VERSION}"
+        if git checkout "tags/v${GL_RELEASE_VERSION}" 2>/dev/null; then
+            display_progress "...checkout of tag v${GL_RELEASE_VERSION} succeeded"
+        else
+            display_failure "...checkout of tag v${GL_RELEASE_VERSION} failed"
+        fi
+
+        popd > /dev/null
+    fi
+else
+    needs_cloned="yes"
+fi
+
+if [ "${needs_cloned}" = "yes" ]; then
+    pushd "${GL_MP_SCRATCH}" > /dev/null
+
+    display_progress "Attempting to clone from ${GL_UPSTREAM}"
+    if git clone "${GL_UPSTREAM}"; then
+        display_progress "...cloning suceeded."
+        pushd "${GL_CHART_REPO}" > /dev/null
+
+        display_progress "Attempting to checkout tag v${GL_RELEASE_VERSION}"
+        if git checkout "tags/v${GL_RELEASE_VERSION}"; then
+            display_progress "...checkout of tag v${GL_RELEASE_VERSION} succeeded"
+        else
+            display_failure "...checkout of tag v${GL_RELEASE_VERSION} failed"
+        fi
+
+        popd > /dev/null
+    else
+        display_failure "...cloning failed."
+    fi
+
+    popd > /dev/null
+
+fi

build-scripts/lib/list-helm-images.sh

+#!/usr/bin/env bash
+
+set -eo pipefail
+
+GL_MP_ENV="$(git rev-parse --show-toplevel)"
+GL_MP_ENV_CONFIG="${GL_MP_ENV}/.gitlab_gke_marketplace_build_env"
+
+[ ! -f "${GL_MP_ENV_CONFIG}" ] && echo "Missing ${GL_MP_ENV_CONFIG}" && exit 1
+
+. "${GL_MP_ENV_CONFIG}"
+
+[ ! -d "${GL_MP_SCRATCH}" ] && mkdir -p "${GL_MP_SCRATCH}"
+
+"${GL_MP_SCRIPT_LIBS}/checkout_upstream_chart.sh"
+
+[ ! -d "${GL_CHART_REPO}" ] && display_progress "Cannot find ${GL_CHART_REPO}" && exit 1
+
+pushd "${GL_CHART_REPO}" > /dev/null
+
+display_progress "Configure helm for templating"
+if helm init --client-only > /dev/null; then
+    display_progress "...complete"
+else
+    display_failure "...failed"
+fi
+
+display_progress "Adding GitLab charts helm repository"
+if helm repo add gitlab https://charts.gitlab.io/ > /dev/null; then
+    display_progress "...complete"
+else
+    display_failure "...failed"
+fi
+
+display_progress "Updating helm repository"
+if helm repo update > /dev/null; then
+    display_progress "...complete"
+else
+    display_failure "...failed"
+fi
+
+display_progress "Updating helm chart dependencies"
+if helm dep update > /dev/null; then
+    display_progress "...complete"
+else
+    display_failure "...failed"
+fi
+
+display_progress "Generating list of container images"
+if helm template . --set certmanager-issuer.email=none@none.com | \
+        yq -r ". | select( .kind == \"Job\" or .kind == \"Deployment\" or .kind == \"StatefulSet\" or .kind == \"DaemonSet\" ) | .spec.template.spec | [.containers,.initContainers] | .[] | select(.!=null) | .[].image" | \
+   sort | uniq; then
+    display_progress "...complete"
+else
+    display_failure "...failed"
+fi
+
+popd > /dev/null

build-scripts/update-schema-yml.sh → build-scripts/lib/update-schema-yml.sh

@@ -5,6 +5,13 @@
 
 set -e
 
+GL_MP_ENV="$(git rev-parse --show-toplevel)"
+GL_MP_ENV_CONFIG="${GL_MP_ENV}/.gitlab_gke_marketplace_build_env"
+
+[ ! -f "${GL_MP_ENV_CONFIG}" ] && echo "Missing ${GL_MP_ENV_CONFIG}" && exit 1
+
+. "${GL_MP_ENV_CONFIG}"
+
 SCHEMA_FILE=${SCHEMA_FILE:-schema.yaml}
 
 function findImage() {
@@ -19,17 +26,35 @@ function replaceTagOnImage() {
 }
 
 if [ ! -f $SCHEMA_FILE ]; then
-  echo "Unable to find file: $SCHEMA_FILE"
+  display_failure "Unable to find file: $SCHEMA_FILE"
 fi
 
-for image in $(build-scripts/list-helm-images.sh); do
+for image in $("${GL_MP_SCRIPT_LIBS}"/list-helm-images.sh); do
     sourceImageName="$(echo "${image##*/}" | cut -d':' -f1 | cut -d'@' -f1)"
     sourceImageTag="${image//*:}"
-    echo -n "Image: ${sourceImageName}"
+    image_msg="Image: ${sourceImageName}"
     if ! findImage $sourceImageName ; then
-        echo ' !! NOT FOUND'
+        image_msg="${image_msg} !! NOT FOUND"
     else
-        echo ''
         replaceTagOnImage $sourceImageName $sourceImageTag
     fi
+    display_progress "${image_msg}"
 done
+
+pushd "${GL_MP_REPO}/chart/gitlab-mp" > /dev/null
+if helm template . --set certmanager-issuer.email=none@none.com > "${GL_MP_SCRATCH}/helm_template.yaml"; then
+    display_progress "...generated yaml from helm template invocation"
+else
+    display_failure "...failed to generate yaml from helm template invocation"
+fi
+popd > /dev/null
+
+"${GL_MP_RBAC_EXPORTER}" "${GL_MP_SCRATCH}/helm_template.yaml" > "${GL_MP_SCRATCH}/rbac_entries.yaml"
+sed -i -e 's/^/  /g' "${GL_MP_SCRATCH}/rbac_entries.yaml"
+
+line_number=$(grep -n '$RBAC_ENTRIES' "${SCHEMA_FILE}"|cut -d ':' -f 1)
+head -n $((line_number-1)) "${SCHEMA_FILE}" > "${GL_MP_SCRATCH}/schema.head"
+sed -n $((line_number+1))', $p' "${SCHEMA_FILE}" > "${GL_MP_SCRATCH}/schema.tail"
+cp "${GL_MP_SCRATCH}/schema.head" "${SCHEMA_FILE}"
+cat "${GL_MP_SCRATCH}/rbac_entries.yaml" >> "${SCHEMA_FILE}"
+cat "${GL_MP_SCRATCH}/schema.tail" >> "${SCHEMA_FILE}"

build-scripts/list-helm-images.sh

-#!/bin/bash
-
-set -exo pipefail
-
-cd gitlab
-
-helm init --client-only > /dev/null
-helm repo add gitlab https://charts.gitlab.io/ > /dev/null
-helm repo update > /dev/null
-helm dep update > /dev/null
-helm template . --set certmanager-issuer.email=none@none.com | \
-   yq -r ". | select( .kind == \"Job\" or .kind == \"Deployment\" or .kind == \"StatefulSet\" or .kind == \"DaemonSet\" ) | .spec.template.spec | [.containers,.initContainers] | .[] | select(.!=null) | .[].image" | \
-   sort | uniq
-
-cd ..
\ No newline at end of file

build-scripts/mirror-helm-images.sh → build-scripts/push-container-dependencies.sh

@@ -2,9 +2,14 @@
 
 set -e
 
-GCR_REGISTRY=${GCR_REGISTRY:-"gcr.io/top-chain-204115/gitlab"}
+GL_MP_ENV="$(git rev-parse --show-toplevel)"
+GL_MP_ENV_CONFIG="${GL_MP_ENV}/.gitlab_gke_marketplace_build_env"
 
-for image in $(build-scripts/list-helm-images.sh); do
+[ ! -f "${GL_MP_ENV_CONFIG}" ] && echo "Missing ${GL_MP_ENV_CONFIG}" && exit 1
+
+. "${GL_MP_ENV_CONFIG}"
+
+for image in $("${GL_MP_SCRIPT_LIBS}"/list-helm-images.sh); do
     SOURCE_IMAGE_NAME="$(echo "${image##*/}" | cut -d':' -f1 | cut -d'@' -f1)"
     SOURCE_IMAGE_TAG="${image//*:}"
     MIRRORED_IMAGE="${GCR_REGISTRY}/${SOURCE_IMAGE_NAME}:${SOURCE_IMAGE_TAG}"

build-scripts/push-marketplace-release.sh

+#!/bin/bash
+
+# Import build environment variables and shared functions
+GL_MP_ENV="$(git rev-parse --show-toplevel)"
+GL_MP_ENV_CONFIG="${GL_MP_ENV}/.gitlab_gke_marketplace_build_env"
+
+[ ! -f "${GL_MP_ENV_CONFIG}" ] && echo "Missing ${GL_MP_ENV_CONFIG}" && exit 1
+
+. "${GL_MP_ENV_CONFIG}"
+
+docker push $GCR_REGISTRY/deployer

build-scripts/tear-down-marketplace-release-test.sh

+#!/bin/bash
+
+# Import build environment variables and shared functions
+GL_MP_ENV="$(git rev-parse --show-toplevel)"
+GL_MP_ENV_CONFIG="${GL_MP_ENV}/.gitlab_gke_marketplace_build_env"
+
+[ ! -f "${GL_MP_ENV_CONFIG}" ] && echo "Missing ${GL_MP_ENV_CONFIG}" && exit 1
+
+. "${GL_MP_ENV_CONFIG}"
+
+function cleanup() {
+    kubectl -n "$GL_MP_TEST_NAMESPACE" get ingress,svc,pdb,hpa,deploy,statefulset,job,pod,secret,configmap,pvc,secret,clusterrole,clusterrolebinding,role,rolebinding,sa,applications 2>&1 \
+      | grep "$GL_MP_TEST_APP_INSTANCE_NAME" \
+      | awk '{print $1}' \
+      | xargs kubectl -n "$GL_MP_TEST_NAMESPACE" delete \
+      || true
+}
+
+display_progress "Starting test environment tear down"
+if  cleanup; then
+    display_progress "...completed successfully"
+else
+    display_failure "...tear down failed"
+fi

build-scripts/test-marketplace-release.sh

+#!/bin/bash
+
+# Import build environment variables and shared functions
+GL_MP_ENV="$(git rev-parse --show-toplevel)"
+GL_MP_ENV_CONFIG="${GL_MP_ENV}/.gitlab_gke_marketplace_build_env"
+
+[ ! -f "${GL_MP_ENV_CONFIG}" ] && echo "Missing ${GL_MP_ENV_CONFIG}" && exit 1
+
+. "${GL_MP_ENV_CONFIG}"
+
+# TODO: figure this out so we can use a standard set of variables if users don't want ot set their own...
+parameter="'{\"APP_INSTANCE_NAME\": \"$GL_MP_TEST_APP_INSTANCE_NAME\",\"NAMESPACE\": \"$GL_MP_TEST_NAMESPACE\", \"global.hosts.domain\": \"$GL_MP_TEST_DOMAIN\"}'"
+echo "${parameter}"
+
+#mpdev /scripts/install --deployer=$GCR_REGISTRY/deployer --parameters="$parameter"
+
+# TODO: remove this and use the one above which uses variables instead
+#mpdev /scripts/install --deployer=$GCR_REGISTRY/deployer --parameters='{"APP_INSTANCE_NAME": "kellin-mkt-test", "NAMESPACE": "mkt-ns", "global.hosts.domain": "cloud-native.win", "APPLICATION_UID": "$app_uid"}'
+mpdev /scripts/install --deployer=$GCR_REGISTRY/deployer --parameters='{"APP_INSTANCE_NAME": "gl-mp-test", "NAMESPACE": "mkt-ns", "global.hosts.domain": "cloud-native.win"}'

chart/gitlab-mp/Chart.yaml

+engine: gotpl
+name: gitlab-mp
+version: 1.0.0

chart/gitlab-mp/requirements.yaml

+dependencies:
+- name: gitlab
+  version: 1.5.0
+  repository: https://charts.gitlab.io/

chart/gitlab-mp/values.yaml

+# Default values for gitlab-chart on the Google GKE Marketplace
+# This is a YAML-formatted file.
+
+###################################################################################################
+# Global Values
+###################################################################################################
+global:
+  operator:
+    enabled: false
+  application:
+    create: true
+    links: []
+  hosts:
+    domain: example.com
+    https: true
+    externalIP:
+    ssh: ~
+  ingress:
+    configureCertmanager: false
+  initialRootPassword: {}
+  psql:
+    password: {}
+  redis:
+    password: {}
+  gitaly:
+    authToken: {}
+    internal:
+      names: ['default']
+    external: []
+  minio:
+    enabled: true
+    credentials: {}
+  appConfig:
+    enableUsagePing: true
+    defaultCanCreateGroup: true
+    usernameChangingEnabled: true
+    issueClosingPattern:
+    defaultTheme:
+    defaultProjectsFeatures:
+      issues: true
+      mergeRequests: true
+      wiki: true
+      snippets: true
+      builds: true
+    webhookTimeout:
+    gravatar:
+      plainUrl:
+      sslUrl:
+    extra:
+      googleAnalyticsId:
+      piwikUrl:
+      piwikSiteId:
+    lfs:
+      bucket: git-lfs
+      connection: {}
+    artifacts:
+      bucket: gitlab-artifacts
+      connection: {}
+    uploads:
+      bucket: gitlab-uploads
+      connection: {}
+    packages:
+      bucket: gitlab-packages
+      connection: {}
+    backups:
+      bucket: gitlab-backups
+      tmpBucket: tmp
+    incomingEmail:
+      enabled: false
+      address: ""
+      host: "imap.gmail.com"
+      port: 993
+      ssl: true
+      startTls: false
+      user: ""
+      password:
+        secret: ""
+        key: password
+      mailbox: inbox
+      idleTimeout: 60
+    ldap:
+      servers: {}
+    omniauth:
+      enabled: false
+      autoSignInWithProvider:
+      syncProfileFromProvider: []
+      syncProfileAttributes: ['email']
+      allowSingleSignOn: ['saml']
+      blockAutoCreatedUsers: true
+      autoLinkLdapUser: false
+      autoLinkSamlUser: false
+      externalProviders: []
+      providers: []
+    pseudonymizer:
+      configMap:
+      bucket: gitlab-pseudo
+      connection: {}
+  shell:
+    authToken: {}
+    hostKeys: {}
+  railsSecrets: {}
+  registry:
+    bucket: registry
+    certificate: {}
+    httpSecret: {}
+  runner:
+    registrationToken: {}
+  # Outgoing email server settings
+  smtp:
+    enabled: false
+    address: smtp.mailgun.org
+    port: 2525
+    user_name: ""
+    password:
+      secret: ""
+      key: password
+    # domain:
+    authentication: "plain"
+    starttls_auto: false
+    openssl_verify_mode: "peer"
+  # Email persona used in email sent by GitLab
+  email:
+    from: ''
+    display_name: GitLab
+    reply_to: ''
+    subject_suffix: ''
+  time_zone: UTC
+  service:
+    annotations: {}
+  antiAffinity: soft
+  workhorse: {}
+  # configuration of certificates container & custom CA injection
+  certificates:
+    image:
+      repository: registry.gitlab.com/gitlab-org/build/cng/alpine-certificates
+      tag: 20171114-r3
+    customCAs: []
+
+###################################################################################################
+# These must live outside the gitlab context because of issues with helm that prevent wrapper charts
+# from understanding the parent-child relationship of charts in requirements.yaml
+###################################################################################################
+certmanager:
+  install: false
+prometheus:
+  install: false
+gitlab-runner:
+  install: false
+
+###################################################################################################
+# Values to send down into the wrapped charts
+###################################################################################################
+gitlab:
+  nginx-ingress:
+    enabled: true
+    tcpExternalConfig: "true"
+    controller:
+      config:
+        hsts-include-subdomains: "false"
+        server-name-hash-bucket-size: "256"
+        enable-vts-status: "true"
+        use-http2: "false"
+        ssl-ciphers: "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4"
+        ssl-protocols: "TLSv1.1 TLSv1.2"
+        server-tokens: "false"
+      extraArgs:
+        force-namespace-isolation: ""
+      service:
+        externalTrafficPolicy: "Local"
+      resources:
+        requests:
+          cpu: 50m
+          memory: 100Mi
+      publishService:
+        enabled: true
+      replicaCount: 1
+      minAvailable: 0
+      scope:
+        enabled: true
+      stats:
+        enabled: true
+      metrics:
+        enabled: true
+        service:
+          annotations:
+            prometheus.io/scrape: "true"
+            prometheus.io/port: "10254"
+    defaultBackend:
+      replicaCount: 1
+      minAvailable: 0
+      resources:
+        requests:
+          cpu: 5m
+          memory: 5Mi
+    rbac:
+      create: true
+    serviceAccount:
+      create: true
+  redis-ha:
+    nameOverride: redis
+    enabled: false
+  postgresql:
+    install: true
+    postgresUser: gitlab
+    postgresDatabase: gitlabhq_production
+    imageTag: 9.6.8
+    usePasswordFile: true
+    existingSecret: 'secret'
+    metrics:
+      enabled: true
+  registry:
+    minReplicas: 1
+  shared-secrets:
+    enabled: true
+    rbac:
+      create: true
+  gitlab:
+    unicorn:
+      minReplicas: 1
+      resources:
+        limits:
+         memory: 1.5G
+        requests:
+          cpu: 100m
+          memory: 900M
+      workhorse:
+        resources:
+          limits:
+            memory: 100M
+          requests:
+            cpu: 10m
+            memory: 10M
+    sidekiq:
+      minReplicas: 1
+      resources:
+        limits:
+          memory: 1.5G
+        requests:
+          cpu: 100m
+          memory: 500Mi
+    gitlab-shell:
+      minReplicas: 1
+    task-runner:
+      enabled: false
+  minio:
+    resources:
+      requests:
+        memory: 64Mi
+        cpu: 10m
+  redis:
+    resources:
+      requests:
+        cpu: 10m
+        memory: 64Mi

deployer/Dockerfile

-FROM launcher.gcr.io/google/debian9 AS build
-
-ENV HELM_VERSION=2.8.2
-ENV HELM_URL=https://kubernetes-helm.storage.googleapis.com/helm-v${HELM_VERSION}-linux-amd64.tar.gz
-
-RUN apt-get update \
-    && apt-get install -y --no-install-recommends gettext wget
-    
-# Install Helm
-RUN wget -q -O - ${HELM_URL} | tar zxf - \
-    && mv linux-amd64/helm /usr/bin/ \
-    && chmod +x /usr/bin/helm \
-    && helm version --client
-
-# Add the chart and initialize Helm
-ADD gitlab /gitlab
-RUN cp -r /gitlab /tmp/chart \
-    && cd /tmp/chart \
-    && helm init --client-only \
-    && helm repo add gitlab https://charts.gitlab.io/ \
-    && helm repo update \
-    && helm dep update
-
-# GZip and Tar the chart
-RUN cd /tmp \
-    && tar -czvf /tmp/gitlab.tar.gz chart
-
-# Add our schema file
-ADD schema.yaml /tmp/schema.yaml
-
-# Provide registry prefix and tag for default values for images.
-ARG REGISTRY
-ARG TAG
-RUN cat /tmp/schema.yaml \
-    | env -i "REGISTRY=$REGISTRY" "TAG=$TAG" envsubst \
-    > /tmp/schema.yaml.new \
-    && mv /tmp/schema.yaml.new /tmp/schema.yaml
-
-
-FROM gcr.io/cloud-marketplace-tools/k8s/deployer_helm
-COPY --from=build /tmp/gitlab.tar.gz /data/chart/
-COPY --from=build /tmp/schema.yaml /data/
-
-# Add the open source license file
-ADD deployer/source /source