Workspaces setup done in runtime in CI

What does this MR do and why?

This MR is the implementation of the Support for Workspaces to be setup in the runtime in the CI and also support using existing cluster - Refer #397005 (closed) and #414915 (closed)

Existing Test

The existing E2E test is moved under qa/qa/specs/features/ee/browser_ui/3_create/remote_development/without_setup/workspace_actions_without_setup_spec.rb

This test is in Quarantine and it is still used by the developers for local testing. This test may be removed in the future when its not needed. It test can be run using the - scripts/remote_development/run-e2e-tests.sh

Pre-requisites for this E2E test

1. It requires two variables AGENT_NAME and DEVFILE_PROJECT to be declared

This E2E tests will do the following:

1. It calls the shared_examples "workspaces actions"
2. Uses the AGENT_NAME and DEVFILE_PROJECT variable values to create a new workspace
3. After the new workspace is successfully running, it stops and then terminates it

This test can be run by

DEVFILE_PROJECT="devfile-test-project" AGENT_NAME="test-agent" bundle exec bin/qa Test::Instance::All "$TEST_INSTANCE_URL" -- --tag quarantine qa/specs/features/ee/browser_ui/3_create/remote_development/without_setup

New Test

The new E2E test is under qa/qa/specs/features/ee/browser_ui/3_create/remote_development/workspace_actions_spec.rb

This is the implementation to install and then create workspace from scratch

Pre-requisites for this E2E test

1. This E2E test runs only on staging as of now, because it needs kubernetes agent and KAS to be running already (In staging this is already enabled)
2. For gitlab-workspaces-proxy, it needs a Oauth application in the environment in which the test will run, so this oauth application is already created in staging, and the client id and secret are stored as CI Variables
3. The workspaces will be launched in a new domain, *.workspaces.staging.gitlabqa.dev, so this new domain is registered and the corresponding cloud dns setup is done already (in GCP project group-qa-tests )
4. The above domain needs SSL certificates, which created through certbot and is passed as WORKSPACES_DOMAIN_CERT, WORKSPACES_DOMAIN_KEY, WORKSPACES_WILDCARD_CERT and WORKSPACES_WILDCARD_KEY. It has validity of 3 months after which it needs to be renewed
5. This test will use the WORKSPACES_CLUSTER_AVAILABLE CI variable to determine if the cluster needs to be created or existing cluster needs to be used ( #414915 (closed))

This E2E tests will do the following:

If WORKSPACES_CLUSTER_AVAILABLE=false (default case)

1. It creates new kubernetes cluster in the GKE - in the test project "group-qa-tests"
2. Installs helm, ingress-controller in the k8s cluster
3. Installs Gitlab-Workspaces-proxy with the Oauth client id, secret, redirect uri and signing key. It also uses the certificates passed
4. Gets the load balancer IP of the Ingress and updates the cloud dns domain with the IP
5. Creates test group with two projects - agent project and devfile project (this contains the devfile for workspace initialisation)
6. In the agent project adds the workspaces config yml file and creates a agent_token for installing the agent in the cluster
7. Setup the above agent with the token in the cluster
8. In the devfile project adds the devfile config yml file
9. It calls the shared_examples "workspaces actions"
10. Uses the agent name and devfile project name from the above setup to create a new workspace
11. After the new workspace is successfully running, it stops and then terminates it

Finally, Once the test is complete the cluster will be deleted, the agent and agent_token will be removed.

If WORKSPACES_CLUSTER_AVAILABLE=true

1. It uses existing kubernetes cluster with CI variables WORKSPACES_CLUSTER_NAME and WORKSPACES_CLUSTER_REGION
2. Helm, ingress-controller, Gitlab-Workspaces-proxy will be installed already in this cluster and load balancer IP will be updated in the cloud dns domain

The next steps are same as above:

1. Creates test group with two projects - agent project and devfile project (this contains the devfile for workspace initialisation)
2. In the agent project add the workspaces config yml and create a agent_token for installing the agent in the cluster
3. Setup the above agent with the token in the cluster
4. In the devfile project add the devfile config yml
5. It calls the shared_examples "workspaces actions"
6. Uses the agent name and devfile project name from the above setup to create a new workspace
7. After the new workspace is successfully running, it stops and then terminates it

Finally, Once the test is complete the cluster will remain and only the agent and agent_token will be removed.

New CI variables introduced in this change:

• WORKSPACES_PROXY_DOMAIN
• WORKSPACES_OAUTH_APP_ID
• WORKSPACES_OAUTH_APP_SECRET
• WORKSPACES_OAUTH_SIGNING_KEY
• WORKSPACES_DOMAIN_CERT
• WORKSPACES_DOMAIN_KEY
• WORKSPACES_WILDCARD_CERT
• WORKSPACES_WILDCARD_KEY

The MR to add these into terraform - https://gitlab.com/gitlab-org/quality/engineering-productivity-infrastructure/-/merge_requests/436

The below variables are not provisioned right now, and can be done once the cluster for the tests are in place

• WORKSPACES_CLUSTER_AVAILABLE
• WORKSPACES_CLUSTER_NAME
• WORKSPACES_CLUSTER_REGION

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Output

nivethaprabakaran@Nivetha qa %  bundle exec bin/qa Test::Instance::All https://staging.gitlab.com -- -- qa/specs/features/ee/browser_ui/3_create/remote_development/workspace_actions_spec.rb
2023-08-01 18:14:56 WARN Selenium [:logger_info] Details on how to use and modify Selenium logger:
  https://selenium.dev/documentation/webdriver/troubleshooting/logging#ruby

2023-08-01 18:14:56 WARN Selenium applicable driver not found; attempting to install with Selenium Manager
2023-08-01 18:14:58 / CONF	:: 
      ==> Base URL: https://staging.gitlab.com
      ==> Browser: #<Selenium::WebDriver::Chrome::Driver:0x000000010cd35768>
      ==> Libraries: Chemlab::Vendor
[Aug 01 2023 18:14:58 IST (QA Tests)] INFO  -- Caching token for username: gitlab-qa-bot, last six chars of token:KSTdjm
[Aug 01 2023 18:14:58 IST (QA Tests)] INFO  -- Caching token for username: gitlab-qa, last six chars of token:abiBtU
[Aug 01 2023 18:14:58 IST (QA Tests)] INFO  -- Browser: chrome
[Aug 01 2023 18:14:58 IST (QA Tests)] INFO  -- Performing sanity check for environment!
Run options: exclude {:orchestrated=>true, :transient=>true, :sanity_feature_flags=>true, :geo=>true, :skip_live_env=>true}

Randomized with seed 2590

Create
  Remote Development
    when setup is done in runtime
      behaves like workspaces actions
[Aug 01 2023 18:15:01 IST (QA Tests)] INFO  -- Starting test: Create Remote Development when setup is done in runtime behaves like workspaces actions creates a new workspace and then stops and terminates it
checking for kubectl... yes
checking for gcloud... yes
[Aug 01 2023 18:15:01 IST (QA Tests)] INFO  -- Executing: `gcloud auth activate-service-account automated-qa-tests@group-qa-tests-566cc6.iam.gserviceaccount.com --key-file /var/folders/96/qv364xss6h75crxtybswsp1c0000gn/T/gcloud-account-key20230801-68626-djhnv2`
.
[Aug 01 2023 18:15:02 IST (QA Tests)] INFO  -- Executing: `gcloud container clusters create qa-cluster-20230801124502-90451dab  --region europe-west4 --disk-size 15GB --num-nodes 1 && gcloud container clusters get-credentials --region europe-west4 qa-cluster-20230801124502-90451dab `
............
[Aug 01 2023 18:21:25 IST (QA Tests)] INFO  -- Executing: `curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 && chmod 700 get_helm.sh && DESIRED_VERSION=v3.7.0 ./get_helm.sh `
.
[Aug 01 2023 18:21:26 IST (QA Tests)] INFO  -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-3' via api in 0.6 seconds
[Aug 01 2023 18:21:28 IST (QA Tests)] INFO  -- ==> Built a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-3/parent-group-to-test-remote-development-cbcce41978320fe4' via api in 2.54 seconds
[Aug 01 2023 18:21:30 IST (QA Tests)] INFO  -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-3/parent-group-to-test-remote-development-cbcce41978320fe4/agent-project-53d9fc1c6695ddbb' via api in 1.77 seconds
[Aug 01 2023 18:21:31 IST (QA Tests)] INFO  -- ==> Built a QA::Resource::Clusters::Agent with name 'remotedev-5696df4e' via api in 0.49 seconds
[Aug 01 2023 18:21:31 IST (QA Tests)] INFO  -- ==> Built a QA::Resource::Clusters::AgentToken with id '1024155' via api in 0.56 seconds
[Aug 01 2023 18:21:33 IST (QA Tests)] INFO  -- ==> Built a QA::Resource::Repository::Commit via api in 1.73 seconds
[Aug 01 2023 18:21:35 IST (QA Tests)] INFO  -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-3/parent-group-to-test-remote-development-cbcce41978320fe4/devfile-project-e7c3718718279fb6' via api in 2.2 seconds
[Aug 01 2023 18:21:38 IST (QA Tests)] INFO  -- ==> Built a QA::Resource::Repository::Commit via api in 2.26 seconds
[Aug 01 2023 18:21:38 IST (QA Tests)] INFO  -- Executing: `helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx && helm repo update && helm install   ingress-nginx ingress-nginx/ingress-nginx   --namespace ingress-nginx   --create-namespace   --version 4.3.0 `
.....................................................
[Aug 01 2023 18:22:26 IST (QA Tests)] INFO  -- Executing: `helm repo add gitlab-workspaces-proxy   https://gitlab.com/api/v4/projects/gitlab-org%2fremote-development%2fgitlab-workspaces-proxy/packages/helm/devel && helm repo update && helm upgrade --install gitlab-workspaces-proxy   gitlab-workspaces-proxy/gitlab-workspaces-proxy   --version 0.1.6   --namespace=gitlab-workspaces   --create-namespace   --set="auth.client_id=xxxxx"   --set="auth.client_secret=xxxxx"   --set="auth.host=https://staging.gitlab.com"   --set="auth.redirect_uri=https://workspaces.staging.gitlabqa.dev/auth/callback"   --set="auth.signing_key=123456789"   --set="ingress.host.workspaceDomain=workspaces.staging.gitlabqa.dev"   --set="ingress.host.wildcardDomain=*.workspaces.staging.gitlabqa.dev"   --set="ingress.tls.workspaceDomainCert=xxxx"   --set="ingress.tls.workspaceDomainKey=xxxx"   --set="ingress.tls.wildcardDomainCert=xxxx"   --set="ingress.tls.wildcardDomainKey=xxxx"   --set="ingress.className=nginx" `
...............
[Aug 01 2023 18:22:40 IST (QA Tests)] INFO  -- Executing: `helm repo add gitlab https://charts.gitlab.io && helm repo update && helm upgrade --install gitlab-agent gitlab/gitlab-agent   --namespace "remotedev-5696df4e"   --create-namespace   --set image.tag=v16.0.0   --set config.token=xxxxxx   --set config.kasAddress=wss://kas.staging.gitlab.com   --set config.kasHeaders="{Cookie: gitlab_canary=false}" `
........................
[Aug 01 2023 18:22:50 IST (QA Tests)] INFO  -- Executing: `kubectl -n ingress-nginx get svc ingress-nginx-controller           -o jsonpath='{.status.loadBalancer.ingress[0].ip}'`
[Aug 01 2023 18:22:53 IST (QA Tests)] INFO  -- Executing: `gcloud dns record-sets update workspaces.staging.gitlabqa.dev --rrdatas= --ttl=300 --type=A --zone=gitlabqa-dev `
..
[Aug 01 2023 18:22:54 IST (QA Tests)] INFO  -- Executing: `gcloud dns record-sets update "*.workspaces.staging.gitlabqa.dev" --rrdatas= --ttl=300 --type=A --zone=gitlabqa-dev `
..
[Aug 01 2023 18:22:55 IST (QA Tests)] INFO  -- filling :login_field with "gitlab-qa"
[Aug 01 2023 18:22:55 IST (QA Tests)] INFO  -- clicking the coordinates of :login_field
[Aug 01 2023 18:23:01 IST (QA Tests)] INFO  -- filling :password_field with "*****"
[Aug 01 2023 18:23:01 IST (QA Tests)] INFO  -- clicking the coordinates of :password_field
[Aug 01 2023 18:23:06 IST (QA Tests)] WARN  -- Potentially Slow Code 'wait_for_requests ' took 0.578s
[Aug 01 2023 18:23:06 IST (QA Tests)] WARN  -- Potentially Slow Code 'find_element password_field' took 0.594s
[Aug 01 2023 18:23:11 IST (QA Tests)] INFO  -- clicking :sign_in_button
[Aug 01 2023 18:23:15 IST (QA Tests)] WARN  -- Potentially Slow Code 'click_element sign_in_button' took 3.858s
[Aug 01 2023 18:23:18 IST (QA Tests)] INFO  -- clicking :nav_item_link
[Aug 01 2023 18:23:19 IST (QA Tests)] WARN  -- Potentially Slow Code 'click_element nav_item_link' took 1.216s
[Aug 01 2023 18:23:21 IST (QA Tests)] WARN  -- Potentially Slow Code 'wait_for_requests ' took 1.851s
[Aug 01 2023 18:23:21 IST (QA Tests)] WARN  -- Potentially Slow Code 'all_elements workspace_list_item' took 1.858s
[Aug 01 2023 18:23:21 IST (QA Tests)] INFO  -- clicking :list_new_workspace_button
[Aug 01 2023 18:23:21 IST (QA Tests)] INFO  -- clicking :workspace_devfile_project_id_field
[Aug 01 2023 18:23:23 IST (QA Tests)] WARN  -- Potentially Slow Code 'wait_for_requests ' took 1.425s
[Aug 01 2023 18:23:23 IST (QA Tests)] WARN  -- Potentially Slow Code 'click_element workspace_devfile_project_id_field' took 1.479s
[Aug 01 2023 18:23:23 IST (QA Tests)] INFO  -- Searching in dropdown: "devfile-project-e7c3718718279fb6"
[Aug 01 2023 18:23:27 IST (QA Tests)] WARN  -- Potentially Slow Code 'wait_for_requests ' took 1.998s
[Aug 01 2023 18:23:27 IST (QA Tests)] WARN  -- Potentially Slow Code 'find_element workspace_cluster_agent_id_field' took 2.006s
[Aug 01 2023 18:23:27 IST (QA Tests)] INFO  -- clicking :save_workspace_button
[Aug 01 2023 18:25:03 IST (QA Tests)] INFO  -- clicking :workspace_stop_button
[Aug 01 2023 18:25:23 IST (QA Tests)] INFO  -- clicking :workspace_terminate_button
[Aug 01 2023 18:25:43 IST (QA Tests)] INFO  -- Executing: `gcloud container clusters delete   --region europe-west4   qa-cluster-20230801124502-90451dab   --quiet --async `
[Aug 01 2023 18:25:48 IST (QA Tests)] INFO  -- Removed a QA::Resource::Clusters::AgentToken with id '1024155'
[Aug 01 2023 18:25:48 IST (QA Tests)] INFO  -- Removed a QA::Resource::Clusters::Agent with name 'remotedev-5696df4e'
[Aug 01 2023 18:25:51 IST (QA Tests)] INFO  -- Removed a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-3/parent-group-to-test-remote-development-cbcce41978320fe4'
        creates a new workspace and then stops and terminates it

Top 1 slowest examples (650.06 seconds, 100.0% of total time):
  Create Remote Development when setup is done in runtime behaves like workspaces actions creates a new workspace and then stops and terminates it
    650.06 seconds ./qa/specs/features/shared_examples/create_and_terminate_workspace_shared_examples.rb:5

Finished in 10 minutes 50 seconds (files took 5.68 seconds to load)
1 example, 0 failures

Randomized with seed 2590

How to set up and validate locally

Because there's new data-testids being introduced here and the test can only be run against Staging, a breakpoint will need to be introduced right after the new workspace listed and to manually add the data-testid required to locate these elements.

The full list of variable and their values to run this spec can be found in 1password "Run workspaces tests against staging"

bundle exec bin/qa Test::Instance::All https://staging.gitlab.com -- -- qa/specs/features/ee/browser_ui/3_create/remote_development/workspace_actions_spec.rb

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

qa/qa/service/cluster_provider/gcloud.rb

@@ -58,6 +58,13 @@ def install_kubernetes_agent(agent_token:, kas_address:, agent_name: "gitlab-age
           CMD
         end
 
+        def uninstall_kubernetes_agent(agent_name: "gitlab-agent")
+          shell <<~CMD.tr("\n", ' ')
+            helm uninstall gitlab-agent \
+              --namespace "#{agent_name}"
+          CMD
+        end
+
         def install_ngnix_ingress
           shell <<~CMD.tr("\n", ' ')
             helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx &&