Create automated jobs for continual testing of offline environments
Background
We need to create a system for testing offline network robustness on an ongoing basis. This issue covers:
- running the gitlab-qa tests as a general product regression suite, and
- running the Secure test projects to validate that the Secure analyzers across languages all work.
Overall Plan
Stage 0: Recording of progress
Quality will provide a video of the GitLab-QA smoke tests and security_reports_spec running locally with wifi turned off to simulate an offline environment. This will be recorded since it can't be done live, and will represent a progress check rather than an official demo.
Step 1: Set up QA tests and prove airgap
- Explain the nightly image with offline environment variable added
- Show changes:
git difftool HEAD~3 qa/qa/runtime/env.rb
-
QA_OFFLINE_ENVIRONMENT
environment variable has been added.
-
- Show changes:
git difftool HEAD~3 qa/qa/service/docker_run/base.rb
-
pull
method uses the offline environment variable to pull from a local registry when needed.
-
- Show the image built from this:
docker image ls | grep gitlab.gitlab-ee-qa | grep nightly
- Show changes:
- Explain the gitlab-qa gem with offline environment variable added
- Show changes:
git difftool HEAD~1 lib/gitlab/qa/runtime/env.rb
-
QA_OFFLINE_ENVIRONMENT
environment variable has been added.
-
- Show changes:
git difftool HEAD~1 lib/gitlab/qa/docker/engine.rb
-
pull
method uses the offline environment variable to pull from a local registry when needed.
-
- Show the gem built from this:
gem list --local gitlab-qa
(5.4.0)
- Show changes:
- Turn off ingress/egress via laptop wifi
- Show the network and no egress
- Show no access to online registry:
nc -vvvv -w 5 registry.gitlab.com 443 (and 80, 22, 5000)
- Show no route to outside world:
ping 8.8.8.8
ping gitlab.com
- Show no access to online registry:
Step 2: Run the full QA regression test suite on an offline environment
- Run an unsuccessful test
- Security reports spec:
gitlab-qa Test::Instance::Image gitlab/gitlab-ee-qa:nightly qa/specs/features/ee/browser_ui/secure/security_reports_spec.rb
- Show the test tried to pull the runner from docker.io via logs with the result: Service Unavailable
- Security reports spec:
- Show the local registry is running
docker container ls
- Set offline environment variable
export QA_OFFLINE_ENVIRONMENT=true
- Run successful tests
- Security reports spec:
gitlab-qa Test::Instance::Image gitlab/gitlab-ee-qa:nightly qa/specs/features/ee/browser_ui/secure/security_reports_spec.rb
- Smoke tests (13 in total):
gitlab-qa Test::Instance::Image gitlab/gitlab-ee-qa:nightly -- --tag smoke
- Security reports spec:
Stage 1: Manual
GitLab QA
Quality will conduct a live demo or demos to show the GitLab-QA tests running locally with offline containers.
Step 0: Kick off pipeline to allow it to run in the background and save time
Step 1: Set up QA tests and prove airgap
- Explain the gitlab-qa gem with iptables changes and offline network
- Show changes:
git difftool HEAD~3 lib/gitlab/qa/scenario/test/instance/airgapped.rb
- Show changes:
- Explain the nightly .gitlab-ci.yml changes to add Airgapped instance
- Temporarily we are building the images from branches, only until the changes are added to master for those projects
- Show changes:
git difftool .gitlab-ci.yml origin/master
- STOP DEMO HERE TO GRADE SCORECARD
Step 2: Run the full QA regression test suite on an offline environment
- Show previously run pipeline
- Show iptables commands
- Show airgap check for instance
- Show airgap check for runner
- Show success for whole job
- Show the currently running pipeline
- Wait for it to complete successfully
- STOP DEMO HERE TO GRADE SCORECARD
Secure test projects
Quality will conduct a live demo or demos to show the Secure test projects within a group on the GCP demo project running as an integration test.
This work is covered in more detail on the demo environment issue: #207063 (closed)
Step 1: Prove airgap
- Log into GCP admin area: https://console.cloud.google.com/home/dashboard
- Navigate to group-secure.
- Navigate to compute.
- Search for “airgap”
- SSH into airgap-test (the running offline instance)
- gcloud beta compute ssh --zone "us-west1-b" "gitlab-airgap-test" --project "group-secure-a89fe7"
- nc -vvvv -w 5 registry.gitlab.com 443 (and 80, 22, 5000)
- RESULT: No access to public internet
- Log into https://34.82.7.216/
- If /etc/hosts file overridden: https://gitlab-airgap-test.us-west1-b.c.group-secure-a89fe7.internal
- Make comments about the URL of the instance and you having .etc-host file modification to easily access.
- STOP DEMO HERE TO GRADE SCORECARD
Step 2: Set up an integration test group
- Create new integration test group
- Show the group-level security dashboard is empty
- STOP DEMO HERE TO GRADE SCORECARD
Step 3: Set up python-pip with all five scanners
- Fork python-pip
- Navigate to Groups => Templates => python-pip
- Fork python-pip into the integration test group
- Remove the fork relationship
- Show the project-level security dashboard is empty
- Show that none of the scanners are configured.
- Create the .gitlab-ci.yml
- Click into Web IDE
- Create new file => .gitlab-ci.yml
- Add the check-airgap job
- Set up Dependency Scanning
- Use documentation: https://docs.gitlab.com/ee/user/application_security/dependency_scanning
- Set up License Compliance
- Use documentation: https://docs.gitlab.com/ee/user/compliance/license_compliance
- Set up SAST
- Use documentation: https://docs.gitlab.com/ee/user/application_security/sast
- Set up DAST
- Use documentation: https://docs.gitlab.com/ee/user/application_security/dast
- Set up Container Scanning
- Use documentation: https://docs.gitlab.com/ee/user/application_security/container_scanning
- Add a Dockerfile to the project
- Commit the changes and open an MR
- Watch the pipeline until it passes.
- Refresh the page to show the MR security tab and licenses tab are populated as expected.
- Merge the MR
- At the project level, show all five scanners are configured.
- Watch the master pipeline until it passes.
- Show the project-level security dashboard and licenses page are populated as expected.
- Show the group-level security dashboard is populated as expected.
- STOP DEMO HERE TO GRADE SCORECARD
Step 4: Set up java-maven with three scanners
- Fork java-maven
- Navigate to Groups => Templates => python-pip
- Fork python-pip into the integration test group
- Remove the fork relationship
- Show the project-level security dashboard is empty
- Show that none of the scanners are configured.
- Create the .gitlab-ci.yml
- Click into Web IDE
- Create new file => .gitlab-ci.yml
- Add the check-airgap job
- Set up Dependency Scanning
- Use documentation: https://docs.gitlab.com/ee/user/application_security/dependency_scanning
- Set up License Compliance
- Use documentation: https://docs.gitlab.com/ee/user/compliance/license_compliance
- Set up Container Scanning
- Use documentation: https://docs.gitlab.com/ee/user/application_security/container_scanning
- Add a Dockerfile to the project
- Commit the changes and open an MR
- Watch the pipeline until it passes.
- Refresh the page to show the MR security tab and licenses tab are populated as expected.
- Merge the MR
- At the project level, show all five scanners are configured.
- Watch the master pipeline until it passes.
- Show the project-level security dashboard and licenses page are populated as expected.
- Show the group-level security dashboard is populated as expected.
- STOP DEMO HERE TO GRADE SCORECARD
Stage 2: Automated
Within https://gitlab.com/gitlab-org/quality/offline-environment-testing, set up a weekly pipeline to run a pared-down, offline version of the GitLab-QA tests (pared down = no quarantined specs, etc).
Stage 3: Automated (out of scope for MVC)
Within https://gitlab.com/gitlab-org/quality/nightly, add an airgapped smoke test.
Within https://gitlab.com/gitlab-org/quality/offline-environment-testing, expand the weekly pipeline to become a multi-project pipeline in order to also run offline versions of the Secure test projects.