[MVC] Offline / Limited Connectivity / Air-gapped vulnerability analysis and license compliance for on-prem instances
### Status [updated 2020-05-07] Please note all code and documentation for MVC are complete We are awaiting some sign off from stakeholders and some quality tests but all functionality is present as expected. #### Complete - ✅[SAST & Secrets Detection](https://gitlab.com/groups/gitlab-org/-/epics/2338) - [Documentation](https://docs.gitlab.com/ee/user/application_security/sast/#gitlab-sast-in-an-offline-air-gapped-installation) - ✅ [Container Scanning](https://gitlab.com/groups/gitlab-org/-/epics/2461) - [Documentation](https://docs.gitlab.com/ee/user/application_security/container_scanning/#running-container-scanning-in-an-offline-air-gapped-installation) - ✅ [DAST](https://gitlab.com/groups/gitlab-org/-/epics/2339)- [Documentation](https://docs.gitlab.com/ee/user/application_security/dast/#running-dast-in-an-offline-air-gapped-installation) - ✅ [Custom CA Support](https://gitlab.com/groups/gitlab-org/-/epics/2748) - Overrides are available, documented, and working for all MVC scanners to bypass cert failure #### In Progress - [License Compliance](https://gitlab.com/groups/gitlab-org/-/epics/2012), waiting on final signoff - ✅ Code Complete (for the MVC languages), [Documentation](https://docs.gitlab.com/ee/user/compliance/license_compliance/index.html#running-license-compliance-in-an-offline-environment) Complete - ⚠️ Demo, need to get passing grade (docs updates are only blocker on grade) - [Dependency](https://gitlab.com/groups/gitlab-org/-/epics/2011) - ⚠️ Code Status - [1 issue](https://gitlab.com/groups/gitlab-org/-/epics/2011) - ✅ [Documentation](https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html#running-dependency-scanning-in-an-offline-environment) complete - Demo Status - ✅[MVC video](https://www.youtube.com/watch?v=FyqwSNFsa4E) - ✅[Create Air-Gap Demo for Dependency Scanning issue](https://gitlab.com/gitlab-org/gitlab/-/issues/209816) - [video April 2](https://youtu.be/u4amXhEfSXs) - ✅[April 9 video](https://www.youtube.com/watch?v=uGhS2Wh6PBE) - ✅[April 17 video](https://www.youtube.com/watch?v=MDgOpZPPucc) - ⚠️ Need to do another demo to get to 5 score (sbt, docs updates) - QA Status - 1 issue for all scanners currently - [Create automated demo environment and test plan for offline/air-gapped deployments](https://gitlab.com/gitlab-org/gitlab/-/issues/207063) schedule for %"13.0" - Environment with GitLab - SAST Test - DAST Test - Container Test - Dependency Test - License Test Test - Demo an environment setup and all scanner run ### Description GitLab integrates a lot of security scans that aim to find vulnerabilities on your software. These scans are performed during the CI/CD pipeline, and they are requiring the runner to be able to reach the internet for a few reasons: 1. access to external Docker images, available on Docker Hub 1. access to external Docker images, available on GitLab.com Container Registry 1. access to Gemnasium services, hosted by GitLab 1. keep vulnerability databases up to date This works very well on GitLab.com, but not on on-prem installations that don't have consistent or open access to the internet, or where the access is really limited to a specific set of hosts. In this case the security checks cannot be done. ### Proposal Allow mirroring or downloading of all the needed information for an offline use. Data will be replicated and kept in sync on a separate server, internal to the same network where the runners are, and will provide all the information needed. Provide documentation & instructions describing how to configure an air-gapped instance to run scanners successfully. ### Approach Users are capable of performing some scans in air-gapped environments today and there are several soft dependencies that we believe need to be tackled in order to create a full, lovable experience across a broad set of use cases for air-gapped security scanning. #### Pre - Requisites * Hard [Make Docker-in-Docker (DinD) optional (not a requirement) for all secure analyzers / scans](https://gitlab.com/groups/gitlab-org/-/epics/971) one issue remains in %12.9 * ~~Soft [Splitting the build and analyze phases of security scans](https://gitlab.com/gitlab-org/gitlab/issues/10479)~~ nice to have but not a blocking pre-req #### Priority * Based on customer feedback we will prioritize the order in which we test and make changes to each scanner (each solution has many scanners) and update it as "available" for offline use, along with the steps required to run it offline. ** Current Priority ** Scanners with upvotes counted 1. [SAST](https://gitlab.com/groups/gitlab-org/-/epics/2338) 1. [DAST](https://gitlab.com/groups/gitlab-org/-/epics/2339) 1. [Dependency](https://gitlab.com/groups/gitlab-org/-/epics/2011) 1. Container Scanning 1. [License Compliance](https://gitlab.com/groups/gitlab-org/-/epics/2012) Languages with upvotes counted 1. Python +3 1. Java +4 (and private maven repos) 1. JavaScript 1. C/C++ +2 1. Go +2 1. NodeJS +2 1. C# 1. PHP 1. Ruby on Rails ### Links See also https://gitlab.com/gitlab-org/gitlab-ee/issues/6603 for historical context. ### Questions To Be Answered By Account Teams If you have an account that needs air-gapped / offline / limited connectivity use of Secure, please answer the following questions in a comment on this issue as well as a Salesforce link to the account. This will allow the Secure PM team to have better traceability of user needs. 1. Which parts of Secure (i.e. which scanners) are they looking to use and evaluate? 1. What languages are they needing support on and what is the priority order? 1. Would it be possible to allow them to evaluate a subset of Secure to confirm air-gapped is working? 1. Have they reviewed the current available air-gap documentation? Do they have feedback or have they experienced issues? ### Proposed Plan Following Gitlab's core values, air-gapped / offline / limited connectivity will be rolled out iteratively. As such, prioritization of features and scanners will be made based on the answers above as well as other customer commitments. We will start with making the [boring solution](https://about.gitlab.com/handbook/values/#boring-solutions) and documentation describing it and iteratively create a lovable experience.
epic