Secure solution for managing GitLab self-managed configuration secrets
<!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION --> *This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.* <!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION --> ### Summary GitLab configuration files contain plain text passwords that are used to authenticate connections to other services required for GitLab operations. This is not a secure way to handle passwords and often becomes a blocker or concern for customers that need GitLab to meet basic security requirements. ### Overview GitLab passwords are currently read from `gitlab.yml`. They are stored in both `gitlab.yml` and `gitlab.rb`. When Chef updates the YAML file the passwords are output to the log. Additionally, secrets are found in `/etc/gitlab/gitlab-secrets.json` and left as files on disk in plain text in individual components `/var/opt/gitlab/{component}`. They are not obfuscated or removed after the application has read them. ### Use cases we have heard * We have a limited number of employees that have admin access to GitLab and our teams tend to be fairly siloed. When someone who is not an admin needs configuration information from GitLab we copy that section of the configuration file and email it to them rather than give them admin access. This can be a problem if the copied settings include clear text passwords. * We do version control of our configuration files. When we change something like a password in a config file, that change shows up in our logs. We don't want clear text passwords in our logs. * Our security team does not allow clear text passwords to be stored in unencrypted files. We were warned about this in our last audit. If it comes up again in our next annual security audit it will be difficult to defend. ### Feedback from customer interviews * The password that users are most concerned about is the LDAP password for connecting to their LDAP server. They are also concerned about the LDAP username being in clear text. * Customers already have their own secrets management system and don't want to introduce another one, especially if it is not bundled in GitLab. That would require them to obtain vendor approval internally which can be a lengthy process. * If a third-party secrets management system is bundled in GitLab they might not need to get approval, but they would still prefer to integrate with the secrets management system they already use. * For some customers, the passwords would ideally be encrypted within the configuration file and would not require integration with a secrets management system. * Other customers said it would be nice to have the ability to store secrets in a third-party or proprietary secrets management system but this is not a hard requirement right now. ### Proposal Encrypt passwords within the configuration files and redesign GitLab to be able to read the encrypted passwords. ### MVP Encrypt the password for the customer's LDAP server. ### Notes * The Release team is considering a BYO or bundled Vault integration for storing passwords used for applications deployed using GitLab ### Secrets that are stored in GitLab files https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5048#note_320640700 ### Related issues containing background and requests * https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2183 * https://gitlab.com/gitlab-org/gitlab-foss/issues/38066 * https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2156 * See https://docs.google.com/document/d/1Rt-xIIEI9buRvLbOwZl4mmmgEQThuFSHUerxNWjSbk4/edit?usp=sharing for research done by @ibaum in 2018 ### Solution After researching various options and conducting two POCs, the following solution has been proposed as the quickest way to solve encrypting the LDAP password. - There will be a new encrypted YAML file that contains the LDAP password. The YAML file needs to be stored on each application node. By default, the file is stored in `/var/opt/gitlab/gitlab-rails/etc/ldap_secret.yaml.enc` but the location can be changed. - An encryption key will be used to encrypt the file. The encryption key is stored in plaintext in `gitlab-secrets.json` and `secrets.json`. This file can only be accessed by the `root` and `git` user. - Passwords can be added or changed using a rake command. Anyone can run the command, but it won't do anything unless you have access permissions to the encryption key file.
epic