I'm going to add some more detail about why this should be done. I don't think there was enough information in the previous description.
What we're trying to protect against:
Malicious modifications of packages on the package server.
Hijacked upgrades for existing installations by modifying DNS to point to a rogue package server (SSL should hopefully prevent this).
Other forms of man-in-the-middle attacks.
Our options for protecting users:
Signed packages.
Package hashes.
There are also infrastructure protections that help but those are handled in other trackers and many are already in-place.
Known knowns:
Packagecloud does not provide support for automatically signing packages as they arrive. It provides signatures for metadata only, using an automatically generated key stored on the server.
Packagecloud does automatically create hashes for each file and distributes them on the package download page.
Omnibus also creates hashes and provides them in the form of a <package>.metadata.json file along with the package when it's finished.
Both RPM and DEB packages support GnuPG signatures inside the packages themselves.
Hashes do not provide any form of automated protection, but they are trivial to generate. Users must manually verify hashes against the known-good list. This list must be kept in a safe place (i.e. not the package server) as it is trivial to modify.
It is possible to sign a hash using GnuPG so that it cannot be modified. The easiest thing to do is sign the file that contains all of the hashes.
Publishing hashes for all releases was seen as a good intermediate step towards package signatures, as they are simple to create and distribute once you have chosen a safe location to publish them.
To properly implement package hashing we need:
Changes to the build tools to automatically generate hashes, preferably in one file per release so users don't have to search through a large list of files to find the right hash.
A published location where these hashes will be uploaded for every release.
Updates to the installation instructions to notify users how to verify hashes and where they are located.
Should we decide to sign DEB and RPM packages (which I think is a good idea) we will need the following steps:
A signing key, safely stored in a location available to release managers as needed.
A public key and fingerprint published in a location available to all users and to automated package verification tools.
Modifications to the release tools to support package signing.
I hope that clears up some of the intent behind this and how I hoped we'd get there. This is still a discussion issue so I don't want to give the impression I'm forcing a set of actions. Please let me know if you agree/disagree.
The next step would be deciding where exactly to publish these hashes. The package server could be used, but if it's compromised they could easily be modified along with the packages.
If this is what is needed, we can easily add a rake task to do that and call it at the end of the build and provide it as an artifact. What to do with that artifact, should be discussed. Maybe RMs can collect all of them and update a page in our website. Or we can think about automating that too.
No manual tasks for this! If we cannot have it fully automatic we shouldn't have it at all. No one should have to do 22 package updates per release for 10-11 releases per month.
@briann First step is to figure out where you want to have this published. Next is to see how we can make that update automatic. We can talk about the rest once we have that figured out.
@marin Most companies publish these hashes on the downloads page. Because we have so many packages and update so often I thought it would be easier to store these hashes in a file in a repository, as I've seen some open source projects do, and link to it from the downloads page.
But on further thought I don't suppose that would be any easier than pushing them to the web repository. What do you think about creating a "Verifying Hashes" or "Verifying package integrity" link on the https://about.gitlab.com/installation/ page and linking to a text file in the web repo with a full list of hashes that's automatically updated with each release?
Turns out Omnibus already creates md5, sha1, sha256, and sha512 hashes for each package. They're stored in the *.deb.metadata.json file and uploaded to S3. It should be easy enough to pull these back down as a final step in the build process, combine them, and upload the result to a location that makes sense.
I made an attempt to publish all hashes to S3 here: !1718 (closed)
Brian Neelchanged title from Creating and publishing hashes for all Omnibus packages to Creating and publishing hashes/checksums for all Omnibus packages
changed title from Creating and publishing hashes for all Omnibus packages to Creating and publishing hashes/checksums for all Omnibus packages
@briann I think we can do this differently than what you proposed in !1718 (closed) . We are trying to completely move away from this way of working anyway. If we are to already upload this information we should tie it to the release task. We should also consider having a separate bucket for this, separate from where the packages are stored.
@marin if Omnibus is creating the builds, uploading them to S3, and pushing them to the package server, is there an opening in there for the release tools to sign them? I suppose the hash creation can be moved to a release task but I'm not seeing an obvious way to do the same for signatures. Not unless the packages are pushed twice?
I took at look at https://about.gitlab.com/downloads/archives/, which is derived from https://gitlab.com/gitlab-com/www-gitlab-com/tree/master/source/downloads/archives/index.html.haml. If we can manage to derive a method to have this auto-generated as a part of the deployment process, perhaps we could have it as a single step in the RM process to update the source list and thus automatically update the page on deployment. We will, however, require that to be a confidential issue, as we'd not want to push that before the packages are also live.
I'd point out that https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/source/downloads/182BAD96.asc is present, but I am not directly sure what it is/was used for at this point in time.
Brian Neelchanged title from Creating and publishing hashes/checksums for all Omnibus packages to Creating signatures and/or hashes for all Omnibus packages
changed title from Creating and publishing hashes/checksums for all Omnibus packages to Creating signatures and/or hashes for all Omnibus packages
Install the private signing key, protected by passphrase, to a private bucket. This key is also stored in a vault associated with the build team.
Modify the Omnibus build CI to pull down the signing key when building a release package. The package will be signed as it's created and prior to uploading to S3.
Add the passphrase for the signing key to the Omnibus project on dev.
Add the public signing key to a test repository on packagecloud.
Upload a package signed with the signing key and install it in a test instance.
Upload an unsigned package and verify that both RPM and DEB are refused by yum and apt, respectively.
Once these steps are completed we should have proper package signing integrated into the release process.
@WarheadsSE Has been kind enough to volunteer to write/update the signing code and @twk3 has performed the initial testing of package signature verification with CentOS and RPMs.
I am rebasing @marin's original work on !922 (closed) to master, and testing appropriate modifications.
Currently, Omnibus itself (not Omnibus GitLab) is limited in that it does not support signing .deb packages. It does support .rpm, and I am testing that at the moment. The (private) repository for this is test-signing. We (@gitlab-build-team) would much prefer to sign the packages directly from the Omnibus run, however some modifications may be required for this.
If we can not use Omnibus for signing .deb packages, then we have a second alternative: alter the available repository:* Rake tasks to handle signing packages prior to uploading, by injecting a step into the Makefile to call bundle exec rake repository:sign_package which would then do the signing prior to the package being uploaded to S3 or PackageCloud.
Some work has already been done by their community in this regard, and we can likely base on that work to submit patches to address the lack of debsigs support for .deb files. We need to note however that the is a dependency on debsigs that is not specified by Omnibus itself in the current state of the previous work.
I have completed the rebase (and test) of @marin's previous !922 (closed) onto master, and seen that we do indeed get a signed .rpm, and then uploaded to PackageCloud, and then installed on a CentOS container. Following completely through the signed package behaviors. (note: this was on an essentially 'fresh' install of CentOS)
I have taken the second step of attempting add .deb signing to our Rake tasks. In this, I have added only the Debian package signing, and noted in the output that .rpm files are handled via Omnibus directly.
I see this as a faster implementation, should it work, than getting the necessary changes into Omnibus upstream. I still strongly feel that we should perform the task of including it in Omnibus and upstreaming those changes, however this seemed like the fastest route to working signed packages.
@WarheadsSE Given that the MR !1752 (closed) is in Ruby, I think it makes more sense to move the deb sign logic into omnibus. We do not have to wait for upstream to do anything. We have our own fork that we maintain where we can create a MR and polish the change. Once we do that, we should try to contribute back to upstream. Let me know if you need help in wrapping your head around omnibus.
I have begun the work to migrate the code into Omnibus itself based on our fork, and closed !1752 (closed).
I will have the MR for the Omnibus additions, likely by end of day. Some handling of gpg via shellout! is troublesome, but less hassle than pulling in GPGME as an additional dependency of Omnibus itself.
@WarheadsSE@twk3 Any idea how this will affect the ability to download unsigned packages? I'm curious if installing the key on packagecloud will prevent users from downloading the old packages.
@briann It should not make any difference apart from users getting a warning (and a prompt) once the packages are signed when they try to install an older package.
Place a mix of signed rpms and debs with older packages which were unsigned
Do the test scenarios for both rpm's and deb's
Test scenarios:
Clean machine: Install the apt(yum) repository
Install the unsigned package
Upgrade to the signed package
Clean machine: Install the apt(yum) repository
Install the signed package
Downgrade to the unsigned package
Note down all(and I mean all) prompts and queries you had to answer.
What we are looking for:
When adding repos, users have to have minimal interaction with it. Yes/No question on accepting key is acceptable for now
Upgrading/installing/downgrading signed/unsigned package should not require any additional actions from users. If they do, note down what was required (copy the prompt and dump it in this issue).
DEB-based distributions tend to handle the signatures at the repository level, and I have not yet found a way to enable debsigs checking at the system level for a specific repository, Debian/Ubuntu should have no issues at all. debsigs can be turned on system wide, however it applies to all repositories, and not even Debian ships with every package in their repository signed.
RPM might be a touch different, as this format has much better integration into the format itself. I agree that we should test all of these (SLES/CentOS/OpenSUSE).
I already have a testing repository at test-signing, and will use that.
Having checked CentOS7 & Debian 9 (using clean Docker images)
Debian has no issues, as it is using the Repository Metadata signing as opposed to the package signing (offered via debsigs).
CentOS/RHEL may have issues because the installation script turns on gpgcheck in the repository configuration because a package signing key is in place. A user will then have to add the --nogpgcheck flag to yum calls for installing older versions.
Below are the logs from clean docker images (snipped for tidiness)
Use repository installation script
Install 9.3.6-ce.0 respectively
Install packages from pipeline (e.g. 8.1.0+git.3216.1f52045-rc1.ce.0)
Debian 9 (docker run -ti debian:9 /bin/bash -l)
No issues
root@b7f80603bb06:/# apt-get update; apt-get install curl gnupg2...After this operation, 32.0 MB of additional disk space will be used.Do you want to continue? [Y/n] y...root@b7f80603bb06:/# curl -s https://:@packages.gitlab.com/install/repositories/gitlab/test-signing/script.deb.sh | bashDetected operating system as debian/9.Checking for curl...Detected curl...Running apt-get update... done.Installing debian-archive-keyring which is needed for installing apt-transport-https on many Debian systems.Installing apt-transport-https... done.A unique ID was not specified, using the machine's hostname...Found unique id: b7f80603bb06 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 105 0 105 0 0 196 0 --:--:-- --:--:-- --:--:-- 195Installing /etc/apt/sources.list.d/gitlab_test-signing.list...done.Importing packagecloud gpg key... done.Running apt-get update... done.The repository is setup! You can now install packages.root@b7f80603bb06:/# root@b7f80603bb06:/# apt-get install gitlab-ce=9.3.6-ce.0...After this operation, 1171 MB of additional disk space will be used.Do you want to continue? [Y/n] y...Setting up gitlab-ce (9.3.6-ce.0) ... *. *. *** *** ***** ***** .****** ******* ******** ******** ,,,,,,,,,***********,,,,,,,,, ,,,,,,,,,,,*********,,,,,,,,,,, .,,,,,,,,,,,*******,,,,,,,,,,,, ,,,,,,,,,*****,,,,,,,,,. ,,,,,,,****,,,,,, .,,,***,,,, ,*,. _______ __ __ __ / ____(_) /_/ / ____ _/ /_ / / __/ / __/ / / __ `/ __ \ / /_/ / / /_/ /___/ /_/ / /_/ / \____/_/\__/_____/\__,_/_.___/gitlab: Thank you for installing GitLab!gitlab: To configure and start GitLab, RUN THE FOLLOWING COMMAND:sudo gitlab-ctl reconfiguregitlab: GitLab should be reachable at http://b7f80603bb06gitlab: Otherwise configure GitLab for your system by editing /etc/gitlab/gitlab.rb filegitlab: And running reconfigure again.gitlab: gitlab: For a comprehensive list of configuration options please see the Omnibus GitLab readmegitlab: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.mdgitlab: It looks like GitLab has not been configured yet; skipping the upgrade script....root@b7f80603bb06:/#root@b7f80603bb06:/# apt-get install gitlab-ce=8.1.0+git.3216.1f52045-rc1.ce.0...After this operation, 19.6 MB of additional disk space will be used.Do you want to continue? [Y/n] yGet:1 https://packages.gitlab.com/gitlab/test-signing/debian stretch/main amd64 gitlab-ce amd64 8.1.0+git.3216.1f52045-rc1.ce.0 [362 MB]Fetched 362 MB in 13s (27.2 MB/s) debconf: delaying package configuration, since apt-utils is not installeddpkg: warning: downgrading gitlab-ce from 9.3.6-ce.0 to 8.1.0+git.3216.1f52045-rc1.ce.0(Reading database ... 83142 files and directories currently installed.)Preparing to unpack .../gitlab-ce_8.1.0+git.3216.1f52045-rc1.ce.0_amd64.deb ...Unpacking gitlab-ce (8.1.0+git.3216.1f52045-rc1.ce.0) over (9.3.6-ce.0) ...Setting up gitlab-ce (8.1.0+git.3216.1f52045-rc1.ce.0) ... *. *. *** *** ***** ***** .****** ******* ******** ******** ,,,,,,,,,***********,,,,,,,,, ,,,,,,,,,,,*********,,,,,,,,,,, .,,,,,,,,,,,*******,,,,,,,,,,,, ,,,,,,,,,*****,,,,,,,,,. ,,,,,,,****,,,,,, .,,,***,,,, ,*,. _______ __ __ __ / ____(_) /_/ / ____ _/ /_ / / __/ / __/ / / __ `/ __ \ / /_/ / / /_/ /___/ /_/ / /_/ / \____/_/\__/_____/\__,_/_.___/gitlab: Thank you for installing GitLab!gitlab: To configure and start GitLab, RUN THE FOLLOWING COMMAND:sudo gitlab-ctl reconfiguregitlab: GitLab should be reachable at http://b7f80603bb06gitlab: Otherwise configure GitLab for your system by editing /etc/gitlab/gitlab.rb filegitlab: And running reconfigure again.gitlab: gitlab: For a comprehensive list of configuration options please see the Omnibus GitLab readmegitlab: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.mdgitlab: It looks like GitLab has not been configured yet; skipping the upgrade script.root@b7f80603bb06:/#
CentOS 7 (docker run -ti centos:7 /bin/bash -l)
Notes: to install an unsigned package, one must add --nogpgcheck to the yum call.
[root@04cd1fb01fd3 /]# curl -s https://:@packages.gitlab.com/install/repositories/gitlab/test-signing/script.rpm.sh | bashDetected operating system as centos/7.Checking for curl...Detected curl...A unique ID was not specified, using the machine's hostname...Found unique id: 04cd1fb01fd3Downloading repository file: https://:@packages.gitlab.com/install/repositories/gitlab/test-signing/config_file.repo?os=centos&dist=7&name=04cd1fb01fd3&source=scriptdone.Installing pygpgme to verify GPG signatures...Loaded plugins: fastestmirror, ovlgitlab_test-signing-source/signature | 836 B 00:00:00 Retrieving key from https://:@packages.gitlab.com/gitlab/test-signing/gpgkeyImporting GPG key 0xE15E78F4: Userid : "GitLab B.V. (package repository signing key) <packages@gitlab.com>" Fingerprint: 1a4c 919d b987 d435 9396 38b9 1421 9a96 e15e 78f4 From : https://:@packages.gitlab.com/gitlab/test-signing/gpgkeyRetrieving key from https://:@packages.gitlab.com/gitlab/test-signing/gpgkey/gitlab-test-signing-331E7725DC169E52.pub.gpggitlab_test-signing-source/signature | 951 B 00:00:02 !!! gitlab_test-signing-source/primary | 175 B 00:00:00 Loading mirror speeds from cached hostfile * base: mirror.cloud-bricks.net * extras: centos.vwtonline.net * updates: mirrors.lga7.us.voxel.netPackage pygpgme-0.3-9.el7.x86_64 already installed and latest versionNothing to doInstalling yum-utils...Loaded plugins: fastestmirror, ovlLoading mirror speeds from cached hostfile * base: mirror.cloud-bricks.net * extras: centos.vwtonline.net * updates: mirrors.lga7.us.voxel.netPackage yum-utils-1.1.31-40.el7.noarch already installed and latest versionNothing to doGenerating yum cache for gitlab_test-signing...Importing GPG key 0xE15E78F4: Userid : "GitLab B.V. (package repository signing key) <packages@gitlab.com>" Fingerprint: 1a4c 919d b987 d435 9396 38b9 1421 9a96 e15e 78f4 From : https://:@packages.gitlab.com/gitlab/test-signing/gpgkeyThe repository is setup! You can now install packages.[root@04cd1fb01fd3 /]# yum --showduplicates list gitlab-ceLoaded plugins: fastestmirror, ovlLoading mirror speeds from cached hostfile * base: mirror.cloud-bricks.net * extras: centos.vwtonline.net * updates: mirrors.lga7.us.voxel.netAvailable Packagesgitlab-ce.x86_64 8.1.0+git.3217.3a100f1-rc1.ce.0.el7 gitlab_test-signinggitlab-ce.x86_64 9.3.6-ce.0.el7 gitlab_test-signing[root@04cd1fb01fd3 /]# yum install gitlab-ce-9.3.6-ce.0.el7 ...Dependencies Resolved============================================================================================= Package Arch Version Repository Size=============================================================================================Installing: gitlab-ce x86_64 9.3.6-ce.0.el7 gitlab_test-signing 338 MInstalling for dependencies: fipscheck x86_64 1.4.1-5.el7 base 21 k fipscheck-lib x86_64 1.4.1-5.el7 base 11 k openssh x86_64 6.6.1p1-35.el7_3 updates 438 k openssh-server x86_64 6.6.1p1-35.el7_3 updates 440 k tcp_wrappers-libs x86_64 7.6-77.el7 base 66 kTransaction Summary=============================================================================================Install 1 Package (+5 Dependent packages)Total download size: 339 MInstalled size: 1.0 GIs this ok [y/d/N]: yDownloading packages:warning: /var/cache/yum/x86_64/7/base/packages/fipscheck-lib-1.4.1-5.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEYPublic key for fipscheck-lib-1.4.1-5.el7.x86_64.rpm is not installed(1/6): fipscheck-lib-1.4.1-5.el7.x86_64.rpm | 11 kB 00:00:00 (2/6): tcp_wrappers-libs-7.6-77.el7.x86_64.rpm | 66 kB 00:00:00 Public key for openssh-6.6.1p1-35.el7_3.x86_64.rpm is not installed/s | 737 kB 00:19:05 ETA (3/6): openssh-6.6.1p1-35.el7_3.x86_64.rpm | 438 kB 00:00:01 (4/6): openssh-server-6.6.1p1-35.el7_3.x86_64.rpm | 440 kB 00:00:01 (5/6): fipscheck-1.4.1-5.el7.x86_64.rpm | 21 kB 00:00:05 Package gitlab-ce-9.3.6-ce.0.el7.x86_64.rpm is not signed=-] 26 MB/s | 338 MB 00:00:00 ETA (6/6): gitlab-ce-9.3.6-ce.0.el7.x86_64.rpm | 338 MB 00:00:13 ---------------------------------------------------------------------------------------------Total 25 MB/s | 339 MB 00:00:13 Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7Importing GPG key 0xF4A80EB5: Userid : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>" Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5 Package : centos-release-7-3.1611.el7.centos.x86_64 (@CentOS) From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7Is this ok [y/N]: yPackage gitlab-ce-9.3.6-ce.0.el7.x86_64.rpm is not signed[root@04cd1fb01fd3 /]# yum install gitlab-ce-9.3.6-ce.0.el7 --nogpgcheck Loaded plugins: fastestmirror, ovlgitlab_test-signing | 1.0 kB 00:00:00 gitlab_test-signing-source | 951 B 00:00:00 Loading mirror speeds from cached hostfile * base: mirror.cloud-bricks.net * extras: centos.vwtonline.net * updates: mirrors.lga7.us.voxel.netResolving Dependencies--> Running transaction check---> Package gitlab-ce.x86_64 0:8.1.0+git.3217.3a100f1-rc1.ce.0.el7 will be updated---> Package gitlab-ce.x86_64 0:9.3.6-ce.0.el7 will be an update--> Finished Dependency ResolutionDependencies Resolved============================================================================================= Package Arch Version Repository Size=============================================================================================Updating: gitlab-ce x86_64 9.3.6-ce.0.el7 gitlab_test-signing 338 MTransaction Summary=============================================================================================Upgrade 1 PackageTotal size: 338 MIs this ok [y/d/N]: yDownloading packages:Running transaction checkRunning transaction testTransaction test succeededRunning transaction Updating : gitlab-ce-9.3.6-ce.0.el7.x86_64 1/2 /var/tmp/rpm-tmp.An8LeF: line 42: which: command not found *. *. *** *** ***** ***** .****** ******* ******** ******** ,,,,,,,,,***********,,,,,,,,, ,,,,,,,,,,,*********,,,,,,,,,,, .,,,,,,,,,,,*******,,,,,,,,,,,, ,,,,,,,,,*****,,,,,,,,,. ,,,,,,,****,,,,,, .,,,***,,,, ,*,. _______ __ __ __ / ____(_) /_/ / ____ _/ /_ / / __/ / __/ / / __ `/ __ \ / /_/ / / /_/ /___/ /_/ / /_/ / \____/_/\__/_____/\__,_/_.___/gitlab: Thank you for installing GitLab!gitlab: To configure and start GitLab, RUN THE FOLLOWING COMMAND:sudo gitlab-ctl reconfiguregitlab: GitLab should be reachable at http://04cd1fb01fd3gitlab: Otherwise configure GitLab for your system by editing /etc/gitlab/gitlab.rb filegitlab: And running reconfigure again.gitlab: gitlab: For a comprehensive list of configuration options please see the Omnibus GitLab readmegitlab: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.mdgitlab: Cleanup : gitlab-ce-8.1.0+git.3217.3a100f1-rc1.ce.0.el7.x86_64 2/2 It looks like GitLab has not been configured yet; skipping the upgrade script. Verifying : gitlab-ce-9.3.6-ce.0.el7.x86_64 1/2 Verifying : gitlab-ce-8.1.0+git.3217.3a100f1-rc1.ce.0.el7.x86_64 2/2 Updated: gitlab-ce.x86_64 0:9.3.6-ce.0.el7 Complete![root@04cd1fb01fd3 /]# yum install gitlab-ce-8.1.0+git.3217.3a100f1-rc1.ce.0.el7.x86_64...Dependencies Resolved============================================================================================= Package Arch Version Repository Size=============================================================================================Installing: gitlab-ce x86_64 8.1.0+git.3217.3a100f1-rc1.ce.0.el7 gitlab_test-signing 343 MInstalling for dependencies: fipscheck x86_64 1.4.1-5.el7 base 21 k fipscheck-lib x86_64 1.4.1-5.el7 base 11 k openssh x86_64 6.6.1p1-35.el7_3 updates 438 k openssh-server x86_64 6.6.1p1-35.el7_3 updates 440 k tcp_wrappers-libs x86_64 7.6-77.el7 base 66 kTransaction Summary=============================================================================================Install 1 Package (+5 Dependent packages)Total size: 344 MTotal download size: 343 MInstalled size: 1.0 GIs this ok [y/d/N]: yDownloading packages:Delta RPMs disabled because /usr/bin/applydeltarpm not installed.warning: /var/cache/yum/x86_64/7/gitlab_test-signing/packages/gitlab-ce-8.1.0+git.3217.3a100f1-rc1.ce.0.el7.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID c0207d21: NOKEYPublic key for gitlab-ce-8.1.0+git.3217.3a100f1-rc1.ce.0.el7.x86_64.rpm is not installedgitlab-ce-8.1.0+git.3217.3a100f1-rc1.ce.0.el7.x86_64.rpm | 343 MB 00:00:13 Retrieving key from https://:@packages.gitlab.com/gitlab/test-signing/gpgkeyImporting GPG key 0xE15E78F4: Userid : "GitLab B.V. (package repository signing key) <packages@gitlab.com>" Fingerprint: 1a4c 919d b987 d435 9396 38b9 1421 9a96 e15e 78f4 From : https://:@packages.gitlab.com/gitlab/test-signing/gpgkeyIs this ok [y/N]: yRetrieving key from https://:@packages.gitlab.com/gitlab/test-signing/gpgkey/gitlab-test-signing-331E7725DC169E52.pub.gpgImporting GPG key 0xC0207D21: Userid : "GitLab Inc. <support@gitlab.com>" Fingerprint: cc95 24b5 9894 c4c0 a51a cd62 66d2 6543 c020 7d21 From : https://:@packages.gitlab.com/gitlab/test-signing/gpgkey/gitlab-test-signing-331E7725DC169E52.pub.gpgIs this ok [y/N]: yRunning transaction checkRunning transaction testTransaction test succeededRunning transaction Installing : fipscheck-1.4.1-5.el7.x86_64 1/6 Installing : fipscheck-lib-1.4.1-5.el7.x86_64 2/6 Installing : openssh-6.6.1p1-35.el7_3.x86_64 3/6 Installing : tcp_wrappers-libs-7.6-77.el7.x86_64 4/6 Installing : openssh-server-6.6.1p1-35.el7_3.x86_64 5/6 Installing : gitlab-ce-8.1.0+git.3217.3a100f1-rc1.ce.0.el7.x86_64 6/6 /var/tmp/rpm-tmp.QbRDko: line 42: which: command not found *. *. *** *** ***** ***** .****** ******* ******** ******** ,,,,,,,,,***********,,,,,,,,, ,,,,,,,,,,,*********,,,,,,,,,,, .,,,,,,,,,,,*******,,,,,,,,,,,, ,,,,,,,,,*****,,,,,,,,,. ,,,,,,,****,,,,,, .,,,***,,,, ,*,. _______ __ __ __ / ____(_) /_/ / ____ _/ /_ / / __/ / __/ / / __ `/ __ \ / /_/ / / /_/ /___/ /_/ / /_/ / \____/_/\__/_____/\__,_/_.___/gitlab: Thank you for installing GitLab!gitlab: To configure and start GitLab, RUN THE FOLLOWING COMMAND:sudo gitlab-ctl reconfiguregitlab: GitLab should be reachable at http://04cd1fb01fd3gitlab: Otherwise configure GitLab for your system by editing /etc/gitlab/gitlab.rb filegitlab: And running reconfigure again.gitlab: gitlab: For a comprehensive list of configuration options please see the Omnibus GitLab readmegitlab: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.mdgitlab: It looks like GitLab has not been configured yet; skipping the upgrade script. Verifying : openssh-6.6.1p1-35.el7_3.x86_64 1/6 Verifying : gitlab-ce-8.1.0+git.3217.3a100f1-rc1.ce.0.el7.x86_64 2/6 Verifying : openssh-server-6.6.1p1-35.el7_3.x86_64 3/6 Verifying : tcp_wrappers-libs-7.6-77.el7.x86_64 4/6 Verifying : fipscheck-lib-1.4.1-5.el7.x86_64 5/6 Verifying : fipscheck-1.4.1-5.el7.x86_64 6/6 Installed: gitlab-ce.x86_64 0:8.1.0+git.3217.3a100f1-rc1.ce.0.el7 Dependency Installed: fipscheck.x86_64 0:1.4.1-5.el7 fipscheck-lib.x86_64 0:1.4.1-5.el7 openssh.x86_64 0:6.6.1p1-35.el7_3 openssh-server.x86_64 0:6.6.1p1-35.el7_3 tcp_wrappers-libs.x86_64 0:7.6-77.el7 Complete![root@04cd1fb01fd3 /]#
How painful would it be to go back and sign these packages?
Very. We are talking about 10s if not 100s of packages (minimum of 4 per release times the number of releases we decide to go back to) that we would have to manually fetch, sign, delete from package server, and upload again. So I think we should not do it.
Is there any way to avoid this?
I am not sure we would want to avoid it. If we are deciding to sign packages from this moment(9.5) onwards, we'll have to have a breaking change. The way we could minimise it though is by backporting the signing change to last 4-5 releases so that next security releases get the signed packages. This way even when you are behind in upgrades you won't have to do any change, unless you are downgrading to a non-signed package.
Very painful. Doable, if we absolutely must, but I don't think it is worth more than the very last versions of every point release. Even then, it is painful. Having only to add a single flag to the command is not much of an impedance. Clearly documented, it will likely be welcomed as well.
The way we could minimise it though is by backporting the signing change to last 4-5 releases so that next security releases get the signed packages.
@briann This is what we are discussing right now. The reason I am a bit hesitant to include it for todays release is that if something does go wrong, we'll have either a major delay or even worse, users would not be able to upgrade.
But the situation is such that we wouldn't be able to do any different testing than what Jason already did.
Here's another trick point I would like to point out: If a user has already installed the repository, prior to adding the package signing key, they will not have the gpgcheck=1 setting on the RPM repository, so will not experience the behaviors above. This is on new repository additions only. They could, however, enable that via editing the appropriate repository definition for YUM, and likely should be instructed to.
The setting of gpgcheck=1 in the repository is done in the script based on the existence of the package signing key inside of PackageCloud
@WarheadsSE - when you say above that for Debian signing validation must be turned on globally, and that not all packages in the official repos are even signed, will that or could that affect other packages?
@marin and I had a quick call, I'm clear now, thanks.
The one question remaining that we should document and confirm is the behavior of the script to add our repository. If you have already added our repo earlier, but want to add RPM signing validation, is running that script again enough? Or do you have to manually remove the repo first then use the script to re-add?
@joshlambert Looks good. I think I would add either a line that says we will bring more info with the 9.5 post or even link to an issue so that users can ask questions or give us feedback.
During exploring / testing of key generation, @briann and I discussed the process to ensure everything is up to par when it comes to complexity and entropy.
What we've discussed is straight forward:
If using a VM, ensure that you're generating enough entropy
Activate haveged to increase available entropy in prng
dd if=/dev/vda of=/dev/null bs=1M & to generate disk activity & provide seed to the prng
make use of iperf3 or a very large download (say, docker pull gitlab/gitlab-ce:latest)
While the above are running, gpg --generate-key
If using physical hardware, ensure a hardware RNG is available, or do as above.
The keys should be RSA/RSA and 4096 bits.
The keys should have a secured, complex, 20+ character shell-compatible passphrase. Use 1Password for the generation of this, as a matter of process clarity and conformity.
On my laptop, I needed to activate the tpm-rng module in order to bring up the /dev/hwrng. Once this was in place, it was important to start rngd to feed /dev/random with the hwrng
Now for the question: What sort of expiration should we be setting? 2, 4, 10 years? Infinite until revoked? @briann@gitlab-build-team ... this will affect the use of the packages in the future, when the existing signed packages are "aged" but possibly still used by some customer needing support to help migrate.
It is possible to update the expiration date. So we can set an initial expiration of 3 years and then update the expiration if we decide not to rotate it at that point. I'm not sure how yum and apt deal with this.
Since we have complete control of the package server I don't mind using a longer expiration or infinite, as we'd still have the ability to revoke the key later.
To be triple sure, I went ahead and re-checked that CentOS can install a signed package directly, without erroring because the package is signed, but the key is not present
== this specifically removes the pubkey# rpm -e gpg-pubkey-c0207d21-57b5baaa# gpg --homedir /var/lib/yum/repos/x86_64/7/gitlab_test-signing/gpgdir --delete-key C0207D21== try running yum with `localinstall`# yum localinstall gitlab-ce-8.1.0+git.3217.3a100f1-rc1.ce.0.el7.x86_64.rpm...Total size: 1.0 GInstalled size: 1.0 GIs this ok [y/d/N]: yDownloading packages:Running transaction checkRunning transaction testTransaction test succeededRunning transaction Installing : gitlab-ce-8.1.0+git.3217.3a100f1-rc1.ce.0.el7.x86_64 1/1... Verifying : gitlab-ce-8.1.0+git.3217.3a100f1-rc1.ce.0.el7.x86_64 1/1 Installed: gitlab-ce.x86_64 0:8.1.0+git.3217.3a100f1-rc1.ce.0.el7 Complete!
Previously missing note: it was held from previous issuing upstream MR so that we could confirm there were no bugs with the rollout. All appearances say there have been no issues at all, and as such, I will now create an MR to upstream.