`license_finder` fails with `trash.lock` not found when building `registry`
Summary
Building an RPM in a Centos 7 VM fails at the registry
build stage during the license_finder
command.
Steps to reproduce
Use this bootstrap.sh script in the Vagrant Centos 7 VM:
#!/bin/bash
# Update the base system and install dependencies
yum -y update
yum -y install epel-release
yum -y install \
autoconf \
automake \
bison \
bzip2 \
ca-certificates \
cmake \
curl \
gcc-c++ \
git-core \
libffi-devel \
libtool \
libyaml-devel \
make \
openssl-devel \
patch \
policycoreutils-python \
readline \
readline-devel \
rpm-build \
sqlite-devel \
wget \
zlib \
zlib-devel
yum -y remove git.x86_64
yum -y install https://centos7.iuscommunity.org/ius-release.rpm
yum -y install git2u-all
# Install Go
GO_VERSION=$(cat /vagrant/config/go-version)
wget https://dl.google.com/go/go${GO_VERSION}.linux-amd64.tar.gz
tar -xzf go${GO_VERSION}.linux-amd64.tar.gz
mv go /usr/local/
echo 'export GOROOT="/usr/local/go"' >> ~/.bash_profile
echo 'export GOPATH="$HOME/GoProjects/"' >> ~/.bash_profile
echo 'export PATH="$GOPATH/bin:$GOROOT/bin:$PATH"' >> ~/.bash_profile
source ~/.bash_profile
# Install Ruby
git clone https://github.com/rbenv/rbenv.git ~/.rbenv
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bash_profile
echo 'eval "$(rbenv init -)"' >> ~/.bash_profile
git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build
echo 'export PATH="$HOME/.rbenv/plugins/ruby-build/bin:$PATH"' >> ~/.bash_profile
source ~/.bash_profile
RUBY_VERSION=$(cat /vagrant/config/ruby-version)
rbenv install ${RUBY_VERSION}
rbenv rehash
rbenv global ${RUBY_VERSION}
echo "gem: --no-ri --no-rdoc" > ~/.gemrc
gem install bundler
gem install license_finder
rbenv rehash
# Install Node and Yarn
NODE_VERSION=$(cat /vagrant/config/node-version)
curl -sL https://rpm.nodesource.com/setup_${NODE_VERSION}.x | bash -
yum -y install nodejs
curl -sL https://dl.yarnpkg.com/rpm/yarn.repo | tee /etc/yum.repos.d/yarn.repo
rpm --import https://dl.yarnpkg.com/rpm/pubkey.gpg
yum -y install yarn
# Build GitLab
export COMPILE_ASSETS=true
export ALTERNATIVE_SOURCES=true
export ee=true
mkdir -p /opt/src
pushd /opt/src
git clone https://gitlab.com/gitlab-org/omnibus-gitlab.git
pushd omnibus-gitlab
# Set the Ruby Version
cp /vagrant/config/ruby-version ./.ruby-version
# Change the GitLab source to ECP-CI and set to the develop branch
sed -i 's/gitlab-org\/gitlab.git/ecp-ci\/gitlab.git/' ./.custom_sources.yml
cp /vagrant/config/VERSION ./VERSION
# The upstream source repo often breaks on definitions:
# https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master/config/software
# and takes a few days to update. So pull in manual fixes here
if [ "$(cat /vagrant/config/software-override)" = "true" ]; then
cp /vagrant/config/software/* /opt/src/omnibus-gitlab/config/software/
fi
bundle install --path .bundle --binstubs
bin/omnibus build gitlab
cp pkg/gitlab-*.el7.x86_64.rpm /vagrant/rpms/
popd
popd
What is the current bug behavior?
The build fails with the following error:
/root/.rbenv/versions/2.6.5/lib/ruby/2.6.0/psych.rb:577:in `initialize': No such file or directory @ rb_sysopen - /var/cache/omnibus/src/registry/src/github.com/docker/distribution/trash.lock (Errno::ENOENT)
and the full traceback is:
gitlab-build: [Software: registry] I | 2020-03-02T16:49:35+00:00 | Building because `zlib' dirtied the cache
gitlab-build: [GitFetcher: registry] I | 2020-03-02T16:49:35+00:00 | Cleaning existing clone
gitlab-build: [Builder: registry] I | 2020-03-02T16:49:36+00:00 | Starting build
gitlab-build: [Builder: registry] I | 2020-03-02T16:49:36+00:00 | Environment:
gitlab-build: [Builder: registry] I | 2020-03-02T16:49:36+00:00 | BUILDTAGS="include_gcs include_oss"
gitlab-build: [Builder: registry] I | 2020-03-02T16:49:36+00:00 | GOPATH="/var/cache/omnibus/src/registry"
gitlab-build: [Builder: registry] I | 2020-03-02T16:49:36+00:00 | MAKE="gmake"
gitlab-build: [Builder: registry] I | 2020-03-02T16:49:36+00:00 | $ gmake build
gitlab-build: [Builder: registry] I | 2020-03-02T16:49:49+00:00 | Execute: `gmake build': 13.1334s
gitlab-build: [Builder: registry] I | 2020-03-02T16:49:49+00:00 | Environment:
gitlab-build: [Builder: registry] I | 2020-03-02T16:49:49+00:00 | BUILDTAGS="include_gcs include_oss"
gitlab-build: [Builder: registry] I | 2020-03-02T16:49:49+00:00 | GOPATH="/var/cache/omnibus/src/registry"
gitlab-build: [Builder: registry] I | 2020-03-02T16:49:49+00:00 | MAKE="gmake"
gitlab-build: [Builder: registry] I | 2020-03-02T16:49:49+00:00 | $ gmake binaries
gitlab-build: [Builder: registry] I | 2020-03-02T16:49:51+00:00 | Execute: `gmake binaries': 2.0439s
gitlab-build: [Builder: registry] I | 2020-03-02T16:49:51+00:00 | move `/var/cache/omnibus/src/registry/src/github.com/docker/distribution/bin/*' to `/opt/gitlab/embedded/bin': 0.0011s
gitlab-build: [Builder: registry] I | 2020-03-02T16:49:51+00:00 | $ license_finder report --decisions-file=/opt/src/omnibus-gitlab/support/dependency_decisions.yml --format=csv --save=license.csv
gitlab-build: [Builder: registry] W | 2020-03-02T16:49:51+00:00 | [1/2] Failed to execute command. Retrying in 10 seconds...
gitlab-build: [Builder: registry] I | 2020-03-02T16:50:01+00:00 | $ license_finder report --decisions-file=/opt/src/omnibus-gitlab/support/dependency_decisions.yml --format=csv --save=license.csv
gitlab-build: [Builder: registry] W | 2020-03-02T16:50:02+00:00 | [2/2] Failed to execute command. Retrying in 20 seconds...
gitlab-build: [Builder: registry] I | 2020-03-02T16:50:22+00:00 | $ license_finder report --decisions-file=/opt/src/omnibus-gitlab/support/dependency_decisions.yml --format=csv --save=license.csv
gitlab-build: [Builder: registry] I | 2020-03-02T16:50:22+00:00 | Execute: `license_finder report --decisions-file=/opt/src/omnibus-gitlab/support/dependency_decisions.yml --format=csv --save=license.csv': 31.2025s
gitlab-build: [Builder: registry] I | 2020-03-02T16:50:22+00:00 | Build registry: 46.3833s
gitlab-build: The following shell command exited with status 1:
gitlab-build:
gitlab-build: $ license_finder report --decisions-file=/opt/src/omnibus-gitlab/support/dependency_decisions.yml --format=csv --save=license.csv
gitlab-build:
gitlab-build: Output:
gitlab-build:
gitlab-build: LicenseFinder::Go15VendorExperiment: is active
gitlab-build: LicenseFinder::Trash: is active
gitlab-build:
gitlab-build: Error:
gitlab-build:
gitlab-build: /root/.rbenv/versions/2.6.5/lib/ruby/2.6.0/psych.rb:577:in `initialize': No such file or directory @ rb_sysopen - /var/cache/omnibus/src/registry/src/github.com/docker/distribution/trash.lock (Errno::ENOENT)
gitlab-build: from /root/.rbenv/versions/2.6.5/lib/ruby/2.6.0/psych.rb:577:in `open'
gitlab-build: from /root/.rbenv/versions/2.6.5/lib/ruby/2.6.0/psych.rb:577:in `load_file'
gitlab-build: from /root/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/license_finder-6.1.0/lib/license_finder/package_managers/trash.rb:26:in `current_packages'
gitlab-build: from /root/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/license_finder-6.1.0/lib/license_finder/package_manager.rb:101:in `current_packages_with_relations'
gitlab-build: from /root/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/license_finder-6.1.0/lib/license_finder/scanner.rb:36:in `each'
gitlab-build: from /root/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/license_finder-6.1.0/lib/license_finder/scanner.rb:36:in `flat_map'
gitlab-build: from /root/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/license_finder-6.1.0/lib/license_finder/scanner.rb:36:in `active_packages'
gitlab-build: from /root/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/license_finder-6.1.0/lib/license_finder/core.rb:83:in `current_packages'
gitlab-build: from /root/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/license_finder-6.1.0/lib/license_finder/core.rb:78:in `decision_applier'
gitlab-build: from /root/.rbenv/versions/2.6.5/lib/ruby/2.6.0/forwardable.rb:224:in `acknowledged'
gitlab-build: from /root/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/license_finder-6.1.0/lib/license_finder/license_aggregator.rb:51:in `block in aggregate_packages'
gitlab-build: from /root/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/license_finder-6.1.0/lib/license_finder/license_aggregator.rb:49:in `each'
gitlab-build: from /root/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/license_finder-6.1.0/lib/license_finder/license_aggregator.rb:49:in `flat_map'
gitlab-build: from /root/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/license_finder-6.1.0/lib/license_finder/license_aggregator.rb:49:in `aggregate_packages'
gitlab-build: from /root/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/license_finder-6.1.0/lib/license_finder/license_aggregator.rb:11:in `dependencies'
gitlab-build: from /root/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/license_finder-6.1.0/lib/license_finder/cli/main.rb:147:in `report'
gitlab-build: from /root/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
gitlab-build: from /root/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
gitlab-build: from /root/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
gitlab-build: from /root/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thor-1.0.1/lib/thor/base.rb:485:in `start'
gitlab-build: from /root/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/license_finder-6.1.0/bin/license_finder:6:in `<top (required)>'
gitlab-build: from /root/.rbenv/versions/2.6.5/bin/license_finder:23:in `load'
gitlab-build: from /root/.rbenv/versions/2.6.5/bin/license_finder:23:in `<main>'
What is the expected correct behavior?
A successful build is expected. When I go into config/software/registry.rb
and comment out the license_finder
commands, the build completes successfully. Further, this workflow worked in the past. The last successful RPM was built some time in December 2019, so somewhere in the registry version bump from v2.7.4 to the current v2.8.1 (commit history), something changed, perhaps some new dependency or gem requirement? And as a final note, when I look at Docker Distribution upstream, I do not see a trash.lock
file, nor do I see a vendor.conf
file which is used by license_finder
to activate the service which looks for trash.lock
.
For our testing purposes, I'd like to keep using VMs instead of containers, so any help is greatly appreciated. Thanks.