Improve logging to packagecloud push
What does this MR do?
Improve logging to packagecloud push to make it easier to understand what is going on.
Why was this MR needed?
When debugging #27470 (closed) it was hard to understand what is going on inside of the job log like https://gitlab.com/gitlab-org/gitlab-runner/-/jobs/974893081 it's not really clear what we are pushing for which distro.
Update to log the distro that we are going to push and also enable verbose mode to get more detail of what we are pushing.
What's the best way to test this MR?
-
Build runner binaries
make runner-bin
$ make runner-bin go get github.com/mitchellh/gox go: finding github.com/mitchellh/gox v1.0.1 go: downloading github.com/mitchellh/gox v1.0.1 go: extracting github.com/mitchellh/gox v1.0.1 go: downloading github.com/hashicorp/go-version v1.2.0 go: downloading github.com/mitchellh/iochan v1.0.0 go: extracting github.com/hashicorp/go-version v1.2.0 go: extracting github.com/mitchellh/iochan v1.0.0 go: finding github.com/mitchellh/iochan v1.0.0 go: finding github.com/hashicorp/go-version v1.2.0 # Building gitlab-runner in version 13.9.0~beta.53.gb28b1652 for -osarch 'darwin/amd64' -os 'linux' -os 'freebsd' -os 'windows' -arch '386' -arch 'arm' -arch 'amd64' -arch 'arm64' -arch 's390x' gox -osarch 'darwin/amd64' -os 'linux' -os 'freebsd' -os 'windows' -arch '386' -arch 'arm' -arch 'amd64' -arch 'arm64' -arch 's390x' \ -ldflags "-X gitlab.com/gitlab-org/gitlab-runner/common.NAME=gitlab-runner -X gitlab.com/gitlab-org/gitlab-runner/common.VERSION=13.9.0~beta.53.gb28b1652 -X gitlab.com/gitlab-org/gitlab-runner/common.REVISION=b28b1652 -X gitlab.com/gitlab-org/gitlab-runner/common.BUILT=2021-01-22T07:14:54+0000 -X gitlab.com/gitlab-org/gitlab-runner/common.BRANCH=27470-improve-packagecloud-logging -s -w" \ -output="out/binaries/gitlab-runner-{{.OS}}-{{.Arch}}" \ gitlab.com/gitlab-org/gitlab-runner Number of parallel builds: 5 --> windows/amd64: gitlab.com/gitlab-org/gitlab-runner --> linux/arm64: gitlab.com/gitlab-org/gitlab-runner --> freebsd/amd64: gitlab.com/gitlab-org/gitlab-runner --> freebsd/arm: gitlab.com/gitlab-org/gitlab-runner --> linux/s390x: gitlab.com/gitlab-org/gitlab-runner --> freebsd/386: gitlab.com/gitlab-org/gitlab-runner --> linux/386: gitlab.com/gitlab-org/gitlab-runner --> darwin/amd64: gitlab.com/gitlab-org/gitlab-runner --> linux/amd64: gitlab.com/gitlab-org/gitlab-runner --> linux/arm: gitlab.com/gitlab-org/gitlab-runner --> windows/386: gitlab.com/gitlab-org/gitlab-runner -
Build helper images
make helper-dockerarchive
$ make helper-dockerarchive go get github.com/mitchellh/gox gox -osarch=linux/amd64 -ldflags "-X gitlab.com/gitlab-org/gitlab-runner/common.NAME=gitlab-runner -X gitlab.com/gitlab-org/gitlab-runner/common.VERSION=13.9.0~beta.53.gb28b1652 -X gitlab.com/gitlab-org/gitlab-runner/common.REVISION=b28b1652 -X gitlab.com/gitlab-org/gitlab-runner/common.BUILT=2021-01-22T07:23:58+0000 -X gitlab.com/gitlab-org/gitlab-runner/common.BRANCH=27470-improve-packagecloud-logging -s -w" -output=out/binaries/gitlab-runner-helper/gitlab-runner-helper.x86_64 gitlab.com/gitlab-org/gitlab-runner/apps/gitlab-runner-helper Number of parallel builds: 7 --> linux/amd64: gitlab.com/gitlab-org/gitlab-runner/apps/gitlab-runner-helper Building helper image for: linux/amd64 docker-buildx Successfully created context "docker-buildx" buildx-builder [+] Building 13.6s (13/13) FINISHED => [internal] booting buildkit 4.0s => => pulling image moby/buildkit:buildx-stable-1 3.0s => => creating container buildx_buildkit_buildx-builder0 1.0s => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 917B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/alpine:3.12.0 2.9s => [1/7] FROM docker.io/library/alpine:3.12.0@sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321 0.7s => => resolve docker.io/library/alpine:3.12.0@sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321 0.0s => => sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c 2.80MB / 2.80MB 0.6s => => extracting sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c 0.1s => [internal] load build context 0.7s => => transferring context: 21.90MB 0.6s => [2/7] RUN apk add --no-cache bash ca-certificates dumb-init git git-lfs miniperl && ln -s miniperl /usr/bin/perl 3.5s => [3/7] RUN git lfs install --skip-repo 0.2s => [4/7] COPY ./helpers/entrypoint / 0.0s => [5/7] RUN chmod +x /entrypoint 0.2s => [6/7] COPY ./scripts/ ./binaries/gitlab-runner-helper /usr/bin/ 0.1s => [7/7] RUN echo 'hosts: files dns' >> /etc/nsswitch.conf 0.2s => exporting to client 1.4s => => sending tarball 1.4s xz -f -9 out/helper-images/prebuilt-x86_64.tar gox -osarch=linux/arm -ldflags "-X gitlab.com/gitlab-org/gitlab-runner/common.NAME=gitlab-runner -X gitlab.com/gitlab-org/gitlab-runner/common.VERSION=13.9.0~beta.53.gb28b1652 -X gitlab.com/gitlab-org/gitlab-runner/common.REVISION=b28b1652 -X gitlab.com/gitlab-org/gitlab-runner/common.BUILT=2021-01-22T07:23:58+0000 -X gitlab.com/gitlab-org/gitlab-runner/common.BRANCH=27470-improve-packagecloud-logging -s -w" -output=out/binaries/gitlab-runner-helper/gitlab-runner-helper.arm gitlab.com/gitlab-org/gitlab-runner/apps/gitlab-runner-helper Number of parallel builds: 7 --> linux/arm: gitlab.com/gitlab-org/gitlab-runner/apps/gitlab-runner-helper Building helper image for: linux/arm docker-buildx Successfully created context "docker-buildx" buildx-builder [+] Building 17.4s (13/13) FINISHED => [internal] booting buildkit 3.0s => => pulling image moby/buildkit:buildx-stable-1 2.1s => => creating container buildx_buildkit_buildx-builder0 0.9s => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 917B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/alpine:3.12.0 3.0s => [internal] load build context 0.6s => => transferring context: 18.22MB 0.6s => [1/7] FROM docker.io/library/alpine:3.12.0@sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321 0.7s => => resolve docker.io/library/alpine:3.12.0@sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321 0.0s => => sha256:52278dd8e57993669c5b72a9620e89bebdc098f2af2379caaa8945f7403f77a2 2.41MB / 2.41MB 0.5s => => extracting sha256:52278dd8e57993669c5b72a9620e89bebdc098f2af2379caaa8945f7403f77a2 0.1s => [2/7] RUN apk add --no-cache bash ca-certificates dumb-init git git-lfs miniperl && ln -s miniperl /usr/bin/perl 8.3s => [3/7] RUN git lfs install --skip-repo 0.6s => [4/7] COPY ./helpers/entrypoint / 0.0s => [5/7] RUN chmod +x /entrypoint 0.2s => [6/7] COPY ./scripts/ ./binaries/gitlab-runner-helper /usr/bin/ 0.1s => [7/7] RUN echo 'hosts: files dns' >> /etc/nsswitch.conf 0.2s => exporting to client 0.9s => => sending tarball 0.9s xz -f -9 out/helper-images/prebuilt-arm.tar gox -osarch=linux/arm64 -ldflags "-X gitlab.com/gitlab-org/gitlab-runner/common.NAME=gitlab-runner -X gitlab.com/gitlab-org/gitlab-runner/common.VERSION=13.9.0~beta.53.gb28b1652 -X gitlab.com/gitlab-org/gitlab-runner/common.REVISION=b28b1652 -X gitlab.com/gitlab-org/gitlab-runner/common.BUILT=2021-01-22T07:23:58+0000 -X gitlab.com/gitlab-org/gitlab-runner/common.BRANCH=27470-improve-packagecloud-logging -s -w" -output=out/binaries/gitlab-runner-helper/gitlab-runner-helper.arm64 gitlab.com/gitlab-org/gitlab-runner/apps/gitlab-runner-helper Number of parallel builds: 7 --> linux/arm64: gitlab.com/gitlab-org/gitlab-runner/apps/gitlab-runner-helper Building helper image for: linux/arm64 docker-buildx Successfully created context "docker-buildx" buildx-builder [+] Building 21.6s (13/13) FINISHED => [internal] booting buildkit 2.9s => => pulling image moby/buildkit:buildx-stable-1 2.4s => => creating container buildx_buildkit_buildx-builder0 0.5s => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 917B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/alpine:3.12.0 4.9s => [1/7] FROM docker.io/library/alpine:3.12.0@sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321 0.9s => => resolve docker.io/library/alpine:3.12.0@sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321 0.0s => => sha256:b538f80385f9b48122e3da068c932a96ea5018afa3c7be79da00437414bd18cd 2.71MB / 2.71MB 0.8s => => extracting sha256:b538f80385f9b48122e3da068c932a96ea5018afa3c7be79da00437414bd18cd 0.1s => [internal] load build context 0.7s => => transferring context: 20.26MB 0.7s => [2/7] RUN apk add --no-cache bash ca-certificates dumb-init git git-lfs miniperl && ln -s miniperl /usr/bin/perl 10.1s => [3/7] RUN git lfs install --skip-repo 0.6s => [4/7] COPY ./helpers/entrypoint / 0.0s => [5/7] RUN chmod +x /entrypoint 0.2s => [6/7] COPY ./scripts/ ./binaries/gitlab-runner-helper /usr/bin/ 0.1s => [7/7] RUN echo 'hosts: files dns' >> /etc/nsswitch.conf 0.2s => exporting to client 1.2s => => sending tarball 1.2s xz -f -9 out/helper-images/prebuilt-arm64.tar gox -osarch=linux/s390x -ldflags "-X gitlab.com/gitlab-org/gitlab-runner/common.NAME=gitlab-runner -X gitlab.com/gitlab-org/gitlab-runner/common.VERSION=13.9.0~beta.53.gb28b1652 -X gitlab.com/gitlab-org/gitlab-runner/common.REVISION=b28b1652 -X gitlab.com/gitlab-org/gitlab-runner/common.BUILT=2021-01-22T07:23:58+0000 -X gitlab.com/gitlab-org/gitlab-runner/common.BRANCH=27470-improve-packagecloud-logging -s -w" -output=out/binaries/gitlab-runner-helper/gitlab-runner-helper.s390x gitlab.com/gitlab-org/gitlab-runner/apps/gitlab-runner-helper Number of parallel builds: 7 --> linux/s390x: gitlab.com/gitlab-org/gitlab-runner/apps/gitlab-runner-helper Building helper image for: linux/s390x docker-buildx Successfully created context "docker-buildx" buildx-builder [+] Building 15.9s (13/13) FINISHED => [internal] booting buildkit 2.9s => => pulling image moby/buildkit:buildx-stable-1 2.3s => => creating container buildx_buildkit_buildx-builder0 0.7s => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 917B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/alpine:3.12.0 2.9s => [1/7] FROM docker.io/library/alpine:3.12.0@sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321 0.8s => => resolve docker.io/library/alpine:3.12.0@sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321 0.0s => => sha256:8fb3d41b2e9a59630b51745f257cd2561f96bcd15cf309fcc20120d5fcee8c5b 2.57MB / 2.57MB 0.6s => => extracting sha256:8fb3d41b2e9a59630b51745f257cd2561f96bcd15cf309fcc20120d5fcee8c5b 0.2s => [internal] load build context 0.7s => => transferring context: 21.31MB 0.7s => [2/7] RUN apk add --no-cache bash ca-certificates dumb-init git git-lfs miniperl && ln -s miniperl /usr/bin/perl 6.5s => [3/7] RUN git lfs install --skip-repo 0.5s => [4/7] COPY ./helpers/entrypoint / 0.1s => [5/7] RUN chmod +x /entrypoint 0.2s => [6/7] COPY ./scripts/ ./binaries/gitlab-runner-helper /usr/bin/ 0.1s => [7/7] RUN echo 'hosts: files dns' >> /etc/nsswitch.conf 0.2s => exporting to client 1.3s => => sending tarball 1.3s xz -f -9 out/helper-images/prebuilt-s390x.tar -
Start a new shell inside of the CI image with
docker run --rm -it -v $PWD:/code -w /code registry.gitlab.com/gitlab-org/gitlab-runner/ci:1.13.8-9 -
Build rpm packages
make package-rpm
$ make package-rpm # Installing packaging dependencies... command -v fpm 1>/dev/null && fpm --help 1>/dev/null || gem install rake fpm:1.10.2 --no-document Doing `require 'backports'` is deprecated and will not load any backport in the next major release. Require just the needed backports instead, or 'backports/latest'. chmod 755 packaging/root/usr/share/gitlab-runner/ chmod 755 packaging/root/usr/share/gitlab-runner/* # Building RedHat compatible packages... make package-rpm-arch ARCH=amd64 PACKAGE_ARCH=amd64 make[1]: Entering directory '/code' Doing `require 'backports'` is deprecated and will not load any backport in the next major release. Require just the needed backports instead, or 'backports/latest'. Invalid package configuration: Cannot package the path 'out/helper-images', does it exist? {:level=>:error} make[1]: *** [Makefile.package.mk:48: package-rpm-arch] Error 1 make[1]: Leaving directory '/code' make: *** [Makefile.package.mk:28: package-rpm] Error 2 root@b995388f4781:/code# make helper-image make: *** No rule to make target 'helper-image'. Stop. root@b995388f4781:/code# make helper-image make: *** No rule to make target 'helper-image'. Stop. root@b995388f4781:/code# ^C root@b995388f4781:/code# make package-rpm # Installing packaging dependencies... command -v fpm 1>/dev/null && fpm --help 1>/dev/null || gem install rake fpm:1.10.2 --no-document Doing `require 'backports'` is deprecated and will not load any backport in the next major release. Require just the needed backports instead, or 'backports/latest'. chmod 755 packaging/root/usr/share/gitlab-runner/ chmod 755 packaging/root/usr/share/gitlab-runner/* # Building RedHat compatible packages... make package-rpm-arch ARCH=amd64 PACKAGE_ARCH=amd64 make[1]: Entering directory '/code' Doing `require 'backports'` is deprecated and will not load any backport in the next major release. Require just the needed backports instead, or 'backports/latest'. /var/lib/gems/2.5.0/gems/fpm-1.10.2/lib/fpm/package/rpm.rb:424: warning: Insecure world writable dir /go/bin in PATH, mode 040777 Created package {:path=>"out/rpm/gitlab-runner_amd64.rpm"} make[1]: Leaving directory '/code' make package-rpm-arch ARCH=386 PACKAGE_ARCH=i686 make[1]: Entering directory '/code' Doing `require 'backports'` is deprecated and will not load any backport in the next major release. Require just the needed backports instead, or 'backports/latest'. /var/lib/gems/2.5.0/gems/fpm-1.10.2/lib/fpm/package/rpm.rb:424: warning: Insecure world writable dir /go/bin in PATH, mode 040777 Created package {:path=>"out/rpm/gitlab-runner_i686.rpm"} make[1]: Leaving directory '/code' make package-rpm-arch ARCH=arm64 PACKAGE_ARCH=aarch64 make[1]: Entering directory '/code' Doing `require 'backports'` is deprecated and will not load any backport in the next major release. Require just the needed backports instead, or 'backports/latest'. /var/lib/gems/2.5.0/gems/fpm-1.10.2/lib/fpm/package/rpm.rb:424: warning: Insecure world writable dir /go/bin in PATH, mode 040777 Created package {:path=>"out/rpm/gitlab-runner_aarch64.rpm"} make[1]: Leaving directory '/code' make package-rpm-arch ARCH=arm64 PACKAGE_ARCH=arm64 make[1]: Entering directory '/code' Doing `require 'backports'` is deprecated and will not load any backport in the next major release. Require just the needed backports instead, or 'backports/latest'. /var/lib/gems/2.5.0/gems/fpm-1.10.2/lib/fpm/package/rpm.rb:424: warning: Insecure world writable dir /go/bin in PATH, mode 040777 Created package {:path=>"out/rpm/gitlab-runner_arm64.rpm"} make[1]: Leaving directory '/code' make package-rpm-arch ARCH=arm PACKAGE_ARCH=arm make[1]: Entering directory '/code' Doing `require 'backports'` is deprecated and will not load any backport in the next major release. Require just the needed backports instead, or 'backports/latest'. /var/lib/gems/2.5.0/gems/fpm-1.10.2/lib/fpm/package/rpm.rb:424: warning: Insecure world writable dir /go/bin in PATH, mode 040777 Created package {:path=>"out/rpm/gitlab-runner_arm.rpm"} make[1]: Leaving directory '/code' make package-rpm-arch ARCH=arm PACKAGE_ARCH=armhf make[1]: Entering directory '/code' Doing `require 'backports'` is deprecated and will not load any backport in the next major release. Require just the needed backports instead, or 'backports/latest'. /var/lib/gems/2.5.0/gems/fpm-1.10.2/lib/fpm/package/rpm.rb:424: warning: Insecure world writable dir /go/bin in PATH, mode 040777 Created package {:path=>"out/rpm/gitlab-runner_armhf.rpm"} make[1]: Leaving directory '/code' make package-rpm-arch ARCH=s390x PACKAGE_ARCH=s390x make[1]: Entering directory '/code' Doing `require 'backports'` is deprecated and will not load any backport in the next major release. Require just the needed backports instead, or 'backports/latest'. /var/lib/gems/2.5.0/gems/fpm-1.10.2/lib/fpm/package/rpm.rb:424: warning: Insecure world writable dir /go/bin in PATH, mode 040777 Created package {:path=>"out/rpm/gitlab-runner_s390x.rpm"} make[1]: Leaving directory '/code' -
Update
packagecloud-rpmcommand to and appendechoto thepackage_cloud pushcommandgit diff
diff --git a/Makefile b/Makefile index e3b443ca4..ab9313df2 100644 --- a/Makefile +++ b/Makefile @@ -245,7 +245,7 @@ packagecloud-rpm: echo "===================="; \ echo "$$DIST"; \ echo "====================" ;\ - package_cloud push --verbose --url $(PACKAGE_CLOUD_URL) $(PACKAGE_CLOUD)/$$DIST out/rpm/*.rpm; \ + echo package_cloud push --verbose --url $(PACKAGE_CLOUD_URL) $(PACKAGE_CLOUD)/$$DIST out/rpm/*.rpm; \ done -
Run
packagecloud-rpmcommand and you should see the same logs as belowmake packagecloud-rpm
$ make packagecloud-rpm # Sending RedHat compatible packages... for DIST in el/6 el/7 el/8 ol/6 ol/7 fedora/30; do \ echo "===================="; \ echo "$DIST"; \ echo "====================" ;\ echo package_cloud push --verbose --url https://packagecloud.io/ ayufan/gitlab-ci-multi-runner/$DIST out/rpm/*.rpm; \ done ==================== el/6 ==================== package_cloud push --verbose --url https://packagecloud.io/ ayufan/gitlab-ci-multi-runner/el/6 out/rpm/gitlab-runner_aarch64.rpm out/rpm/gitlab-runner_amd64.rpm out/rpm/gitlab-runner_arm.rpm out/rpm/gitlab-runner_arm64.rpm out/rpm/gitlab-runner_armhf.rpm out/rpm/gitlab-runner_i686.rpm out/rpm/gitlab-runner_s390x.rpm ==================== el/7 ==================== package_cloud push --verbose --url https://packagecloud.io/ ayufan/gitlab-ci-multi-runner/el/7 out/rpm/gitlab-runner_aarch64.rpm out/rpm/gitlab-runner_amd64.rpm out/rpm/gitlab-runner_arm.rpm out/rpm/gitlab-runner_arm64.rpm out/rpm/gitlab-runner_armhf.rpm out/rpm/gitlab-runner_i686.rpm out/rpm/gitlab-runner_s390x.rpm ==================== el/8 ==================== package_cloud push --verbose --url https://packagecloud.io/ ayufan/gitlab-ci-multi-runner/el/8 out/rpm/gitlab-runner_aarch64.rpm out/rpm/gitlab-runner_amd64.rpm out/rpm/gitlab-runner_arm.rpm out/rpm/gitlab-runner_arm64.rpm out/rpm/gitlab-runner_armhf.rpm out/rpm/gitlab-runner_i686.rpm out/rpm/gitlab-runner_s390x.rpm ==================== ol/6 ==================== package_cloud push --verbose --url https://packagecloud.io/ ayufan/gitlab-ci-multi-runner/ol/6 out/rpm/gitlab-runner_aarch64.rpm out/rpm/gitlab-runner_amd64.rpm out/rpm/gitlab-runner_arm.rpm out/rpm/gitlab-runner_arm64.rpm out/rpm/gitlab-runner_armhf.rpm out/rpm/gitlab-runner_i686.rpm out/rpm/gitlab-runner_s390x.rpm ==================== ol/7 ==================== package_cloud push --verbose --url https://packagecloud.io/ ayufan/gitlab-ci-multi-runner/ol/7 out/rpm/gitlab-runner_aarch64.rpm out/rpm/gitlab-runner_amd64.rpm out/rpm/gitlab-runner_arm.rpm out/rpm/gitlab-runner_arm64.rpm out/rpm/gitlab-runner_armhf.rpm out/rpm/gitlab-runner_i686.rpm out/rpm/gitlab-runner_s390x.rpm ==================== fedora/30 ==================== package_cloud push --verbose --url https://packagecloud.io/ ayufan/gitlab-ci-multi-runner/fedora/30 out/rpm/gitlab-runner_aarch64.rpm out/rpm/gitlab-runner_amd64.rpm out/rpm/gitlab-runner_arm.rpm out/rpm/gitlab-runner_arm64.rpm out/rpm/gitlab-runner_armhf.rpm out/rpm/gitlab-runner_i686.rpm out/rpm/gitlab-runner_s390x.rpm
What are the relevant issue numbers?
Closes #27470 (closed)
Edited by Steve Xuereb