Building runner helper images with Windows nanoserver
What does this MR do?
This builds on previous work, such as supporting pwsh
(PowerShell v7), to experiment with using a helper image built with Windows NanoServer.
It builds both the Server Core (unchanged from before) and Nano Server Helper Images, with the latter not being built for official releases just yet (for some hands-on trials).
Why was this MR needed?
The benefits of basing the helper image off Nano Server are reducing the image size and potentially improving job runtimes
Image Size
The helper image built from Nano Server is almost 10x smaller than Server Core
REPOSITORY TAG IMAGE ID CREATED SIZE
gitlab-runner-helper x86_64-76c0cffb-nanoserver20H2 05fdc7e2f5cf 7 weeks ago 633MB
gitlab-runner-helper x86_64-76c0cffb-servercore20H2 ba555b9a56d0 3 months ago 6.2GB
The smaller size also is a benefit when downloading the helper image for the first time. It can take 10-15 minutes to pull and decompress the Server Core image layers from Microsoft, whereas Nano Server is a few minutes.
What's the best way to test this MR?
Demonstration
https://gitlab.com/82phil/nano_runner_helper_demo
Nano Server: https://gitlab.com/82phil/nano_runner_helper_demo/-/jobs/3960482644
Server Core: https://gitlab.com/82phil/nano_runner_helper_demo/-/jobs/3491388095
Building the Image
I tested this on Windows 10 Pro on 21H1 (19043) with the following
- Docker Desktop 4.8.2
- Git for Windows
Pull down the gitlab-runner repo, change to this branch.
Add the gitlab-runner-helper executable under "..\out\binaries\gitlab-runner-helper\gitlab-runner-helper.x86_64-windows.exe" I pulled it from the 15.0 release artifacts CI - https://gitlab.com/gitlab-org/gitlab-runner/-/jobs/2481477621/artifacts/file/out/binaries/gitlab-runner-helper/gitlab-runner-helper.x86_64-windows.exe
Open a PowerShell terminal and cd to the gitlab-runner directory, execute the following:
$env:WINDOWS_VERSION = "nanoserver20H2"
$env:GIT_VERSION = "2.39.2"
$env:GIT_VERSION_BUILD = "1"
$env:GIT_WINDOWS_AMD64_CHECKSUM = "a53b90a42d9a5e3ac992f525b5805c4dbb8a013b09a32edfdcf9a551fd8cfe2d"
$env:GIT_LFS_VERSION = "3.2.0"
$env:GIT_LFS_WINDOWS_AMD64_CHECKSUM = "c2ee1f7b22d98f614cab94e1033052143f4dbf1207c09ce57e9390acc4bbf86e"
$env:PWSH_WINDOWS_AMD64_CHECKSUM = "D234CFD6E9DD4A2CA68BFBC64C416DE42F31760E954812E8ADE02B6D57FE243F"
$env:PWSH_VERSION = "7.1.1"
$env:SKIP_CLEANUP = "true"
$env:PUSH_TO_ECR_PUBLIC = "false"
$env:PUBLISH_IMAGES = "false"
$env:CI_REGISTRY_IMAGE = "test"
I ended up using 20H2 as 21H1 will change to ltsc2022 in the build script and fail to build the image. 20H2 will work with Docker Desktop as it runs in Hyper-V isolation by default, supporting images <= 21H1. If anyone else has issues this may help: https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/version-compatibility?tabs=windows-server-2022%2Cwindows-11-21H2
Run the CI script
.\ci\build_release_windows_images.ps1
Note the image that was built.
Test the runner helper
Create a new GitLab project
Setup and register a gitlab-runner, configuring it with the docker-windows
executor. Under config.toml
add the following options
[runners.docker]
image = "mcr.microsoft.com/powershell:windowsservercore-20h2"
pull_policy = "never"
helper_image = "test/gitlab-runner-helper:x86_64-54745ef6-nanoserver20H2"
Note the helper image will match the one previously built.
Since the pull policy was disabled to demonstrate the helper image, issue the following in a console to pull down the Server Core PowerShell image so it's available when running the demonstration.
docker pull mcr.microsoft.com/powershell:windowsservercore-20h2
Add a .gitlab-ci.yml
to the project with the following
stages:
- test runner helper
test:
stage: test runner helper
tags:
- <your_runner_tag>
script:
- pwsh --version
After committing the file the CI job should execute.
What are the relevant issue numbers?
Merge request reports
Activity
Thank you for your contribution to GitLab. We believe that everyone can contribute and contributions like yours are what make GitLab great!
Some contributions require several iterations of review and we try to mentor contributors during this process. However, we understand that some reviews can be very time consuming. If you would prefer for us to continue the work you've submitted now or at any point in the future please let us know.
If you're okay with being part of our review process (and we hope you are!), there are several initial checks we ask you to make:
- The merge request description clearly explains:
- The problem being solved.
- The best way a reviewer can test your changes (is it possible to provide an example?).
- If the pipeline failed, do you need help identifying what failed?
- Check that Go code follows our Go guidelines.
- Read our contributing to GitLab Runner document.
This message was generated automatically. You're welcome to improve it.
- The merge request description clearly explains:
added Community contribution label
assigned to @82phil
added 1 commit
- 76c0cffb - Fixed nanoserver Dockerfile, updated build script to have the pwsh image in lowercase
added [Deprecated] Category:Runner devopsverify grouprunner labels
added 1st contribution label
added sectionops label
added 51 commits
-
76c0cffb...293a8895 - 49 commits from branch
gitlab-org:main
- 3eb217c3 - Merge remote-tracking branch 'origin/main' into runner-windows-nanoserver-images
- b0a32886 - Updating pre-release for windows helper image builds for dev and bleeding edge
-
76c0cffb...293a8895 - 49 commits from branch
mentioned in issue gitlab-org/quality/triage-reports#7744 (closed)
added maintenancepipelines workflowready for review labels
added typemaintenance label
mentioned in issue gitlab-org/quality/triage-reports#7819 (closed)
mentioned in issue gitlab-org/quality/triage-reports#7957 (closed)
mentioned in issue gitlab-org/quality/triage-reports#8042 (closed)
@ajwalker Could you please review? Thanks!
requested review from @ajwalker
@ajwalker, this Community contribution was recently assigned to you for review.
- Do you still have capacity to review this? We are mindful of your time, so if you are not able to take this on, please re-assign to one or more other reviewers.
- Add the workflowin dev label if the merge request needs action from the author.
added automation:reviewers-reminded label
mentioned in issue gitlab-org/quality/triage-reports#8224 (closed)
mentioned in issue gitlab-org/quality/triage-reports#8327 (closed)
added 134 commits
-
b0a32886...9bb7f5f8 - 132 commits from branch
gitlab-org:main
- 8108dca3 - Merge remote-tracking branch 'origin/main' into runner-windows-nanoserver-images
- a2dbcb5c - Updated with changes from main branch
-
b0a32886...9bb7f5f8 - 132 commits from branch
mentioned in issue gitlab-org/quality/triage-reports#8496 (closed)
mentioned in issue gitlab-org/quality/triage-reports#8560 (closed)
mentioned in issue gitlab-org/quality/triage-reports#8707 (closed)
added idle label
mentioned in issue gitlab-org/quality/triage-reports#8779 (closed)
mentioned in issue gitlab-org/quality/triage-reports#8843 (closed)
@82phil , I want to help you and our wider community to move this forward. Can you work on the feedback provided? Also, the less thinking on the conceptual phase there is, the easier this review will go and hence, we increase our chances towards merge success together. I suggest you answer the following questions in the summary of the issue at the top.- Runner is a complicated product, and has a lot of parameters that can make it fairly complex to test all functionality. Given that GitLab maintains this project we require all changes to have tests for the change that they bring. Please see what we can do to be able to test this automatically in some form or another. See .gitlab/ci/test.gitlab-ci.yml for examples on other tests suites specific to windows.
- add more details what the impact, both positive & negative are from this change to end users. The description as is doesn't add too much of the why / usefulness. That doesn't mean it isn't, it just isn't clear enough yet.
- think of a changelog record, try to describe what the value is this brings to end users
- rebase & resolve potential merge conflicts
Thanks!
@nick_vh - Thanks for providing feedback. I have clarified the PR's intention. I hope this hits on some of the points you brought up. I will work on pulling in changes from the main branch soon.
added needs tests workflowin dev labels and removed workflowready for review label
removed automation:reviewers-reminded label
removed idle label
removed [Deprecated] Category:Runner label
added 131 commits
-
a2dbcb5c...1b8d5e38 - 130 commits from branch
gitlab-org:main
- 17ebab0b - Merge remote-tracking branch 'upstream/main' into runner-windows-nanoserver-images
-
a2dbcb5c...1b8d5e38 - 130 commits from branch
added 1 commit
- 5f8e77ac - Updating prerelease CI to also not build nanoserver images for > win1809
@82phil, it seems we're waiting on an action from you for approximately two weeks.
- Do you still have capacity to work on this? If not, you might want to close this MR and/or ask someone to take over.
- Do you need help in getting it ready? At any time, you can ask for help with
@gitlab-bot help
. - If you're actually ready for a review, you can post
@gitlab-bot ready
.
Could use @gitlab-bot help in figuring out what broke the CI. Otherwise, when I find the time, I will take a good look at the logs and go from there.
@82phil you need to write
@gitlab-bot help
as a separate line. It doesn't really detect it in a sentence. Apologies for that!@gitlab-bot help
added automation:author-reminded label
added idle label
removed needs tests label
added 303 commits
-
5f8e77ac...90fed0b3 - 302 commits from branch
gitlab-org:main
- d7709e41 - Merge remote-tracking branch 'upstream/main' into runner-windows-nanoserver-images
-
5f8e77ac...90fed0b3 - 302 commits from branch
removed idle label
@ajwalker - MinGit 2.38.x is broken for nanoserver, this explains why the pipeline failed when building the nanoserver containers. See https://github.com/git-for-windows/git/issues/4052. I confirmed it builds with the RC candidates for 2.39.x.
Edited by Hoff_IOI'll put your MR back in review, as it looks like you found a way to unblock this. @ajwalker could you offer your guidance here and/or assign another reviewer if you would not have the capacity for it?
I went ahead and backported Git to 2.37.3 so the CI should succeed. If everything else looks good, then it's just a question of whether it's okay to merge it in with the older version of Git for now until 2.39 releases or to hold off until Git for Windows cuts an official 2.39 release.
@82phil Thank you for figuring out what the problem was here and for contributing this.
The nanoserver helper images are the direction we need to go in, given the size of servercore, so I'm really happy to see this being pushed forward!
I'm wondering if we should wait for 2.39, as it doesn't look like it's far off until done? Then we can target this for %15.8
@ajwalker - 2.39 released a few days ago. I got a chance to test it out. Looks good, however I did encounter the Git safe directory issue discussed here (#29022 (closed)). However, I did not try the latest runner to see if it was resolved there, so instead I set
safe.directory
to*
in the GitLab runner config as a workaround (nano_runner_helper_demo job: 3491389193).edit: And here is the failed job showing the safe directory issue: https://gitlab.com/82phil/nano_runner_helper_demo/-/jobs/3491384304
Edited by Hoff_IO@82phil Sorry for the delay on this. Thank you for updating and for highlighting the
safe.directory
issue.I'm going to run the pipeline again and if everything passes, we can get this merged
@ajwalker ping to get this back in your queue. Can you take a look?
@82phil It looks like this was failing CI because it was attempting to push development images to Docker Hub, which we don't do (environment to push doesn't have credentials). Can you apply the suggestions and this should complete the pipeline successfully
@82phil I think we're also be okay to add the
stable
release of these.In a follow-up, we can select the image by using the
image_flavor
config setting. As long as they're not used as a default image yet, I think we'll be okay to release.@ajwalker Changes are in and fingers crossed
Looking at the
TestInteractiveTerminal
test, I'm not sure that the changes here caused this error https://gitlab.com/gitlab-org/gitlab-runner/-/jobs/3907551818#L5615It's been noted to have been a flaky test in the past, see #27579 (closed) and !3042 (comment 631678641)
I brought in the latest changes from
main
and updated the bleeding edge prerelease ci builds to match the changes made to servercore frommain
. @ajwalker thanks for your help, please give it a go again.@82phil Pipeline looks good
Thank you for working on this (and persevering with our flaky pipelines
- I'm spending this week trying to fix these issues!).Adding to merge train
. It'll be great to move over to this image in the near future!🥳 Thank you @ajwalker for your time and help on this!
added 1 commit
- 1966a3c3 - Aligning nanoserver dockerfile with servercore
Updated env variables to match current CI with the exception of Git being bumped to the release candidate.
$env:WINDOWS_VERSION = "nanoserver20H2" $env:GIT_VERSION = "2.39.0-rc1" $env:GIT_VERSION_BUILD = "1" $env:GIT_WINDOWS_AMD64_CHECKSUM = "d0bf3d86da0571cfc99b39d9b06c1d58136870e0dde0235db6b3f22685633072" $env:GIT_LFS_VERSION = "3.2.0" $env:GIT_LFS_WINDOWS_AMD64_CHECKSUM = "c2ee1f7b22d98f614cab94e1033052143f4dbf1207c09ce57e9390acc4bbf86e" $env:PWSH_WINDOWS_AMD64_CHECKSUM = "D234CFD6E9DD4A2CA68BFBC64C416DE42F31760E954812E8ADE02B6D57FE243F" $env:PWSH_VERSION = "7.1.1" $env:SKIP_CLEANUP = "true" $env:PUSH_TO_ECR_PUBLIC = "false" $env:PUBLISH_IMAGES = "false" $env:CI_REGISTRY_IMAGE = "test"
removed automation:author-reminded label
added workflowready for review label and removed workflowin dev label
@ajwalker
, this Community contribution is ready for review.- Do you have capacity and domain expertise to review this? We are mindful of your time, so if you are not able to take this on, please re-assign to one or more other reviewers.
- Add the workflowin dev label if the merge request needs action from the author. This message was generated automatically. You're welcome to improve it.
added 1 commit
- 0dcc662c - Backporting to Git 2.37.3 due to Nano Server issue with Min Git for Windows
@ajwalker, this Community contribution was recently assigned to you for review.
- Do you still have capacity to review this? We are mindful of your time, so if you are not able to take this on, please re-assign to one or more other reviewers.
- Add the workflowin dev label if the merge request needs action from the author.
added automation:reviewers-reminded label
added 122 commits
-
0dcc662c...77ece377 - 120 commits from branch
gitlab-org:main
- f87ed3f7 - Merge remote-tracking branch 'upstream/main' into runner-windows-nanoserver-images
- 488c8514 - Git 2.39.0 released, looks good!
-
0dcc662c...77ece377 - 120 commits from branch
mentioned in issue gitlab-org/quality/triage-reports#10522 (closed)
mentioned in issue gitlab-org/quality/triage-reports#10634 (closed)
mentioned in issue gitlab-org/quality/triage-reports#10738 (closed)
added idle label
mentioned in issue gitlab-org/quality/triage-reports#10824 (closed)
removed idle label
mentioned in issue gitlab-org/quality/triage-reports#10995 (closed)
mentioned in issue gitlab-org/quality/triage-reports#11110 (closed)
mentioned in issue gitlab-org/quality/triage-reports#11206 (closed)
added idle label
mentioned in issue gitlab-org/quality/triage-reports#11286 (closed)
mentioned in issue gitlab-org/quality/triage-reports#11376 (closed)
mentioned in issue gitlab-org/quality/triage-reports#11485 (closed)
removed idle label
- Resolved by Hoff_IO
- Resolved by Hoff_IO
- Resolved by Hoff_IO
- Resolved by Hoff_IO
added 370 commits
-
488c8514...84b1b297 - 368 commits from branch
gitlab-org:main
- 9f7e8d91 - Merge remote-tracking branch 'upstream/main' into runner-windows-nanoserver-images
- 17c738fa - Fix to not push dev nanoserver images to Docker Hub
-
488c8514...84b1b297 - 368 commits from branch
added 70 commits
-
17c738fa...c95b0482 - 68 commits from branch
gitlab-org:main
- 59f741b6 - Merge remote-tracking branch 'upstream/main' into runner-windows-nanoserver-images
- 54745ef6 - Updated prerelease CI for nanoserver image builds to match upstream
-
17c738fa...c95b0482 - 68 commits from branch
mentioned in commit c444d394
@82phil, how was your code review experience with this merge request? Please tell us how we can continue to iterate and improve:
- React with a
or a on this comment to describe your experience. - Create a new comment starting with
@gitlab-bot feedback
below, and leave any additional feedback you have for us in the comment.
Interested in learning more tips and tricks to solve your next challenge faster? Subscribe to the GitLab Community Newsletter for contributor-focused content and opportunities to level up.
Thanks for your help!
This message was generated automatically. You're welcome to improve it.
- React with a
changed milestone to %15.11