Skip to content

GitLab Next

Closed
Created by bugoutfr@bugoutfr

Add support for Powershell Core (investigation and analysis)

Description

Gitlab Runner currently only supports Windows batch and Windows PowerShell on Windows platforms. Recently Microsoft introduced PowerShell 6 (Core) which can run indifferently on Linux or Windows: https://docs.microsoft.com/en-us/powershell/scripting/powershell-scripting?view=powershell-6

This new PowerShell doesn't overwrite the pre-installed Powershell (v5.x) on Windows systems, but runs alongside the old version, thus requiring the user to launch Powershell Core with a different command than previously. The new executable is named "pwsh" and can be launched with the "pwsh" command if the PATH environment variable is correctly set to point to Powershell Core's installation path.

Setting the PATH variable using PowerShell :

[Environment]::SetEnvironmentVariable("PATH", "$env:PATH;C:\Program Files\Powershell\6.0.2", [EnvironmentVariableTarget]::Machine)

%12.9 milestone

The goal for the %12.9 milestone to understand, investigate what needs to be done for us to add PowerShell core support, either by providing a PoC or having some kind of MVC ready and better answers to the following questions:

  • What needs to happen to support both Linux/Windows?
  • What needs to happen to support Docker executor Linux/Windows?
  • What needs to happen to support Kubernetes executor to use pwsh?
  • What is the upgrade path for users?

Proposal

Executors

  • Allow user to specify pwsh as a shell, which will use the powershell shell writer, but instead of invoking powershell invoke pwsh. Something similar to what we do in bash and sh just no fallback. There is WIP merge request !1855 (closed). The reason for having a new shell the user has to configured instead of just using powershell is because if we make powershell just use PowerShell 6/7 out of the box if its available we are going to end up breaking existing scripts for users that expect to use PowerShell 5, like this users need to be aware of the change they make.
  • pwsh accepts both / and \ in the path for Unix based systems, but git doesn't accept them. When we use the powershell script generator we use \ since that is the acceptable version for Windows. This leads to problems like this fails CI job. In !1855 (diffs) we extend the Extend the ShellWriter interface and implement a Join function to extend the which will set the correct slash depending on script type and call it instead of path.Join. Another way to do this is don't covert to backslash if pwsh is chosen.
  • We use $env:computername to print out the first line of the job Running on ..., $env:computername doesn't seem available on Unix pwsh, so have some kind of fallback environment such as hostname.

Shell Executor #13134 (closed)

With the points above the shell executor should work out of the box, apart from some git failures which we need to fix the path generation.

Docker Executor Windows #13139 (closed)

Docker Executor Linux #4021 (closed)

  • Have docker executor choose between pwsh and bash depending on the user config.toml. We need to remove any assumptions we have that we are using a bash shell.
  • Update the linux helper image to have pwsh installed. One thing to consider is how much of size increase there would be by having pwsh installed since this container is used multiple times, it might be worth having a separate image just for pwsh and then have the executor chooses the right image.
  • Somehow update gitlab-runner-build or the usage of it to support both bash and pwsh so it doesn't mess up helper image commands.

Kubernetes #13145 (closed)

  • Have kubernets executor choose between pwsh and bash depending on the user config.toml. We need to remove any assumptions we have that we are using a bash shell.

Development environment

Plan to migrate from PowerShell 5 to PowerShell core

  • Have dual support for PowerShell 5 and PowerShell 6/7 up till 14.0
  • In 14.0 have pwsh the default shell, instead of powershell
  • In 15.0 completely remove powershell and only allow pwsh which would be PowerShell core.

Possible follow up issues

  • Update our helper image to use nanoserver instead of servercore when you use pwsh (backwords compatibility needs to be considered for users still using PowerShell 5). This should make our builds faster instead of starting a 4GB container it will be around 1GB of a container. This can also bring some maintenance cost if we need to support both PowerShell 5 and PowerShell 6, since we would have to keep the servercore helper images up to date as well.

Dev log

2020-02-17
2020-02-18
2020-02-19

  • Run job on Windows Docker executor

    • Update helper image to have pwsh available
    git diff 
    diff --git a/dockerfiles/build/Dockerfile.x86_64_servercore1809 b/dockerfiles/build/Dockerfile.x86_64_servercore1809
index 8c48126ab..f4e50a4b6 100644
--- a/dockerfiles/build/Dockerfile.x86_64_servercore1809
+++ b/dockerfiles/build/Dockerfile.x86_64_servercore1809
@@ -22,7 +22,7 @@ RUN powershell -File .\checksum.ps1 -TargetFile git-lfs.zip -ExpectedHash ${Env:
 RUN Expand-Archive -Path git.zip -DestinationPath git
 RUN Expand-Archive -Path git-lfs.zip -DestinationPath git-lfs

-FROM mcr.microsoft.com/windows/servercore:1809_amd64
+FROM mcr.microsoft.com/powershell:6.2.3-windowsservercore-1809
    - Rung job with `windows-docker` executor
    .gitlab-ci.yml 
    job:
  image: mcr.microsoft.com/powershell/6.2.3-windowsservercore-1809
  script:
  - echo $PSVersionTable.PSVersion
2020-02-20
  • Investigate kubernetes executor with pwsh
    • We need update the executor to support multiple shells
    • Update the helper command gitlab-runner-build to use pwsh or bash depending on the shell.
  • Update proposal

Merge Requests

  1. Use chocolatey to install software in Vagrant boxes
  2. Install PowerShell core in Vagrant
  3. Add PowerShell 6 support for shell executor
Edited by Steve Azzopardi