## Description Gitlab Runner currently only supports Windows batch and Windows PowerShell on Windows platforms. Recently Microsoft introduced PowerShell 6 (Core) which can run indifferently on Linux or Windows: https://docs.microsoft.com/en-us/powershell/scripting/powershell-scripting?view=powershell-6 This new PowerShell doesn't overwrite the pre-installed Powershell (v5.x) on Windows systems, but runs alongside the old version, thus requiring the user to launch Powershell Core with a different command than previously. The new executable is named "pwsh" and can be launched with the "pwsh" command if the PATH environment variable is correctly set to point to Powershell Core's installation path. Setting the PATH variable using PowerShell : `[Environment]::SetEnvironmentVariable("PATH", "$env:PATH;C:\Program Files\Powershell\6.0.2", [EnvironmentVariableTarget]::Machine)` ## %"12.9" milestone The goal for the %"12.9" milestone to understand, investigate what needs to be done for us to add PowerShell core support, either by providing a PoC or having some kind of MVC ready and better answers to the following questions: - What needs to happen to support both Linux/Windows? - What needs to happen to support Docker executor Linux/Windows? - What needs to happen to support Kubernetes executor to use pwsh? - What is the upgrade path for users? ## Proposal ### Executors - Allow user to specify `pwsh` as a shell, which will use the [`powershell`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/master/shells/powershell.go) shell writer, but instead of invoking [`powershell`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/master/shells/powershell.go#L260-264) invoke `pwsh`. Something similar to what we do in `bash` and `sh` just no fallback. There is WIP merge request https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1855. The reason for having a new shell the user has to configured instead of just using `powershell` is because if we make `powershell` just use PowerShell 6/7 out of the box if its available we are going to end up breaking existing scripts for users that expect to use PowerShell 5, like this users need to be aware of the change they make. - `pwsh` accepts both `/` and `\` in the path for Unix based systems, but `git` doesn't accept them. When we use the powershell script generator we use `\` since that is the acceptable version for Windows. This leads to problems like [this fails CI job](https://gitlab.com/steveazz/playground/-/jobs/439548305). In https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1855/diffs#968cddea646e20e57b2edd3e8dc966cf4c268ab6_302_301 we extend the Extend the [ShellWriter interface](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/1e7ba9565f234a57bbe354bd10fc23118d198d5a/shells/shell_writer.go) and implement a `Join` function to extend the which will set the correct slash depending on script type and call it instead of [`path.Join`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/1e7ba9565f234a57bbe354bd10fc23118d198d5a/shells/abstract.go#L301). Another way to do this is don't covert to backslash if `pwsh` is chosen. - We use [`$env:computername`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/058d0c5e98afbb54eb2d12614ed73b563109038c/shells/powershell.go#L274-280) to print out the first line of the job `Running on ...`, `$env:computername` doesn't seem available on Unix pwsh, so have some kind of fallback environment such as `hostname`. #### Shell Executor https://gitlab.com/gitlab-org/gitlab-runner/issues/13134 With the points above the shell executor should work out of the box, apart from some [git failures ](/uploads/fd44f8e435a3fc7b90bd789ff7e80442/Screen_Shot_2020-02-18_at_12.00.35.png) which we need to fix the path generation. #### Docker Executor Windows https://gitlab.com/gitlab-org/gitlab-runner/issues/13139 - Update the Windows helper image to have pwsh installed and also PowerShell 5 installed, for example https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1855/diffs#972203cd1d33a1b728ea6f82644ab3da840a4ad6_26_25 - Update https://gitlab.com/gitlab-org/gitlab-runner/-/blob/058d0c5e98afbb54eb2d12614ed73b563109038c/docs/executors/docker.md#L52-56 to make it clear that users can use `nanoserver` now. #### Docker Executor Linux https://gitlab.com/gitlab-org/gitlab-runner/issues/4021 - Have `docker` executor choose between `pwsh` and `bash` depending on the user `config.toml`. We need to remove any assumptions we have that we are using a `bash` shell. - Update the linux helper image to have [pwsh installed](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1855/diffs#492e036ba59375803225611df55ed9944a8dfa69_2_1). One thing to consider is how much of size increase there would be by having pwsh installed since this container is used multiple times, it might be worth having a separate image just for `pwsh` and then have the executor chooses the right image. - Somehow update [gitlab-runner-build](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/058d0c5e98afbb54eb2d12614ed73b563109038c/dockerfiles/build/scripts/gitlab-runner-build) or the usage of it to support both bash and `pwsh` so it doesn't mess up helper image commands. #### Kubernetes https://gitlab.com/gitlab-org/gitlab-runner/issues/13145 - Have `kubernets` executor choose between `pwsh` and `bash` depending on the user `config.toml`. We need to remove any assumptions we have that we are using a `bash` shell. ### Development environment - Update [Vagrant box](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/master/Vagrantfile) to have PowerShell 6 installed. https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1848 ### Plan to migrate from PowerShell 5 to PowerShell core - Have dual support for PowerShell 5 and PowerShell 6/7 up till 14.0 - In 14.0 have `pwsh` the default shell, instead of `powershell` - In 15.0 completely remove `powershell` and only allow `pwsh` which would be PowerShell core. ## Possible follow up issues - [Update our helper image to use nanoserver instead of servercore when you use `pwsh`](https://gitlab.com/gitlab-org/gitlab-runner/issues/3291#note_289446444) *(backwords compatibility needs to be considered for users still using PowerShell 5)*. This should make our builds faster instead of starting a 4GB container it will be around 1GB of a container. This can also bring some maintenance cost if we need to support both PowerShell 5 and PowerShell 6, since we would have to keep the servercore helper images up to date as well. ## Dev log <details> <summary> 2020-02-17 </summary> - Set up the environment to have powershell 6 installed - Hardcode Runner to use `pwsh` when `powershell` is specified as a shell. - [Test simple job on Windows](http://127.0.0.1:3000/root/playground/-/jobs/1186) - [Test simple job in Linux](https://gitlab.com/steveazz/playground/-/jobs/439548305) - Start investigating the scope of the issue </details> <details> <summary> 2020-02-18 </summary> - Get https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1848 in a mergeable state, required me to change how we provision our environment https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1853 - Start with Extend the [ShellWriter interface](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/1e7ba9565f234a57bbe354bd10fc23118d198d5a/shells/shell_writer.go) and implement a `Join` function to extend the which will set the correct slash depending on script type and call it instead of [`path.Join`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/1e7ba9565f234a57bbe354bd10fc23118d198d5a/shells/abstract.go#L301) to validate the idea. - With https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1855 we manage to get a [job running on Linux with `pwsh`](/uploads/fd44f8e435a3fc7b90bd789ff7e80442/Screen_Shot_2020-02-18_at_12.00.35.png) with some warnings. </details> <details> <summary> 2020-02-19 </summary> - [Run job on Windows Docker executor](/uploads/fe1a626e3d3b451289a5743fd6ad5a6e/Screen_Shot_2020-02-19_at_09.00.00.png) - Update helper image to have `pwsh` available <details> <summary> git diff </summary> ```diff diff --git a/dockerfiles/build/Dockerfile.x86_64_servercore1809 b/dockerfiles/build/Dockerfile.x86_64_servercore1809 index 8c48126ab..f4e50a4b6 100644 --- a/dockerfiles/build/Dockerfile.x86_64_servercore1809 +++ b/dockerfiles/build/Dockerfile.x86_64_servercore1809 @@ -22,7 +22,7 @@ RUN powershell -File .\checksum.ps1 -TargetFile git-lfs.zip -ExpectedHash ${Env: RUN Expand-Archive -Path git.zip -DestinationPath git RUN Expand-Archive -Path git-lfs.zip -DestinationPath git-lfs -FROM mcr.microsoft.com/windows/servercore:1809_amd64 +FROM mcr.microsoft.com/powershell:6.2.3-windowsservercore-1809 ```` </details> - Rung job with `windows-docker` executor <details> <summary> .gitlab-ci.yml </summary> ```yaml job: image: mcr.microsoft.com/powershell/6.2.3-windowsservercore-1809 script: - echo $PSVersionTable.PSVersion ``` </details> </details> <details> <summary> 2020-02-20 </summary> - Investigate kubernetes executor with pwsh - We need update the executor to support multiple shells - Update the helper command `gitlab-runner-build` to use `pwsh` or `bash` depending on the shell. - Update proposal </details> ## Merge Requests 1. [Use chocolatey to install software in Vagrant boxes](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1853) 1. [Install PowerShell core in Vagrant](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1848) 1. [Add PowerShell 6 support for shell executor](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1855)