Skip to content

Builds on Kubernetes Windows nodes fails without 'shell = "pwsh"'

Summary

Without providing shell = "pwsh" builds on Windows-nodes 'hang'.

Steps to reproduce

Install the Kubernetes-runner with the values.yaml provided below, and run Windows-build. Important is to omit shell = "pwsh" from the toml-section.

.gitlab-ci.yml
stages:
  - build

windows build:
  image: fppkg.azurecr.io/fpc-322-windows-buildimage
  stage: build
  tags:
   - windows
  script:
    - fppkg build
    - fppkg archive
  artifacts:
    paths:
      - fpcdab-*.source.zip
    expire_in: 1 month

linux build:
  image: quay.io/loesje/fpc-322-linux-buildimage
  stage: build
  tags:
   - linux
  script:
    - fppkg build
    - fppkg archive
  artifacts:
    paths:
      - fpcdab-*.source.zip
    expire_in: 1 month

Actual behavior

The build times out after (in my case) an hour.

This is the log of the build-container:

error: container builder is not valid for pod runner-sx55k3qp-project-28515712-concurrent-0kwlwd

This is the log of the helper-container:

Running on RUNNER-SX55K3QP via 
gitlab-runner-fpcprojects-windows-gitlab-runner-777fcf8b77wb4cw...

{"command_exit_code": 0, "script": "runner_script_trap"}

Expected behavior

A successful build.

Relevant logs and/or screenshots

Description of pod
Name:         runner-sx55k3qp-project-28515712-concurrent-0kwlwd
Namespace:    gitlab
Priority:     0
Node:         aksmspool00000c/10.240.0.35
Start Time:   Mon, 23 Aug 2021 20:04:11 +0200
Labels:       pod=runner-sx55k3qp-project-28515712-concurrent-0
Annotations:  <none>
Status:       Running
IP:           10.240.0.36
IPs:
  IP:  10.240.0.36
Init Containers:
  init-permissions:
    Container ID:  docker://ee942d83b20e6191a0390c3f83729af11f7cb3ee56578d7d6afda39025db074e
    Image:         gitlab/gitlab-runner-helper:x86_64-307aeab7-servercore1809
    Image ID:      docker-pullable://gitlab/gitlab-runner-helper@sha256:037a4999e4c6fa904a4cc13a7fa938e7a4cc2df0872fe4dec93ed0f867ebc559
    Port:          <none>
    Host Port:     <none>
    Command:
      pwsh
      -c
      icacls $ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath("/logs-28515712-1527012460") /grant 'Everyone:(OI)(CI)F' /t /q | out-null;
      icacls $ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath("C:\\builds") /grant 'Everyone:(OI)(CI)F' /t /q | out-null
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 23 Aug 2021 20:04:15 +0200
      Finished:     Mon, 23 Aug 2021 20:04:16 +0200
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /logs-28515712-1527012460 from logs (rw)
      /scripts-28515712-1527012460 from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-4dhfs (ro)
      C:\builds from repo (rw)
Containers:
  build:
    Container ID:  docker://baee6725284726fcaa9e735289df77e0c5c77e18488cfad9c0737bdeb86c489d
    Image:         fppkg.azurecr.io/fpc-322-windows-buildimage
    Image ID:      docker-pullable://fppkg.azurecr.io/fpc-322-windows-buildimage@sha256:81c59ec05072710a4ee24cd4e1c6a0d7a53baddf8f1c2216e87253edfdd201fa
    Port:          <none>
    Host Port:     <none>
    Command:
      powershell
      -NoProfile
      -NoLogo
      -InputFormat
      text
      -OutputFormat
      text
      -NonInteractive
      -ExecutionPolicy
      Bypass
      -Command
      -
    State:          Running
      Started:      Mon, 23 Aug 2021 20:04:24 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION:          false
      FF_NETWORK_PER_BUILD:                                  false
      FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY:           false
      FF_USE_DIRECT_DOWNLOAD:                                true
      FF_SKIP_NOOP_BUILD_STAGES:                             true
      FF_USE_FASTZIP:                                        false
      FF_GITLAB_REGISTRY_HELPER_IMAGE:                       true
      FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR:                  false
      FF_ENABLE_BASH_EXIT_CODE_CHECK:                        false
      FF_USE_WINDOWS_LEGACY_PROCESS_STRATEGY:                true
      FF_SKIP_DOCKER_MACHINE_PROVISION_ON_CREATION_FAILURE:  true
      FF_USE_NEW_BASH_EVAL_STRATEGY:                         false
      FF_USE_POWERSHELL_PATH_RESOLVER:                       false
      FF_USE_DYNAMIC_TRACE_FORCE_SEND_INTERVAL:              false
      FF_SCRIPT_SECTIONS:                                    false
      CI_JOB_IMAGE:                                          fppkg.azurecr.io/fpc-322-windows-buildimage
      CI_RUNNER_SHORT_TOKEN:                                 sx55k3Qp
      CI_BUILDS_DIR:                                         C:\builds
      CI_PROJECT_DIR:                                        C:/builds/sx55k3Qp/0/fpcprojects/fpcdab
      CI_CONCURRENT_ID:                                      0
      CI_CONCURRENT_PROJECT_ID:                              0
      CI_SERVER:                                             yes
      CI_JOB_STATUS:                                         running
      CI_SERVER_TLS_CA_FILE:                                 -----BEGIN CERTIFICATE-----
                                                             MIIGBzCCBO+gAwIBAgIQXCLSMilzZJR9TSABzbgKzzANBgkqhkiG9w0BAQsFADCB
                                                             woE3CsYogf1SfQM=
                                                             -----END CERTIFICATE-----
                                                             -----BEGIN CERTIFICATE-----
                                                             MIIGEzCCA/ugAwIBAgIQfVtRJrR2uhHbdBYLvFMNpzANBgkqhkiG9w0BAQwFADCB
                                                             00u/I5sUKUErmgQfky3xxzlIPK1aEn8=
                                                             -----END CERTIFICATE-----
                                                             -----BEGIN CERTIFICATE-----
                                                             MIIF3jCCA8agAwIBAgIQAf1tMPyjylGoG7xkDjUDLTANBgkqhkiG9w0BAQwFADCB
                                                             jjxDah2nGN59PRbxYvnKkKj9
                                                             -----END CERTIFICATE-----
                                                             -----BEGIN CERTIFICATE-----
                                                             MIIFgTCCBGmgAwIBAgIQOXJEOvkit1HX02wQ3TE1lTANBgkqhkiG9w0BAQwFADB7
                                                             MQswCQYDVQQGEwJHQjEbMBkGA1UECAwSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYD
                                                             vGp4z7h/jnZymQyd/teRCBaho1+V
                                                             -----END CERTIFICATE-----
                                                             -----BEGIN CERTIFICATE-----
                                                             MIIEMjCCAxqgAwIBAgIBATANBgkqhkiG9w0BAQUFADB7MQswCQYDVQQGEwJHQjEb
                                                             smPi9WIsgtRqAEFQ8TmDn5XpNpaYbg==
                                                             -----END CERTIFICATE-----
      CI_PIPELINE_ID:                                        357473838
      CI_PIPELINE_URL:                                       https://gitlab.com/fpcprojects/fpcdab/-/pipelines/357473838
      CI_JOB_ID:                                             1527012460
      CI_JOB_URL:                                            https://gitlab.com/fpcprojects/fpcdab/-/jobs/1527012460
      CI_JOB_STARTED_AT:                                     2021-08-23T18:04:10Z
      CI_BUILD_ID:                                           1527012460
      CI_REGISTRY_USER:                                      gitlab-ci-token
      CI_DEPENDENCY_PROXY_USER:                              gitlab-ci-token
      CI_JOB_NAME:                                           windows build
      CI_JOB_STAGE:                                          build
      CI_NODE_TOTAL:                                         1
      CI_BUILD_NAME:                                         windows build
      CI_BUILD_STAGE:                                        build
      CI:                                                    true
      GITLAB_CI:                                             true
      CI_SERVER_URL:                                         https://gitlab.com
      CI_SERVER_HOST:                                        gitlab.com
      CI_SERVER_PORT:                                        443
      CI_SERVER_PROTOCOL:                                    https
      CI_SERVER_NAME:                                        GitLab
      CI_SERVER_VERSION:                                     14.3.0-pre
      CI_SERVER_VERSION_MAJOR:                               14
      CI_SERVER_VERSION_MINOR:                               3
      CI_SERVER_VERSION_PATCH:                               0
      CI_SERVER_REVISION:                                    17389a8bb35
      GITLAB_FEATURES:                                       audit_events,blocked_issues,board_iteration_lists,code_owners,code_review_analytics,contribution_analytics,description_diffs,elastic_search,full_codequality_report,group_activity_analytics,group_bulk_edit,group_webhooks,issuable_default_templates,issue_weights,iterations,ldap_group_sync,member_lock,merge_request_approvers,milestone_charts,multiple_issue_assignees,multiple_ldap_servers,multiple_merge_request_assignees,multiple_merge_request_reviewers,project_merge_request_analytics,protected_refs_for_users,push_rules,repository_mirrors,repository_size_limit,resource_access_token,seat_link,scoped_issue_board,usage_quotas,visual_review_app,wip_limits,send_emails_from_admin_area,adjourned_deletion_for_projects_and_groups,admin_audit_log,auditor_user,blocking_merge_requests,board_assignee_lists,board_milestone_lists,ci_cd_projects,ci_secrets_management,cluster_agents,cluster_deployments,code_owner_approval_required,commit_committer_check,compliance_framework,custom_compliance_frameworks,cross_project_pipelines,custom_file_templates,custom_file_templates_for_namespace,custom_project_templates,cycle_analytics_for_groups,cycle_analytics_for_projects,db_load_balancing,default_branch_protection_restriction_in_groups,default_project_deletion_protection,disable_name_update_for_users,email_additional_text,epics,extended_audit_events,external_authorization_service_api_management,feature_flags_related_issues,file_locks,geo,generic_alert_fingerprinting,git_two_factor_enforcement,github_project_service_integration,group_allowed_email_domains,group_coverage_reports,group_forking_protection,group_ip_restriction,group_merge_request_analytics,group_merge_request_approval_settings,group_milestone_project_releases,group_project_templates,group_repository_analytics,group_saml,group_saml_group_sync,group_scoped_ci_variables,group_wikis,incident_sla,incident_metric_upload,ide_schema_config,issues_analytics,jira_issues_integration,ldap_group_sync_filter,merge_pipelines,merge_request_performance_metrics,admin_merge_request_approvers_rules,merge_trains,metrics_reports,multiple_alert_http_integrations,multiple_approval_rules,multiple_group_issue_boards,multiple_iteration_cadences,object_storage,operations_dashboard,package_forwarding,pages_size_limit,productivity_analytics,project_aliases,protected_environments,reject_unsigned_commits,required_ci_templates,scoped_labels,smartcard_auth,swimlanes,type_of_work_analytics,minimal_access_role,unprotection_restrictions,ci_project_subscriptions,incident_timeline_view,oncall_schedules,escalation_policies,export_user_permissions,api_fuzzing,auto_rollback,cilium_alerts,cluster_image_scanning,external_status_checks,container_scanning,coverage_fuzzing,credentials_inventory,dast,dependency_scanning,devops_adoption,dora4_analytics,enforce_personal_access_token_expiration,enforce_ssh_key_expiration,enterprise_templates,environment_alerts,evaluate_group_level_compliance_pipeline,group_ci_cd_analytics,group_level_compliance_dashboard,group_level_devops_adoption,incident_management,inline_codequality,insights,instance_level_devops_adoption,issuable_health_status,jira_vulnerabilities_integration,jira_issue_association_enforcement,license_scanning,personal_access_token_expiration_policy,prometheus_alerts,pseudonymizer,quality_management,release_evidence_test_artifacts,report_approver_rules,requirements,sast,sast_custom_rulesets,sast_fp_reduction,secret_detection,security_dashboard,security_on_demand_scans,security_orchestration_policies,status_page,subepics,threat_monitoring,vulnerability_auto_fix,vulnerability_finding_signatures
      CI_PROJECT_ID:                                         28515712
      CI_PROJECT_NAME:                                       fpcdab
      CI_PROJECT_TITLE:                                      fpcdab
      CI_PROJECT_PATH:                                       fpcprojects/fpcdab
      CI_PROJECT_PATH_SLUG:                                  fpcprojects-fpcdab
      CI_PROJECT_NAMESPACE:                                  fpcprojects
      CI_PROJECT_ROOT_NAMESPACE:                             fpcprojects
      CI_PROJECT_URL:                                        https://gitlab.com/fpcprojects/fpcdab
      CI_PROJECT_VISIBILITY:                                 public
      CI_PROJECT_REPOSITORY_LANGUAGES:                       pascal
      CI_PROJECT_CLASSIFICATION_LABEL:                       
      CI_DEFAULT_BRANCH:                                     trunk
      CI_CONFIG_PATH:                                        .gitlab-ci.yml
      CI_PAGES_DOMAIN:                                       gitlab.io
      CI_PAGES_URL:                                          https://fpcprojects.gitlab.io/fpcdab
      CI_REGISTRY:                                           registry.gitlab.com
      CI_REGISTRY_IMAGE:                                     registry.gitlab.com/fpcprojects/fpcdab
      CI_DEPENDENCY_PROXY_SERVER:                            gitlab.com:443
      CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX:                gitlab.com:443/fpcprojects/dependency_proxy/containers
      CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX:         gitlab.com:443/fpcprojects/dependency_proxy/containers
      CI_API_V4_URL:                                         https://gitlab.com/api/v4
      CI_PIPELINE_IID:                                       6
      CI_PIPELINE_SOURCE:                                    push
      CI_PIPELINE_CREATED_AT:                                2021-08-22T23:00:01Z
      CI_COMMIT_SHA:                                         29b9e6c8f0a3243143ea2f1783881b9253984ab4
      CI_COMMIT_SHORT_SHA:                                   29b9e6c8
      CI_COMMIT_BEFORE_SHA:                                  c7614b9ddf5834e119ec4b992161f4092699ce82
      CI_COMMIT_REF_NAME:                                    trunk
      CI_COMMIT_REF_SLUG:                                    trunk
      CI_COMMIT_BRANCH:                                      trunk
      CI_COMMIT_MESSAGE:                                      * Windows and Linux build
                                                             
      CI_COMMIT_TITLE:                                        * Windows and Linux build
      CI_COMMIT_DESCRIPTION:                                 
      CI_COMMIT_REF_PROTECTED:                               true
      CI_COMMIT_TIMESTAMP:                                   2021-08-23T00:59:56+02:00
      CI_COMMIT_AUTHOR:                                      Joost van der Sluis <joost@cnoc.nl>
      CI_BUILD_REF:                                          29b9e6c8f0a3243143ea2f1783881b9253984ab4
      CI_BUILD_BEFORE_SHA:                                   c7614b9ddf5834e119ec4b992161f4092699ce82
      CI_BUILD_REF_NAME:                                     trunk
      CI_BUILD_REF_SLUG:                                     trunk
      CI_RUNNER_ID:                                          9801189
      CI_RUNNER_DESCRIPTION:                                 aks_cnoc_jvdsluis_windows
      CI_RUNNER_TAGS:                                        windows
      GITLAB_USER_ID:                                        9100338
      GITLAB_USER_EMAIL:                                     joost@cnoc.nl
      GITLAB_USER_LOGIN:                                     jvdsluis
      GITLAB_USER_NAME:                                      Joost van der Sluis
      CI_DISPOSABLE_ENVIRONMENT:                             true
      CI_RUNNER_VERSION:                                     14.2.0
      CI_RUNNER_REVISION:                                    58ba2b95
      CI_RUNNER_EXECUTABLE_ARCH:                             linux/amd64
    Mounts:
      /logs-28515712-1527012460 from logs (rw)
      /scripts-28515712-1527012460 from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-4dhfs (ro)
      C:\builds from repo (rw)
  helper:
    Container ID:  docker://2f971be8bba0b98fb045a98ec3e7f2ecb4ff19b414979d6c08eb0f427467ee6a
    Image:         gitlab/gitlab-runner-helper:x86_64-307aeab7-servercore1809
    Image ID:      docker-pullable://gitlab/gitlab-runner-helper@sha256:037a4999e4c6fa904a4cc13a7fa938e7a4cc2df0872fe4dec93ed0f867ebc559
    Port:          <none>
    Host Port:     <none>
    Command:
      powershell
      -NoProfile
      -NoLogo
      -InputFormat
      text
      -OutputFormat
      text
      -NonInteractive
      -ExecutionPolicy
      Bypass
      -Command
      -
    State:          Running
      Started:      Mon, 23 Aug 2021 20:04:27 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION:          false
      FF_NETWORK_PER_BUILD:                                  false
      FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY:           false
      FF_USE_DIRECT_DOWNLOAD:                                true
      FF_SKIP_NOOP_BUILD_STAGES:                             true
      FF_USE_FASTZIP:                                        false
      FF_GITLAB_REGISTRY_HELPER_IMAGE:                       true
      FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR:                  false
      FF_ENABLE_BASH_EXIT_CODE_CHECK:                        false
      FF_USE_WINDOWS_LEGACY_PROCESS_STRATEGY:                true
      FF_SKIP_DOCKER_MACHINE_PROVISION_ON_CREATION_FAILURE:  true
      FF_USE_NEW_BASH_EVAL_STRATEGY:                         false
      FF_USE_POWERSHELL_PATH_RESOLVER:                       false
      FF_USE_DYNAMIC_TRACE_FORCE_SEND_INTERVAL:              false
      FF_SCRIPT_SECTIONS:                                    false
      CI_JOB_IMAGE:                                          fppkg.azurecr.io/fpc-322-windows-buildimage
      CI_RUNNER_SHORT_TOKEN:                                 sx55k3Qp
      CI_BUILDS_DIR:                                         C:\builds
      CI_PROJECT_DIR:                                        C:/builds/sx55k3Qp/0/fpcprojects/fpcdab
      CI_CONCURRENT_ID:                                      0
      CI_CONCURRENT_PROJECT_ID:                              0
      CI_SERVER:                                             yes
      CI_JOB_STATUS:                                         running
      CI_SERVER_TLS_CA_FILE:                                 -----BEGIN CERTIFICATE-----
                                                             MIIGBzCCBO+gAwIBAgIQXCLSMilzZJR9TSABzbgKzzANBgkqhkiG9w0BAQsFADCB
                                                             woE3CsYogf1SfQM=
                                                             -----END CERTIFICATE-----
                                                             -----BEGIN CERTIFICATE-----
                                                             MIIGEzCCA/ugAwIBAgIQfVtRJrR2uhHbdBYLvFMNpzANBgkqhkiG9w0BAQwFADCB
                                                             00u/I5sUKUErmgQfky3xxzlIPK1aEn8=
                                                             -----END CERTIFICATE-----
                                                             -----BEGIN CERTIFICATE-----
                                                             MIIF3jCCA8agAwIBAgIQAf1tMPyjylGoG7xkDjUDLTANBgkqhkiG9w0BAQwFADCB
                                                             jjxDah2nGN59PRbxYvnKkKj9
                                                             -----END CERTIFICATE-----
                                                             -----BEGIN CERTIFICATE-----
                                                             MIIFgTCCBGmgAwIBAgIQOXJEOvkit1HX02wQ3TE1lTANBgkqhkiG9w0BAQwFADB7
                                                             zZOFli9d31kWTz9RvdVFGD/tSo7oBmF0Ixa1DVBzJ0RHfxBdiSprhTEUxOipakyA
                                                             vGp4z7h/jnZymQyd/teRCBaho1+V
                                                             -----END CERTIFICATE-----
                                                             -----BEGIN CERTIFICATE-----
                                                             MIIEMjCCAxqgAwIBAgIBATANBgkqhkiG9w0BAQUFADB7MQswCQYDVQQGEwJHQjEb
                                                             smPi9WIsgtRqAEFQ8TmDn5XpNpaYbg==
                                                             -----END CERTIFICATE-----
      CI_PIPELINE_ID:                                        357473838
      CI_PIPELINE_URL:                                       https://gitlab.com/fpcprojects/fpcdab/-/pipelines/357473838
      CI_JOB_ID:                                             1527012460
      CI_JOB_URL:                                            https://gitlab.com/fpcprojects/fpcdab/-/jobs/1527012460
      CI_JOB_STARTED_AT:                                     2021-08-23T18:04:10Z
      CI_BUILD_ID:                                           1527012460
      CI_REGISTRY_USER:                                      gitlab-ci-token
      CI_DEPENDENCY_PROXY_USER:                              gitlab-ci-token
      CI_JOB_NAME:                                           windows build
      CI_JOB_STAGE:                                          build
      CI_NODE_TOTAL:                                         1
      CI_BUILD_NAME:                                         windows build
      CI_BUILD_STAGE:                                        build
      CI:                                                    true
      GITLAB_CI:                                             true
      CI_SERVER_URL:                                         https://gitlab.com
      CI_SERVER_HOST:                                        gitlab.com
      CI_SERVER_PORT:                                        443
      CI_SERVER_PROTOCOL:                                    https
      CI_SERVER_NAME:                                        GitLab
      CI_SERVER_VERSION:                                     14.3.0-pre
      CI_SERVER_VERSION_MAJOR:                               14
      CI_SERVER_VERSION_MINOR:                               3
      CI_SERVER_VERSION_PATCH:                               0
      CI_SERVER_REVISION:                                    17389a8bb35
      GITLAB_FEATURES:                                       audit_events,blocked_issues,board_iteration_lists,code_owners,code_review_analytics,contribution_analytics,description_diffs,elastic_search,full_codequality_report,group_activity_analytics,group_bulk_edit,group_webhooks,issuable_default_templates,issue_weights,iterations,ldap_group_sync,member_lock,merge_request_approvers,milestone_charts,multiple_issue_assignees,multiple_ldap_servers,multiple_merge_request_assignees,multiple_merge_request_reviewers,project_merge_request_analytics,protected_refs_for_users,push_rules,repository_mirrors,repository_size_limit,resource_access_token,seat_link,scoped_issue_board,usage_quotas,visual_review_app,wip_limits,send_emails_from_admin_area,adjourned_deletion_for_projects_and_groups,admin_audit_log,auditor_user,blocking_merge_requests,board_assignee_lists,board_milestone_lists,ci_cd_projects,ci_secrets_management,cluster_agents,cluster_deployments,code_owner_approval_required,commit_committer_check,compliance_framework,custom_compliance_frameworks,cross_project_pipelines,custom_file_templates,custom_file_templates_for_namespace,custom_project_templates,cycle_analytics_for_groups,cycle_analytics_for_projects,db_load_balancing,default_branch_protection_restriction_in_groups,default_project_deletion_protection,disable_name_update_for_users,email_additional_text,epics,extended_audit_events,external_authorization_service_api_management,feature_flags_related_issues,file_locks,geo,generic_alert_fingerprinting,git_two_factor_enforcement,github_project_service_integration,group_allowed_email_domains,group_coverage_reports,group_forking_protection,group_ip_restriction,group_merge_request_analytics,group_merge_request_approval_settings,group_milestone_project_releases,group_project_templates,group_repository_analytics,group_saml,group_saml_group_sync,group_scoped_ci_variables,group_wikis,incident_sla,incident_metric_upload,ide_schema_config,issues_analytics,jira_issues_integration,ldap_group_sync_filter,merge_pipelines,merge_request_performance_metrics,admin_merge_request_approvers_rules,merge_trains,metrics_reports,multiple_alert_http_integrations,multiple_approval_rules,multiple_group_issue_boards,multiple_iteration_cadences,object_storage,operations_dashboard,package_forwarding,pages_size_limit,productivity_analytics,project_aliases,protected_environments,reject_unsigned_commits,required_ci_templates,scoped_labels,smartcard_auth,swimlanes,type_of_work_analytics,minimal_access_role,unprotection_restrictions,ci_project_subscriptions,incident_timeline_view,oncall_schedules,escalation_policies,export_user_permissions,api_fuzzing,auto_rollback,cilium_alerts,cluster_image_scanning,external_status_checks,container_scanning,coverage_fuzzing,credentials_inventory,dast,dependency_scanning,devops_adoption,dora4_analytics,enforce_personal_access_token_expiration,enforce_ssh_key_expiration,enterprise_templates,environment_alerts,evaluate_group_level_compliance_pipeline,group_ci_cd_analytics,group_level_compliance_dashboard,group_level_devops_adoption,incident_management,inline_codequality,insights,instance_level_devops_adoption,issuable_health_status,jira_vulnerabilities_integration,jira_issue_association_enforcement,license_scanning,personal_access_token_expiration_policy,prometheus_alerts,pseudonymizer,quality_management,release_evidence_test_artifacts,report_approver_rules,requirements,sast,sast_custom_rulesets,sast_fp_reduction,secret_detection,security_dashboard,security_on_demand_scans,security_orchestration_policies,status_page,subepics,threat_monitoring,vulnerability_auto_fix,vulnerability_finding_signatures
      CI_PROJECT_ID:                                         28515712
      CI_PROJECT_NAME:                                       fpcdab
      CI_PROJECT_TITLE:                                      fpcdab
      CI_PROJECT_PATH:                                       fpcprojects/fpcdab
      CI_PROJECT_PATH_SLUG:                                  fpcprojects-fpcdab
      CI_PROJECT_NAMESPACE:                                  fpcprojects
      CI_PROJECT_ROOT_NAMESPACE:                             fpcprojects
      CI_PROJECT_URL:                                        https://gitlab.com/fpcprojects/fpcdab
      CI_PROJECT_VISIBILITY:                                 public
      CI_PROJECT_REPOSITORY_LANGUAGES:                       pascal
      CI_PROJECT_CLASSIFICATION_LABEL:                       
      CI_DEFAULT_BRANCH:                                     trunk
      CI_CONFIG_PATH:                                        .gitlab-ci.yml
      CI_PAGES_DOMAIN:                                       gitlab.io
      CI_PAGES_URL:                                          https://fpcprojects.gitlab.io/fpcdab
      CI_REGISTRY:                                           registry.gitlab.com
      CI_REGISTRY_IMAGE:                                     registry.gitlab.com/fpcprojects/fpcdab
      CI_DEPENDENCY_PROXY_SERVER:                            gitlab.com:443
      CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX:                gitlab.com:443/fpcprojects/dependency_proxy/containers
      CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX:         gitlab.com:443/fpcprojects/dependency_proxy/containers
      CI_API_V4_URL:                                         https://gitlab.com/api/v4
      CI_PIPELINE_IID:                                       6
      CI_PIPELINE_SOURCE:                                    push
      CI_PIPELINE_CREATED_AT:                                2021-08-22T23:00:01Z
      CI_COMMIT_SHA:                                         29b9e6c8f0a3243143ea2f1783881b9253984ab4
      CI_COMMIT_SHORT_SHA:                                   29b9e6c8
      CI_COMMIT_BEFORE_SHA:                                  c7614b9ddf5834e119ec4b992161f4092699ce82
      CI_COMMIT_REF_NAME:                                    trunk
      CI_COMMIT_REF_SLUG:                                    trunk
      CI_COMMIT_BRANCH:                                      trunk
      CI_COMMIT_MESSAGE:                                      * Windows and Linux build
                                                             
      CI_COMMIT_TITLE:                                        * Windows and Linux build
      CI_COMMIT_DESCRIPTION:                                 
      CI_COMMIT_REF_PROTECTED:                               true
      CI_COMMIT_TIMESTAMP:                                   2021-08-23T00:59:56+02:00
      CI_COMMIT_AUTHOR:                                      Joost van der Sluis <joost@cnoc.nl>
      CI_BUILD_REF:                                          29b9e6c8f0a3243143ea2f1783881b9253984ab4
      CI_BUILD_BEFORE_SHA:                                   c7614b9ddf5834e119ec4b992161f4092699ce82
      CI_BUILD_REF_NAME:                                     trunk
      CI_BUILD_REF_SLUG:                                     trunk
      CI_RUNNER_ID:                                          9801189
      CI_RUNNER_DESCRIPTION:                                 aks_cnoc_jvdsluis_windows
      CI_RUNNER_TAGS:                                        windows
      GITLAB_USER_ID:                                        9100338
      GITLAB_USER_EMAIL:                                     joost@cnoc.nl
      GITLAB_USER_LOGIN:                                     jvdsluis
      GITLAB_USER_NAME:                                      Joost van der Sluis
      CI_DISPOSABLE_ENVIRONMENT:                             true
      CI_RUNNER_VERSION:                                     14.2.0
      CI_RUNNER_REVISION:                                    58ba2b95
      CI_RUNNER_EXECUTABLE_ARCH:                             linux/amd64
    Mounts:
      /logs-28515712-1527012460 from logs (rw)
      /scripts-28515712-1527012460 from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-4dhfs (ro)
      C:\builds from repo (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  repo:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      runner-sx55k3qp-project-28515712-concurrent-0-scriptsz2n4p
    Optional:  false
  logs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  default-token-4dhfs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-4dhfs
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  kubernetes.io/arch=amd64
                 kubernetes.io/os=windows
                 node.kubernetes.io/windows-build=10.0.17763
Tolerations:     kubernetes.azure.com/scalesetpriority=spot:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age        From                      Message
  ----    ------     ----       ----                      -------
  Normal  Scheduled  <unknown>                            Successfully assigned gitlab/runner-sx55k3qp-project-28515712-concurrent-0kwlwd to aksmspool00000c
  Normal  Pulled     12m        kubelet, aksmspool00000c  Container image "gitlab/gitlab-runner-helper:x86_64-307aeab7-servercore1809" already present on machine
  Normal  Created    12m        kubelet, aksmspool00000c  Created container init-permissions
  Normal  Started    12m        kubelet, aksmspool00000c  Started container init-permissions
  Normal  Pulling    12m        kubelet, aksmspool00000c  Pulling image "fppkg.azurecr.io/fpc-322-windows-buildimage"
  Normal  Pulled     12m        kubelet, aksmspool00000c  Successfully pulled image "fppkg.azurecr.io/fpc-322-windows-buildimage" in 310.8298ms
  Normal  Created    12m        kubelet, aksmspool00000c  Created container build
  Normal  Started    12m        kubelet, aksmspool00000c  Started container build
  Normal  Pulled     12m        kubelet, aksmspool00000c  Container image "gitlab/gitlab-runner-helper:x86_64-307aeab7-servercore1809" already present on machine
  Normal  Created    12m        kubelet, aksmspool00000c  Created container helper
  Normal  Started    12m        kubelet, aksmspool00000c  Started container helper
Events
2m         Normal   Created             pod/runner-sx55k3qp-project-28515712-concurrent-0kwlwd                  Created container init-permissions
12m         Normal   Started             pod/runner-sx55k3qp-project-28515712-concurrent-0kwlwd                  Started container init-permissions
11m         Normal   Pulling             pod/runner-sx55k3qp-project-28515712-concurrent-0kwlwd                  Pulling image "fppkg.azurecr.io/fpc-322-windows-buildimage"
11m         Normal   Pulled              pod/runner-sx55k3qp-project-28515712-concurrent-0kwlwd                  Successfully pulled image "fppkg.azurecr.io/fpc-322-windows-buildimage" in 310.8298ms
11m         Normal   Created             pod/runner-sx55k3qp-project-28515712-concurrent-0kwlwd                  Created container build
11m         Normal   Started             pod/runner-sx55k3qp-project-28515712-concurrent-0kwlwd                  Started container build
11m         Normal   Pulled              pod/runner-sx55k3qp-project-28515712-concurrent-0kwlwd                  Container image "gitlab/gitlab-runner-helper:x86_64-307aeab7-servercore1809" already present on machine
11m         Normal   Created             pod/runner-sx55k3qp-project-28515712-concurrent-0kwlwd                  Created container helper
11m         Normal   Started             pod/runner-sx55k3qp-project-28515712-concurrent-0kwlwd                  Started container helper

Environment description

A failed build can be found here: https://gitlab.com/fpcprojects/fpcdab/-/jobs/1527094203. Scripts and such are here: https://gitlab.com/fpcprojects/aks-runners

config.toml contents
    [[runners]]
      executor = "kubernetes"
      [runners.feature_flags]
        FF_USE_POWERSHELL_PATH_RESOLVER = true
      [runners.kubernetes]
        poll_timeout = 3600
        [runners.kubernetes.node_selector]
          "kubernetes.io/os" = "windows"
          "node.kubernetes.io/windows-build" = "10.0.17763"
        [runners.kubernetes.node_tolerations]
           "kubernetes.azure.com/scalesetpriority=spot" = "NoSchedule"

Used GitLab Runner version

14.2.0

Possible fixes

As said, adding shell = "pwsh" to toml fixes the issue.

Edited by Joost van der Sluis