Error when running 4 CI jobs in parallel: CreateProcess: The user name or password is incorrect. (0x52e)

Summary

I am having a runner system failure when I run 4 CI jobs in parallel. It doesn't seem to happen when only 3 jobs are running in parallel. The error states

error=container 93cd96f337e65f... 
encountered an error during hcsshim::System::CreateProcess: failure in a Windows system call: The user name or password is incorrect. (0x52e)

Steps to reproduce

.gitlab-ci.yml 
  JOB_NAME:
    image: registry.gitlab.com/myimage
    stage: build
    script:      
      - python <some arguments>
    tags: [vs2019]

Actual behavior

3 out of 4 jobs start running. The remaining one fails with the error.

Expected behavior

4 jobs should run fine in parallel.

Relevant logs and/or screenshots

   module=libcontainerd namespace=moby container=93cd96f337e65fea918930f12c0d04737dcd0b16c8c41598fac769498216b496 exec=c79c265e42cc5b51ed9bada466b097023a8f7ed5089b59bb8b8cfb1bd1fd7e2f
error=container93cd96f337e65fea918930f12c0d04737dcd0b16c8c41598fac769498216b496 encountered an error during
hcsshim::System::CreateProcess: failure in a Windows system call: The user name or password is incorrect. (0x52e) extra info: 
{"CommandLine":"cmd.exe /C \"ECHO 131.101.246.244 host.docker.internal \u003e\u003e %systemroot%\\system32\\drivers\\etc\\hosts \u0026 ECHO 131.101.246.244 gateway.docker.internal \u003e\u003e 
%systemroot%\\system32\\drivers\\etc\\hosts\"","User":"Administrator","WorkingDirectory":"/","Environment":
{"CI":"true","CI_API_V4_URL":"https://gitlab.com/api/v4","CI_BUILDS_DIR":"c:\\builds","CI_BUILD_BEFORE_SHA":"0000000000000000000000000000000000000000","CI_BUILD_ID":"974343241","CI_BUILD_NAME":"JOB_NAME","CI_BUILD_REF":"51240f69eac4d72a05fa628cf46400b6380f39fa","CI_BUILD_REF_NAME":"master","CI_BUILD_REF_SLUG":"master","CI_BUILD_STAGE":"build","CI_BUILD_TOKEN":"qdTwEyeABNNYH...","CI_COMMIT_BEFORE_SHA":"0000000000000000000000000000000000000000","CI_COMMIT_BRANCH":"master","CI_COMMIT_DESCRIPTION":"","CI_COMMIT_MESSAGE":"Update .gitlab-ci.yml","CI_COMMIT_REF_NAME":"master","CI_COMMIT_REF_PROTECTED":"false","CI_COMMIT_REF_SLUG":"master","CI_COMMIT_SHA":"51240f69eac4d72a05fa628cf46400b6380f39fa","CI_COMMIT_SHORT_SHA":"51240f69","CI_COMMIT_TIMESTAMP":"2021-01-18T15:31:54+00:00","CI_COMMIT_TITLE":"Update .gitlab-ci.yml","CI_CONCURRENT_ID":"0","CI_CONCURRENT_PROJECT_ID":"0","CI_CONFIG_PATH":".gitlab-ci.yml","CI_DEFAULT_BRANCH":"master","CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX":"gitlab.com:443/masterest-eng/dependency_proxy/containers","CI_DEPENDENCY_PROXY_PASSWORD":"qdTwEyeABNNY...","CI_DEPENDENCY_PROXY_SERVER":"gitlab.com:443","CI_DEPENDENCY_PROXY_USER":"gitlab-ci-token","CI_DISPOSABLE_ENVIRONMENT":"true","CI_JOB_ID":"974343241","CI_JOB_IMAGE":"registry.gitlab.com/myimageframe4.8","CI_JOB_JWT":"eyJhbGciOiJSUzI1NiIsImtpZCI6IjRpM3NGRTdzeHFOU...","CI_JOB_NAME":"JOB_NAME","CI_JOB_STAGE":"build","CI_JOB_STATUS":"failed","CI_JOB_TOKEN":"qdTwEyeABNNYHR...","CI_JOB_URL":"https://gitlab.com/masterest-eng/path/974343241","CI_NODE_TOTAL":"1","CI_PAGES_DOMAIN":"gitlab.io","CI_PAGES_URL":"https://masterest-eng.gitlab.io/path","CI_PIPELINE_ID":"244242912","CI_PIPELINE_IID":"57","CI_PIPELINE_SOURCE":"web","CI_PIPELINE_URL":"https://gitlab.com/masterest-eng/path/-/pipelines/244242912","CI_PROJECT_CONFIG_PATH":".gitlab-ci.yml","CI_PROJECT_DIR":"c:\\builds\\masterest-eng\\path","CI_PROJECT_ID":"23134460","CI_PROJECT_NAME":"src","CI_PROJECT_NAMESPACE":"masterest-eng/path","CI_PROJECT_PATH":"masterest-eng/path","CI_PROJECT_PATH_SLUG":"masterest-eng-path","CI_PROJECT_REPOSITORY_LANGUAGES":"c++,max,c,perl,rich text format","CI_PROJECT_ROOT_NAMESPACE":"masterest-eng","CI_PROJECT_TITLE":"src","CI_PROJECT_URL":"https://gitlab.com/masterest-eng/path","CI_PROJECT_VISIBILITY":"private","CI_REGISTRY":"registry.gitlab.com","CI_REGISTRY_IMAGE":"registry.gitlab.com/masterest-eng/path","CI_REGISTRY_PASSWORD":"qdTwEyeABNNYHRiuqq79","CI_REGISTRY_USER":"gitlab-ci-token","CI_REPOSITORY_URL":"https://gitlab-ci-token:qdTwEyeABNNYHRiuqq79@gitlab.com/masterest-eng/path.git","CI_RUNNER_DESCRIPTION":"host-e-build-master-runner-4","CI_RUNNER_EXECUTABLE_ARCH":"windows/386","CI_RUNNER_ID":"3969180","CI_RUNNER_REVISION":"ece86343","CI_RUNNER_SHORT_TOKEN":"Kn9LekN9","CI_RUNNER_TAGS":"vs2019","CI_RUNNER_VERSION":"13.5.0","CI_SERVER":"yes","CI_SERVER_HOST":"gitlab.com","CI_SERVER_NAME":"GitLab","CI_SERVER_PORT":"443","CI_SERVER_PROTOCOL":"https","CI_SERVER_REVISION":"1194e3e3c4d","CI_SERVER_TLS_CA_FILE":"-----BEGIN CERTIFICATE-----\nMIIF1TCCBL2gAwIBAgIRAI...ybCiN\n-----END CERTIFICATE-----
\n-----BEGIN CERTIFICATE-----\nMIIGEzCCA/ugAwIBAgI...=\n-----END CERTIFICATE-----
\n-----BEGIN CERTIFICATE-----\nMIIFgTCCBGmgAwIBAgIQOXJEOvkit1HX02wQ3TE1lTANBgkqhkGp4z7h/jnZymQyd/teRCBaho1+V\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIEMjCCAxqgAwIBAgIBATANBgkqhkiG9w0BAQUFADB7MQswCQYD...==\n-----END CERTIFICATE-----","CI_SERVER_URL":"https://gitlab.com","CI_SERVER_VERSION":"13.8.0-pre","CI_SERVER_VERSION_MAJOR":"13","CI_SERVER_VERSION_MINOR":"8","CI_SERVER_VERSION_PATCH":"0","FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION":"false","FF_NETWORK_PER_BUILD":"false","FF_RESET_HELPER_IMAGE_ENTRYPOINT":"true","FF_SHELL_EXECUTOR_USE_LEGACY_PROCESS_KILL":"false","FF_SKIP_NOOP_BUILD_STAGES":"true","FF_USE_DIRECT_DOWNLOAD":"true","FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY":"true","GITLAB_CI":"true","GITLAB_FEATURES":"audit_events,blocked_issues,board_iteration_lists,code_owners,code_review_analytics,contribution_analytics,description_diffs,elastic_search,group_activity_analytics,group_bulk_edit,group_webhooks,instance_level_devops_adoption,issuable_default_templates,issue_weights,iterations,ldap_group_sync,member_lock,merge_request_approvers,milestone_charts,multiple_issue_assignees,multiple_ldap_servers,multiple_merge_request_assignees,multiple_merge_request_reviewers,project_merge_request_analytics,protected_refs_for_users,push_rules,repository_mirrors,repository_size_limit,resource_access_token,seat_link,send_emails_from_admin_area,scoped_issue_board,usage_quotas,visual_review_app,wip_limits,adjourned_deletion_for_projects_and_groups,admin_audit_log,auditor_user,blocking_merge_requests,board_assignee_lists,board_milestone_lists,ci_cd_projects,ci_secrets_management,cluster_agents,cluster_deployments,code_owner_approval_required,commit_committer_check,compliance_framework,custom_compliance_frameworks,cross_project_pipelines,custom_file_templates,custom_file_templates_for_namespace,custom_project_templates,cycle_analytics_for_groups,db_load_balancing,default_branch_protection_restriction_in_groups,default_project_deletion_protection,disable_name_update_for_users,email_additional_text,epics,extended_audit_events,external_authorization_service_api_management,feature_flags_related_issues,file_locks,geo,generic_alert_fingerprinting,github_project_service_integration,group_allowed_email_domains,group_coverage_reports,group_forking_protection,group_ip_restriction,group_merge_request_analytics,group_merge_request_approval_settings,group_milestone_project_releases,group_project_templates,group_repository_analytics,group_saml,group_saml_group_sync,group_wikis,incident_sla,incident_metric_upload,ide_schema_config,issues_analytics,jira_issues_integration,ldap_group_sync_filter,merge_pipelines,merge_request_performance_metrics,admin_merge_request_approvers_rules,merge_trains,metrics_reports,multiple_alert_http_integrations,multiple_approval_rules,multiple_group_issue_boards,object_storage,operations_dashboard,package_forwarding,pages_size_limit,productivity_analytics,project_aliases,protected_environments,reject_unsigned_commits,required_ci_templates,scoped_labels,smartcard_auth,swimlanes,group_timelogs,type_of_work_analytics,minimal_access_role,unprotection_restrictions,ci_project_subscriptions,incident_timeline_view,oncall_schedules,export_user_permissions,enterprise_templates","GITLAB_USER_EMAIL":"isaac.lastname@masterest.com","GITLAB_USER_ID":"7384865","GITLAB_USER_LOGIN":"isccha","GITLAB_USER_NAME":"myname","PROJECT":"c:\\path\\myscript.py"},"CreateStdInPipe":true,"CreateStdOutPipe":true,"CreateStdErrPipe":true,"ConsoleSize":[0,0]}

Environment description

config.toml contents

My config.toml looks like this:

concurrent = 4
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "runner1"
  limit = 1
  (...)
[[runners]]
  name = "runner2"
  limit = 1
  (...)
[[runners]]
  name = "runner3"
  limit = 1
  (...)
[[runners]]
  name = "runner4"
  limit = 1
  (...)

Used GitLab Runner version

I am using docker-windows executor, inside a VM with OS = Windows Server 2019 1809

Gitlab Runner

Version:      13.5.0
Git revision: ece86343
Git branch:   13-5-stable
GO version:   go1.13.8
Built:        2020-10-20T12:05:24+0000
OS/Arch:      windows/386

Docker version

Client: Docker Engine - Community
 Cloud integration: 1.0.2
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        4484c46d9d
 Built:             Wed Sep 16 17:00:27 2020
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.24)
  Go version:       go1.13.15
  Git commit:       4484c46d9d
  Built:            Wed Sep 16 17:14:20 2020
  OS/Arch:          windows/amd64
  Experimental:     false

Possible fixes

Looks related to this issue.

Also, looks related to this issue.

Edited by Isaac Chaves