pods runner can't handle many concurrent jobs at the same time
Hi all, i have issue when i run many jobs at the same time, in this case i run 75 jobs runner at the same time with same gitlab-runner tags
15 from 75 jobs failed when run at the same time, but if i retry when another jobs done, the jobs can finish with no issue
i setup gitlab-runner on k8s cluster with helm values like below
## How many runner pods to launch.
##
## Note: Using more than one replica is not supported with a runnerToken. Use a runnerRegistrationToken
## to create multiple runner replicas.
replicas: 2
## Configure the maximum number of concurrent jobs
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
concurrent: 50
## Defines in seconds how often to check GitLab for a new builds
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
checkInterval: 2
the jobs failed because the unit test failed when try to connect service alias, but when another jobs done and i retry jobs failed, its normal and can finish
example i have jobs like below
job:
services:
- name: mysql5.7
alias: database
image: app-server:3.7
script:
- yarn migration
- yarn build
attempt 1
i try to increase replica on values.yaml helm like below
## How many runner pods to launch.
##
## Note: Using more than one replica is not supported with a runnerToken. Use a runnerRegistrationToken
## to create multiple runner replicas.
replicas: 10
## Configure the maximum number of concurrent jobs
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
concurrent: 10
## Defines in seconds how often to check GitLab for a new builds
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
checkInterval: 2
after upgrade values helm and apply config, my jobs failed decrease from 15 failed to only 3 failed jobs
do you know why gitlab runner can't handle many jobs at the same time
also i already test setup gitlab runner on vm with ubuntu os, same still issue exist when handle many many jobs at the same time
btw i use helm gitlab runner v0.37.2