Unable to register runners

Hey, when deploying the helm chart (v0.37.0), the runners don't seem to be able to register.

Some details:

  • The registration token itself seems fine since I'm able to use it locally to create a new runner.
  • Using a runner token directly gets further but I get a 404 when it tries to pull jobs.
  • I'm loading the token(s) via a kubernetes secrets, I'm able to print these out if I update the entrypoint script.
  • I'm running in an AWS EKS
  • I'm deploying via terraform (helm_release resource)

The error when trying to use the registration token is:

Runtime platform                                    arch=arm64 os=linux pid=22 revision=98daeee0 version=14.7.0
WARNING: Running in user-mode.                     
WARNING: The user-mode requires you to manually start builds processing: 
WARNING: $ gitlab-runner run                       
WARNING: Use sudo for system-mode:                 
WARNING: $ sudo gitlab-runner...                   
                                                   
ERROR: Registering runner... failed                 runner=io_NB6z4 status=GET https://gitlab.com/api/v4/runners: 401 Unauthorized
PANIC: Failed to register the runner. You may be having network problems.

I'm able to run curl -X POST 'https://gitlab.com/api/v4/runners?token=MY_TOKEN from the gitlab-runner pod to create a runner-token. When I include this in my k8s secret, I instead get:

Merging configuration from template file "/configmaps/config.template.toml" 
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded! 
Runtime platform                                    arch=arm64 os=linux pid=7 revision=98daeee0 version=14.7.0
Starting multi-runner from /home/gitlab-runner/.gitlab-runner/config.toml...  builds=0
WARNING: Running in user-mode.                     
WARNING: Use sudo for system-mode:                 
WARNING: $ sudo gitlab-runner...                   
                                                   
Configuration loaded                                builds=0
Metrics server listening                            address=:9252 builds=0
[session_server].listen_address not defined, session endpoints disabled  builds=0
WARNING: Checking for jobs... failed                runner=wo_wjk9N status=GET https://gitlab.com/api/v4/jobs/request: 404 Not Found ()
WARNING: Checking for jobs... failed                runner=wo_wjk9N status=GET https://gitlab.com/api/v4/jobs/request: 404 Not Found ()
WARNING: Checking for jobs... failed                runner=wo_wjk9N status=GET https://gitlab.com/api/v4/jobs/request: 404 Not Found ()

Am I doing something obviously wrong? How can I debug this further? Could there actually be a network issue? How would you suggest I test this?

Thanks for your help!