Skip to content

Expose runner limit error message on registration

What does this MR do?

This MR adds additional logic to decode and log the message present in the body of the /api/v4/runners response in the case where a Bad Request (400) status code is returned.

Why was this MR needed?

As noted in #27840, when we enabled the :ci_runner_limits FF globally some users started hitting the limits when registering a specific type of runner and were left wondering what had happened since the error message reported by gitlab-runner register is too generic (it just reported that "you may be having network problems"). When a plan limit is hit on GitLab, it returns an error in the body and a Bad Request (400) status code:

Runner type HTTP response body
group {"message":{"runner_namespaces.base":["Maximum number of ci registered group runners (50) exceeded"]}}
project {"message":{"runner_projects.base":["Maximum number of ci registered project runners (50) exceeded"]}}

In this MR, we now treat the 400 error in a special way and try to decode the JSON payload. If we find any of the expected fields, we use that in the log message in the status field. For that, I needed to slightly alter network/client.go so that it accepts more than one status code to decide when to decode the JSON response. I also created a GitLabClient.doJSONMulti for the same purpose (which is now called by the previous GitLabClient.doJSON for compatibility).

What's the best way to test this MR?

  1. On GitLab.com, create a new group named <your-user-name>-test-runner-limits.

  2. Enable the :ci_runner_limits FF for that group. For that, go to #production and type:

    /chatops run feature set --group=<your-user-name>-test-runner-limits ci_runner_limits true
  3. Build this branch:

    git switch pedropombeiro/27840/expose-runner-limit-error-message && make runner-bin-host
  4. Register 51 runners on the new group:

    Assuming an environment variable GROUP_RUNNERS_REGISTRATION_TOKEN set to the runners registration token in your group's (https://gitlab.com/groups/your-username-test-runner-limits/-/settings/ci_cd) Runner registration page:

    for i in $(seq 1 51); do \
        out/binaries/gitlab-runner register -config /tmp/config.test.toml \
                --non-interactive \
                --executor "shell" \
                --url "https://gitlab.com/" \
                --description "group test runner" \
                --tag-list "shell,gdk,mac,test" \
                --run-untagged="false" \
                --locked="false" \
                --access-level="not_protected" \
                --registration-token="$GROUP_RUNNERS_REGISTRATION_TOKEN"; \
    done

    Upon registering the 51st runner (assuming your account is on the free plan) you should see the following error message:

    image

  5. On GitLab.com, create a new project named test-project-runner-limits under the group you created earlier.

  6. Enable the :ci_runner_limits FF for that project. For that, go to #production and type:

    /chatops run feature set --project=<your-user-name>-test-runner-limits/test-project-runner-limits ci_runner_limits true
  7. Register 51 runners on the new project:

    Assuming an environment variable PROJECT_RUNNERS_REGISTRATION_TOKEN set to the runners registration token in your project's (https://gitlab.com/your-username-test-runner-limits/test-project-runner-limits/-/settings/ci_cd) Runner registration page:

    for i in $(seq 1 51); do \
        out/binaries/gitlab-runner register -config /tmp/config.test.toml \
                --non-interactive \
                --executor "shell" \
                --url "https://gitlab.com/" \
                --description "group test runner" \
                --tag-list "shell,gdk,mac,test" \
                --run-untagged="false" \
                --locked="false" \
                --access-level="not_protected" \
                --registration-token="$PROJECT_RUNNERS_REGISTRATION_TOKEN"; \
    done

    Upon registering the 51st runner (assuming your account is on the free plan) you should see the following error message:

    image

  8. Unregister all runners from this MR:

    out/binaries/gitlab-runner unregister --all-runners --config /tmp/config.test.toml

What are the relevant issue numbers?

Closes #27840

Edited by Pedro Pombeiro

Merge request reports