Submit exit code back to Rails when a job fails (!2562) · Merge requests · GitLab.org / gitlab-runner

Marius Bobin requested to merge submit-exit-code-back-to-rails into master Nov 11, 2020

What does this MR do?

In gitlab#273157 (closed) we want to allow users to control allow_failure functionality using the job's exit code:

test_job:
  script:
    - execute_script_that_will_fail
  allow_failure:
    exit_codes:
      - 137
      - 255

To do this, we need to capture and send the exit code back to Rails.

Why was this MR needed?

What's the best way to test this MR?

Kubernetes executors

Have the following .gitlab-ci.yml

.gitlab-ci.yml

image: busybox:latest

test1:
  variables:
    FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY: 1
  script:
    - exit 42

test2:
  variables:
    FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY: 0
  script:
    - exit 42

Configure a k8s runner

K8s runner configuration example

[[runners]]
  name = "k8s"
  url = "http://gdev.bobin.ro:3000/"
  token = "xxxxx"
  executor = "kubernetes"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.kubernetes]
    host = ""
    bearer_token_overwrite_allowed = false
    image = ""
    helper_image = "gitlab/gitlab-runner-helper:x86_64-af099ad9"
    namespace = ""
    namespace_overwrite_allowed = ""
    privileged = false
    service_account_overwrite_allowed = ""
    pod_annotations_overwrite_allowed = ""
    [runners.kubernetes.affinity]
    [runners.kubernetes.pod_security_context]
    [runners.kubernetes.volumes]

Tail the GitLab API log file using tail -f log/api_json.log | grep exit_code
Build executable for your arch and OS: make runner-and-helper-bin-host
Start the runner: out/binaries/gitlab-runner --debug run

The exit code should appear in the logs two times:

Example log entry

{"time":"2020-12-02T10:19:19.072Z","severity":"INFO","duration_s":0.0482,"db_duration_s":0.01883,"view_duration_s":0.02937,"status":200,"method":"PUT","path":"/api/v4/jobs/5818","params":[{"key":"info","value":{"name":"gitlab-runner","version":"13.7.0~beta.92.g0f7d48f1","revision":"0f7d48f1","platform":"darwin","architecture":"amd64","executor":"kubernetes","shell":"bash","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null}}},{"key":"token","value":"[FILTERED]"},{"key":"state","value":"failed"},{"key":"failure_reason","value":"script_failure"},{"key":"checksum","value":"crc32:ebc1bb65"},{"key":"output","value":"[FILTERED]"},{"key":"exit_code","value":42}],"host":"gdev.bobin.ro","remote_ip":"192.168.22.142","ua":"gitlab-runner 13.7.0~beta.92.g0f7d48f1 (submit-exit-code-back-to-rails; go1.15.5; darwin/amd64)","route":"/api/:version/jobs/:id","queue_duration_s":0.281487,"redis_calls":5,"redis_duration_s":0.001596,"redis_read_bytes":379,"redis_write_bytes":3371,"redis_cache_calls":2,"redis_cache_duration_s":0.001093,"redis_cache_read_bytes":205,"redis_cache_write_bytes":168,"redis_queues_calls":3,"redis_queues_duration_s":0.000503,"redis_queues_read_bytes":174,"redis_queues_write_bytes":3203,"correlation_id":"01ERHFD9TNE82WJ7EPSACKS4TV","meta.user":"root","meta.project":"root/k8s-exec","meta.root_namespace":"root","meta.caller_id":"/api/:version/jobs/:id","meta.feature_category":"continuous_integration","content_length":"763"}

Note: when using attach strategy to execute scripts the job is failing with a timeout error: ERROR: Job failed (system failure): prepare environment: waiting for pod running: timed out waiting for pod to start. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information I think this is related to my config.

Docker executor

Have the following .gitlab-ci.yml

.gitlab-ci.yml

image: busybox:latest

test1:
  script:
    - exit 42

Configure a Docker runner

Docker runner configuration example

[[runners]]
  name = "gdev.bobin.ro"
  url = "http://gdev.bobin.ro:3000/"
  token = "xxxxxx"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.docker]
    tls_verify = false
    image = "alpine"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

Tail the GitLab API log file using tail -f log/api_json.log | grep exit_code
Build executable for your arch and OS: make runner-and-helper-bin-host
Start the runner: out/binaries/gitlab-runner --debug run
The exit code should appear in the logs:
Example log entry

Shell executor

Have the following .gitlab-ci.yml
.gitlab-ci.yml
```
test1:
  script:
    - exit 42
```

Configure a shell runner

Shell runner configuration example

[[runners]]
  name = "shell"
  url = "http://gdev.bobin.ro:3000/"
  token = "xxxxx"
  executor = "shell"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

Tail the GitLab API log file using tail -f log/api_json.log | grep exit_code
Build executable for your arch and OS: make runner-and-helper-bin-host
Start the runner: out/binaries/gitlab-runner --debug run

The exit code should appear in the logs:

Example log entry

{"time":"2020-12-02T11:26:18.747Z","severity":"INFO","duration_s":0.0899,"db_duration_s":0.05391,"view_duration_s":0.03599,"status":200,"method":"PUT","path":"/api/v4/jobs/5824","params":[{"key":"info","value":{"name":"gitlab-runner","version":"13.7.0~beta.92.g0f7d48f1","revision":"0f7d48f1","platform":"darwin","architecture":"amd64","executor":"shell","shell":"bash","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null}}},{"key":"token","value":"[FILTERED]"},{"key":"state","value":"failed"},{"key":"failure_reason","value":"script_failure"},{"key":"checksum","value":"crc32:13e5809a"},{"key":"output","value":"[FILTERED]"},{"key":"exit_code","value":42}],"host":"gdev.bobin.ro","remote_ip":"192.168.22.142","ua":"gitlab-runner 13.7.0~beta.92.g0f7d48f1 (submit-exit-code-back-to-rails; go1.15.5; darwin/amd64)","route":"/api/:version/jobs/:id","queue_duration_s":0.23975,"redis_calls":4,"redis_duration_s":0.000912,"redis_read_bytes":176,"redis_write_bytes":3314,"redis_cache_calls":1,"redis_cache_duration_s":0.00027,"redis_cache_read_bytes":2,"redis_cache_write_bytes":112,"redis_queues_calls":3,"redis_queues_duration_s":0.000642,"redis_queues_read_bytes":174,"redis_queues_write_bytes":3202,"correlation_id":"01ERHK7Z9HJ3Q42B3JNNTW9MXY","meta.user":"root","meta.project":"root/k8s-exec","meta.root_namespace":"root","meta.caller_id":"/api/:version/jobs/:id","meta.feature_category":"continuous_integration","content_length":"760"}

SSH executor

Have the following .gitlab-ci.yml
.gitlab-ci.yml
```
test1:
  script:
    - exit 42
```

Configure a shell runner

Shell runner configuration example

[[runners]]
  name = "ssh"
  url = "http://gdev.bobin.ro:3000/"
  token = "xxxxxx"
  executor = "ssh"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.ssh]
    user = "root"
    host = "gdev.bobin.ro"
    port = "2222"
    identity_file = "/Users/marius/.ssh/id_ed25519"

Start a Docker container with SSH service enabled: docker run -it --rm -p 2222:22 -e SSH_ENABLE_ROOT=true -v ${HOME}/.ssh/id_ed25519.pub:/root/.ssh/authorized_keys:ro docker.io/panubo/sshd:1.3.0
Tail the GitLab API log file using tail -f log/api_json.log | grep exit_code
Build executable for your arch and OS: make runner-and-helper-bin-host
Start the runner: out/binaries/gitlab-runner --debug run

The exit code should appear in the logs:

Example log entry

{"time":"2020-12-02T11:35:30.150Z","severity":"INFO","duration_s":0.04323,"db_duration_s":0.01711,"view_duration_s":0.02612,"status":200,"method":"PUT","path":"/api/v4/jobs/5825","params":[{"key":"info","value":{"name":"gitlab-runner","version":"13.7.0~beta.92.g0f7d48f1","revision":"0f7d48f1","platform":"darwin","architecture":"amd64","executor":"ssh","shell":"bash","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null}}},{"key":"token","value":"[FILTERED]"},{"key":"state","value":"failed"},{"key":"failure_reason","value":"script_failure"},{"key":"checksum","value":"crc32:4ababebe"},{"key":"output","value":"[FILTERED]"},{"key":"exit_code","value":42}],"host":"gdev.bobin.ro","remote_ip":"192.168.22.142","ua":"gitlab-runner 13.7.0~beta.92.g0f7d48f1 (submit-exit-code-back-to-rails; go1.15.5; darwin/amd64)","route":"/api/:version/jobs/:id","queue_duration_s":0.236205,"redis_calls":4,"redis_duration_s":0.000761,"redis_read_bytes":176,"redis_write_bytes":3314,"redis_cache_calls":1,"redis_cache_duration_s":0.00028,"redis_cache_read_bytes":2,"redis_cache_write_bytes":112,"redis_queues_calls":3,"redis_queues_duration_s":0.000481,"redis_queues_read_bytes":174,"redis_queues_write_bytes":3202,"correlation_id":"01ERHKRSTE7DMX1JF027FKCV81","meta.user":"root","meta.project":"root/k8s-exec","meta.root_namespace":"root","meta.caller_id":"/api/:version/jobs/:id","meta.feature_category":"continuous_integration","content_length":"760"}

What are the relevant issue numbers?

gitlab#273157 (closed)

Edited Dec 02, 2020 by Marius Bobin

Submit exit code back to Rails when a job fails

What does this MR do?

Why was this MR needed?

What's the best way to test this MR?

Kubernetes executors

Docker executor

Shell executor

SSH executor

What are the relevant issue numbers?

Merge request reports