Skip to content

Ensure after_script runs even after E2E test job timeout

Jay McCurerequested to merge
jmc-e2e-job-timeout into master

What does this MR do and why?

Resolves gitlab-org/quality/quality-engineering/team-tasks#3823 (closed)

This MR aims to capture artifacts even when E2E test jobs timeout, so we can look at application logs which may help in figuring out why a job timed out. An example of this is detailed in E2E Test jobs against GDK timeout due to RestCl... (#558136) for GDK tests, and an example for CNG tests in gitlab-org/quality/quality-engineering/team-tasks#3849.

It does this by setting Set RUNNER_SCRIPT_TIMEOUT and RUNNER_AFTER_SCRIPT_TIMEOUT environment variables. https://docs.gitlab.com/ci/runners/configure_runners/#set-script-and-after_script-timeouts

Note: i needed some additional details from the runner team to realise the start-up time needed to be factored in, see: gitlab-org/quality/quality-engineering/team-tasks#3823 (comment 2708417854)

CNG jobs:

Overall timeout of 30m hasn't changed, but now the step_script will timeout if it takes longer than 26m (RUNNER_SCRIPT_TIMEOUT), which will give the after_script some time to run. A startup time of 2m has been factored in, hence the overall timeout of 30m.

    RUNNER_SCRIPT_TIMEOUT: 26m # This does not include the script startup time
    RUNNER_AFTER_SCRIPT_TIMEOUT: 2m
  timeout: 30m

GDK jobs:

Overall timeout set by the project was previously 90m, which seems excessive, so I have reduced to 60m, and given the RUNNER_SCRIPT_TIMEOUT 50m, after_script 5m and the startup 5m

    RUNNER_SCRIPT_TIMEOUT: 50m # This does not include the script startup time
    RUNNER_AFTER_SCRIPT_TIMEOUT: 5m
  timeout: 60m

Example

In the following GDK job, I ran with these parameters to simulate a timeout:

    RUNNER_SCRIPT_TIMEOUT: 10m # not enough time
    RUNNER_AFTER_SCRIPT_TIMEOUT: 5m
  timeout: 20m

On this line we see that the job timed out 10m after the step_script started, and the after_script still ran and captured app logs: https://gitlab.com/gitlab-org/gitlab/-/jobs/11133344923#L1103

References

https://docs.gitlab.com/ci/runners/configure_runners/#set-script-and-after_script-timeouts

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Jay McCure

Merge request reports