Skip to content

Status bar integration tests are a bit flakey

Summary

Today I've noticed the status bar integration test to fail for the second time in the last two months.

test-integration___963512965____Jobs___GitLab_org___gitlab-vscode-extension___GitLab

https://gitlab.com/gitlab-org/gitlab-vscode-extension/-/jobs/963512965

Re-running the test fixed the pipeline.

Since I introduced the tests 3 months ago (!97 (merged)) I've seen it flake out in CI three times (including today's failure). It happens more often when running the tests locally.

Steps to reproduce

Locally it's enough to run the integration test suite multiple times and it fails relatively often.

Possible fixes

The status bar code revolves around several global variables. These variables are getting manipulated by both the running extension and the test suite. This results in race conditions.

The proper fix is to make the variables local so the tests can use a different instance of our status_bar code. This is not trivial.

If the test starts flaking out more, the quick solution might be turning it off.

Edited by Tomas Vik