Possible improvements for Flaky tests issues generation
Context
This is to discuss some things I noticed while working on gitlab-org/gitlab#473397 (comment 2007960164) (from which gitlab-org/gitlab!160578 (merged) came about).
Things I noticed that I'd like to discuss
- If we have a test issue with many statuses on it (e.g. flakiness, slowness), and say a spec wasn’t flaky for a month or so, we should remove the flakiness label on it.
- Closing the issue doesn't make much sense if we have other statuses on it.
- We cannot add metadata to
include_examples(see gitlab-org/gitlab!150260 (merged)), we need to add it to a context around it. - Similarly, we cannot add metadata to
it_behaves_like(see gitlab-org/gitlab!151975 (merged)), we need to add it to a context around it. - Are we adding the quarantine label when a quarantine MR is merged? (e.g. gitlab-org/gitlab#444873 (comment 2014167679))
- There are some faulty flakiness1 labels applied (e.g. gitlab-org/gitlab#444722 (closed)). I think if we have some logic to revisit the label after a certain period, it will help
- I think it would be great to be able to do those manual quarantining automatic. What I ended up doing was when I detected a
it_behaves_likeor a parameterized rspec, I created a context block around it. It might be really hard to do for parameterized rspec, but it should be doable forit_behaves_liketo locate the closest surroundingcontextblock...Not sure.
Edited by David Dieulivol