Possible improvements for Flaky tests issues generation

Context

This is to discuss some things I noticed while working on gitlab-org/gitlab#473397 (comment 2007960164) (from which gitlab-org/gitlab!160578 (merged) came about).

Things I noticed that I'd like to discuss

  1. If we have a test issue with many statuses on it (e.g. flakiness, slowness), and say a spec wasn’t flaky for a month or so, we should remove the flakiness label on it.
    • Closing the issue doesn't make much sense if we have other statuses on it.
  2. We cannot add metadata to include_examples (see gitlab-org/gitlab!150260 (merged)), we need to add it to a context around it.
  3. Similarly, we cannot add metadata to it_behaves_like (see gitlab-org/gitlab!151975 (merged)), we need to add it to a context around it.
  4. Are we adding the quarantine label when a quarantine MR is merged? (e.g. gitlab-org/gitlab#444873 (comment 2014167679))
  5. There are some faulty flakiness1 labels applied (e.g. gitlab-org/gitlab#444722 (closed)). I think if we have some logic to revisit the label after a certain period, it will help
  6. I think it would be great to be able to do those manual quarantining automatic. What I ended up doing was when I detected a it_behaves_like or a parameterized rspec, I created a context block around it. It might be really hard to do for parameterized rspec, but it should be doable for it_behaves_like to locate the closest surrounding context block...Not sure.
Edited by David Dieulivol