Proposal to remove QA issue checkboxes and at-mentions
In #830 (closed) we discussed when to generate QA issues, and that we want to generate them more than we do now. In particular Quality/Engineering Productivity would benefit from this, as stated in #830 (comment 392027355).
But generating more QA issues in the current state will be annoying. For RCs and backports we'd generate a QA issue and ask developers to test their changes, without providing a suitable environment to do so (RCs technically could be tested, but they are deployed to a dedicated environment different from staging and production).
QA issues not being that useful was also brought up over a year ago in #120 (closed). During that issue we also discussed removing the checkboxes, and not at-mentioning developers.
I propose the following:
We remove the checkboxes, and change the at-mentions to "soft" mentions by wrapping them in backticks. This way they don't trigger TODOs. We then generate these change logs (as they are no longer QA issues) for every version we release/deploy, including patch releases and RCs. Since they are no longer release tasks, we could do so in a separate project if desired.
At first sight this may seem a problem for auto-deploys. Not generating QA issues may seem to result in us not catching any big regressions. To some extend this is true: if we no longer require testing in staging, developers may not discover problems until we are already deploying to production.
But this is already a problem today, as not everybody performs their testing (in a timely manner). This will also get worse as we start deploying more frequently. A QA time window works if the interval between deploys is large enough (e.g. six hours of a day). It does not work if we start deploying multiple times per day. In short, manual QA does not scale, and I believe it's time we look elsewhere. Automated QA issues and better unit tests are likely the best solution to this problem.