Pipeline Authoring Testing Questions from 14/4/2021 Meeting
After listening to the discussion during our meeting this morning I think my original questions (below) have been superseded by the question of "How do we want to shape our test culture on this team?" That question may deserve its own issue, and should ultimately be a documented in the handbook under a heading like "How the Pipeline Authoring Team Ensures Quality". This question may also only be answerable in the aggregate, by answering many smaller questions. And, so, in that vein, I posit the questions from the meeting here for further discussion. I've tried to capture the responses in the meeting doc in the comments. Feel free to edit the comments if you feel you have been misrepresented.
do should we use to determine if an end to end test should be written?
For clarity, when I say "end to end tests" I mean the ruby/rspec tests in
gitlab/qa/qa/specs/features. During the meeting I implied some criteria we may want to consider:
- Number of bugs filed against a certain feature
- Evaluate how severe an error could affect our users and decide based on a risk/severity matrix
- How costly an error could be to our revenue stream
- Add more tests around features we intend to expand on in the future
- Security concerns
Quad Planning method of soliciting feedback for issues?Have we considered using the
On other teams I've worked for we use thelabel on issues to solicit feedback from an SET. How does the team feel about using this label/process ongoing?
Is there any place I should be focusing my attention, and in what capacity?
Open question to the team.
Risk Areas for 13.11
I’ve been reading the issues from 13.11 and have found some themes. This is the first step in creating a risk map, though I’m not necessarily going to do one unless we think it would be a good idea. Some high risk areas, based only on the issues in 13.11 seem to be:
- Performance: N+1 database calls
- Transitioning to GraphQL
- Pipeline yaml configuration parameters and syntax:
needsin conjunction with
manualsteps (prevent pipeline from blocking indefinitely),
is_required: false(combinations like this are semantically difficult for people to parse), preventing cycles in the DAG.