Quality Management MVC - Autogenerate test cases
Problem to solve
Users would like to be able to manage testing in a more integrated way within GitLab. This include organizing their testing into logical groupings,
User experience goal
The user should be able to define a structured test approach within GitLab and execute to that approach fluidly, using existing and familiar constructs.
Software Engineer in Test - SET
As a test engineer, I need to be able to execute test sessions on an aperiodic (on demand) basis.
As a test engineer, I need to be able to organize my tests in logical ways such that I can easily understand the completion status and pass / fail status.
As a release manager, I need to know how much testing has been completed against the current product iteration so I can make an informed decision as to whether or not I can release the product.
It is important to agree on definitions in order to allow for clear communication.
Test Suite - A logical grouping of tests. These could be grouped by function, by test needs (testing against code vs. testing on a running instance), or by any other logical manner for the specific application.
Tests - A specific named test which is designed to be executed.
Test Session - A named execution of testing. For example, a series of testing that will occur prior to release. These are utilized to group testing into a single test cycle.
Create an MVC for Quality Management that will automatically generate issues for tests to be run. These issues will be organized into epics, which will represent test suites. Finally, these test suite epics will be put under a parent epic that will represent the test session.
In order to keep this minimal, the idea is to utilize a template to define test sessions, test suites, and tests. This will ideally result in relatively simple code to automate the creation of issues and epics, and link them together.
Allow users to define template files for test sessions. For example, there could be an
Application Release test session that is designed to run prior to release. This could contain all testing. There could also be a
Core Testing test session which would contain a smaller subset of testing designed to be run frequently that tests basic core functionality.
Within the GitLab interface, a user would select a specific template, and an instance name - and create a new test session based on that template.
The testing templates would be written in a simple markdown language (for example, yaml). Each template would be named according to the desired test session.
Within the template, user's could define test suites, and assign tests to each test-suite.
One possible example:
daily-session: session_name: "Example Daily Batch" suites: unit_tests: name: "Unit Tests" tests: - unit_test_1.tst - unit_test_2.tst - unit_test_3.tst application_tests: name: "Application Testing" tests: - app_test_1.tst - app_test_2.tst
Creating a new instance of this test session would produce the following:
- A parent Epic with the instance name provided
- Based on the template, child epics for each test suite
- Issues within these child epics with the name of each tests
graph LR test-session["Test Session: Example Test Session"] --> test-suite1["Test Suite: Unit Tests"] test-session --> test-suite2["Test Suite: Application Testing"] test-suite1 --> unit_test_1["Test: unit_test_1"] test-suite1 --> unit_test_2["Test: unit_test_2"] test-suite1 --> unit_test_3["Test: unit_test_3"] test-suite2 --> app_test_1["Test: app_test_1"] test-suite2 --> app_test_2["Test: app_test_2"] subgraph "Parent Epic" test-session end subgraph "Child Epics" test-suite1 test-suite2 end subgraph "Issues" unit_test_1 unit_test_2 unit_test_3 app_test_1 app_test_2 end
Scoped labels would be defined which would allow users to show progress and status:
Test Progress:Not Run
Test Status:Not Run
All child issues would get the default labels of
Test Progress:Not Run and
Test Status:Not Run applied.
This would allow test instances to be tracked at the epic level, and we could take advantage of the current epic tree view to see all child tasks. As testing progresses, results could be added to the test issues and the appropriate status can be selected. Once complete, the issue would be closed.
Completion of a test session would occur when all Test issues were closed, all test suite epics were closed, and the test session epic was closed.
Some options for expandability after the MVC would be:
- Automatically populating the labels for progress and status from the pipelines
- Allow status roll-up to see progress
- Track test statistics over time
Known problems to solve (to be updated as necessary)
How do we tag epics and issues which are
test relatedfor future use? We don't want to design a solution down the road to roll up metrics but not be able to get the data from previous iterations since we didn't properly tag or label the issues.
Permissions and Security
Need to ensure that proper application limits are put in place so people cannot create a huge configuration file and easily spam issues
This will be the MVC for Quality Management, so will therefore require documentation as follows:
- Define what quality management is to our users
- Document the above feature