Test run object (or issue verification object generally)

A new object called a test run or a test execution or a feature verification.

The object would probably be created initially via the API in the beginning to support automated tests. But in the future it could be created manually via Web UI to support manual testing.

When an automated test is run by a pipeline, the pipeline script could call an API endpoint and pass this information to GitLab, creating the test run object.

• The pipeline itself. So pipeline ID.
• Zero, one, or multiple issue IDs. These issues represent feature(s), requirements, or test cases that the automated test is testing.
• Some test result. Pass, fail.
• A reason code or a free text field giving more context.

So GitLab will have a bunch of these test run objects in its database. It would serve these purposes:

• A feature to export the data to CSV that engineers could use to analyze the data.
• The test run objects would appear in some UI at the project and or group level so you can see it in a list view and sort and filter and order on.
• The test run objects would appear in associated issues. So if you look at a given issue page, you could see test runs that have been made against it, and click through to see the pipeline that created the test run.

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.

added Plan [DEPRECATED] label

changed the description

@meks @mlapierre @at.ramya : What do you think about this?

@jlenny @bikebilly : Any thoughts from a CI perspective? See https://gitlab.slack.com/archives/C3JJET4Q6/p1537408187000100

added GitLab Ultimate devopsplan direction labels

@victorwu sorry but I don't understand the broader context - this is a plan feature that would start tracking individual test cases as objects in the database? I feel like I've missed the start of the conversation so having a hard time catching up.

Hi @jlenny I think this issue might give some context - https://gitlab.com/gitlab-org/gitlab-ee/issues/1538. There are different views on how to incorporate test management in Gitlab.

Linking a couple of related issues here

marked this issue as related to #1538

removed the relation with #1538

marked this issue as related to #1538

So just to understand clearly:

• I develop a feature - I track it using an issue - say, "FeatureIssue".
• I have a Test Plan for this Feature, say "Test Plan Issue"
• I also have a TestRun Object created at the backend every time a "Test Plan Issue" is created.
  • What does the TestRun Object contain?
    • It should allow me to choose multiple TestSuites/TestCases to be run as part of this test plan.
    • Which would mean we should also have a TestCase Object and TestSuite Object defined.
    • And these objects will be used while writing automation tests - we should be exposing an API to instantiate and use these objects. We can even have a UI in groups/projects page showing this list
    • This could be a replacement for the Google docs which we have today to maintain testcases - This would help in testcase management
• The TestRun Object is populated with the list of tests to be run (to test my feature) from the "Test Plan Issues" UI.
• Once the Feature Issue's MR is merged (or we should have a better way to convey whether development of my feature is done) and when it goes through the CI jobs, a test job is run which runs the list of tests in the TestRun Object.
• The results are also stored in the TestRun's objects.
• The TestRun's page's link is again linked back with the Issue's page.(which has all the results/reports)
• If all tests are green, the feature could be auto-deployed! The "Feature Issue" and "TestPlan Issue" would be closed. (or can have a different state saying "DEPLOYED"?)
• If tests fail, the issues remain as such until fixed.

I'm just thinking out loud, Please feel free to disagree or propose different ideas!

@victorwu I like the idea! It would increase quality test automation for our internal process.

I think pipelines are "generic" enough to support this flow if we use API. What I cannot see now is how to link the pipeline (that is linked to a commit) back to an issue. If this can be managed by something external to the pipeline itself (e.g., environment variables), you can implement a tool/docker image that can achieve whatever communication you need.

Decoupling the testrun object and logic from the pipeline could be simpler to iterate on.

added Quality label

Looks good. Thanks @victorwu! This is something we in Quality really need to get a better view of test results.

Zero, one, or multiple issue IDs.

Only issues? Or issues and/or MRs? I assume both because test execution is currently triggered in MRs. It would be handy to have info about test runs in issues too, but it's essential in MRs.

These issues represent feature(s), requirements, or test cases that the automated test is testing.

It should allow me to choose multiple TestSuites/TestCases to be run as part of this test plan.

Some test result. Pass, fail.

I think it's worth highlighting that there should be an overall test result as well as results of individual tests. For example, it would be very helpful to be able to expose details of individual failing test cases in an MR instead of just seeing that a pipeline failed and then having to drill down into the pipeline to find out what test(s) failed and why.

The pipeline itself. So pipeline ID.

QA tests executed by the package-and-qa job currently involve multiple pipelines but only the final one includes the jobs that execute the tests. But any of the pipelines might fail and need to be investigated even though only one contains tests. So there might have to be multiple pipeline IDs. And also job IDs?

/cc @sliaquat

This is a good conversation starter for tracking test runs that would ultimately come in handy for #1538

We would probably have to call the API from the test suites as well and they would have information about individual tests and pass/fail statuses. I presume the pipeline would create the test run object and the suites would populate them in the required info. Coupling of tests with one or more issues would also be required to create that relation.

mentioned in issue #7799 (closed)

@mlapierre: So you think seeing test runs in mrs first would be a better first iteration?

@bikebilly:

Few ways I’m thinking about it:

If code tests are deeply associated with GitLab, then this would be the best experience. This means it would be language dependent. (So this would be similar to SAST in that GitLab needs to understand actual source code right?) So if GitLab understood, say Ruby rspec, then if a test failed, you could associate that failure back to the merge request directly in GitLab right? So I suppose this would be a big investment and essentially would be TDD integrated in GitLab right?

Another approach is more decoupling like you suggested. So would there be any way for a test to tie back to an mr? Like I’m thinking a test file or a commit or some way to identify a code test with some id is passed to the job/pipeline. When there is a failure, the id and the mr id is sent to GitLab via API call and the association can be saved to the database. But it seems messy (impossible? to surface that info to the pipeline/job level? When a job runs, can results from the code be returned back to it? (Environment vars allow you to pass info. Is there a way to get info back?)

mentioned in issue #1538

added [deprecated] Accepting merge requests label

Anyone thought of partnering with a test orchestration product like http://robotframework.org/?

added ~8299320 Category:Quality Management labels

mentioned in issue gitlab-qa#345 (closed)

added 1 deleted label

added Enterprise Edition label

@victorwu Is this test run objects only for automation tetsing? Can we achieve same for manual testing? Also, I have below queries that could me to better the understand the Gitlab from testing perspective.

1. Can I create test case, defects in Gitlab?
2. Do I need to use any other tool to create test case and defects?
3. Can we generate test reports and requirements traceability reports for manual tests?

I am interested to have test management inside the Gitlab linking the respective issues. Let me know if this works, any suggestions would be great.

Thanks.

removed Plan [DEPRECATED] label

changed the description

added sectiondev label

added groupproduct planning label and removed 1 deleted label

added 1 deleted label

removed groupproduct planning label

changed the description

added groupproduct planning label and removed 1 deleted label