Integrate k6 for Integrated Testing
Problem to solve
As the first step for the Integrated Load Testing MVC, we should integrate k6 which is already used by the Quality team for performing load testing against GitLab itself.
Intended users
The intended users would include Personas such as Devon, DevOps Engineer as well as the internal customer of the Quality team.
Further details
This contributes to our vision for advancing the ~"Category:Load Testing" to minimal
Plan
Per https://gitlab.slack.com/archives/C3MAZRM8W/p1583188543124000
- Quality @gl-quality/enablement-qe will take on moving this feature to maturityminimal we will lean on the Backend Engineer for guidance.
- Scope the work to backend (minimal ui work)
- Have a new performance/load type configuration driven by the existing
gitlab-ci.yml
- YAML file takes in k6 configuration as desired and points to test script.
- We will take the learnings from the original proposal and see where we can fit this into maturityminimal
- Sid: @james Heimbuck do you think Mek his group can contribute this feature? Cc @mek
👍 - James Heimbuck: I’m sure we can build on some of the work they have done for the performance testing tool. i think we can have more impact on IACV by using that time to focus on other categories though. *Sid: @james Heimbuck that makes sense. @mek when you get back maybe you can comment on the feasibility of the quality group adding k6 in a minimal form to auto DevOps
- mek: @james Heimbuck @sid I think this is feasible, as long as its Ruby code, we can pick it up and contribute. We know our usecase and have a good working relationship with the K6 folks. We will likely scope the work to backend (minimal ui work). E.g. a new performance job type configuration driven by the existing gitlab-ci.yml file that can load test given endpoints/urls. If help is needed we can lean on Backend engineers for guidance.
- Sid @mek that would be awesome!
Previous Proposal
- Integrate into the Custom Metrics widget to display specific k6 based metrics.
- Have the user specify URL, count and N value just as we do for browser performance testing
load-testing:
stage: performance
image: 8.15.1-slim
variables:
URL: https://example.com
COUNT: 10
NTIMES: 20
script:
- docker pull loadimpact/k6:0.25.1
- k6 run -u 10 -d 30s script.js
artifacts:
reports:
load: out.json
- TBD how to use the k6 docker image
- Do we need to transform the out.json somehow?
- TBD how k6 results look.
Example k6 output
{
"type":"Metric",
"data":{
"type":"gauge",
"contains":"default",
"tainted":null,
"thresholds":[
],
"submetrics":null
},
"metric":"vus"
}{
"type":"Point",
"data":{
"time":"2017-05-09T14:34:45.625742514+02:00",
"value":5,
"tags":null
},
"metric":"vus"
}{
"type":"Metric",
"data":{
"type":"trend",
"contains":"time",
"tainted":null,
"thresholds":[
"avg\u003c1000"
],
"submetrics":null
},
"metric":"http_req_duration"
}{
"type":"Point",
"data":{
"time":"2017-05-09T14:34:45.239531499+02:00",
"value":459.865729,
"tags":{
"group":"::my group::json",
"method":"GET",
"status":"200",
"url":"https://httpbin.org/get"
}
},
"metric":"http_req_duration"
}
Permissions and Security
The feature will not introduce any new concepts to the permission and security model - which will be the same as it is for any aspect of the .gitlab-ci.yml
Documentation
Will require a new documentation section for Integrated Load Testing.
Testing
For
- Additional unit/integration level tests
✅ - Good documentation
✅ - package-and-qa not necessary but captured E2E testing requirement
Risks
- Most significant risk is possibility of shared runners unable to handle the load. This is called out specifically in the added documentation. It is expected most users who desire to enable load testing will have the experience to account for this and take note of this call out in the documentation to address it appropriately through runner setup.
What does success look like, and how can we measure that?
This issue will measure usage: https://gitlab.com/gitlab-org/gitlab-ee/issues/10687
What is the type of buyer?
The same buyer as the rest of ~"performance testing" (Director of Release Management, Director of DevOps, etc.) thus this feature is included in GitLab Premium