Integrated Load Testing MVC

Customer Problem

Load testing is a common and typically important stage in the CI process, ensuring changes to the app can scale before they are introduced into production.

Advanced deployment strategies like Canary and Blue/Green can mitigate the risk for deploying code that does not scale, however that is not a true alternative for a few reasons:

• You can't improve what you don't measure. Running repeatable and consistent performance tests lets you compare results between runs, to ensure performance doesn't slowly degrade over time.
• Canary and Blue/Green do minimize risk, but you are still essentially testing in production. Best practices would dictate that you use CI test what you can, to reduce the frequency and occurrence of rollbacks and errors in production.

Overview

There are a number of open source load testing tools, for example:

• Locust
• Gatling
• JMeter
• k6
• Artillery
• ... and more

While we should eventually support the top few load testing tool output formats, we should test and select one for further integration with gitLab. With our integration, we could automatically configure and run the load tests as part of Auto DevOps.

Our Quality group has begun to use Artillery.io to perform load testing against GitLab itself here: https://gitlab.com/gitlab-org/gitlab-ce/issues/57913.

Proposal

• Automatically generate a load testing CI job against /
• Allow customers to specify additional URL's to test (https://gitlab.com/gitlab-org/gitlab-ee/issues/3540)
• Utilize the existing review environment, or one customer specifies
• Archive and persist the performance results
• Allow for customers to specify their own load test scripts
• Display the results on the Merge Request page, similar to Performance, although we will indicate the source of the performance metrics (Load or Browser test)

The value of this being integrated with GitLab is to able to clearly show performance changes using a repeatable and consistent test, at the time of deciding whether to merge. This simplifies the developer workflow, by reducing the number of tools a reviewer needs to consult or look at. (Similar to CodeClimate)

In the outputted report type, we should create a generic interface that will be agnostic to the specific open source tool we select such that users can also "bring their own" testing suite to this feature.

Out of Scope for MVC

• Adding to Auto DevOps: #10681 (closed)
• Adding support for HAR archives: #10682 (closed)

Thoughts @markpundsack? This may be an interesting addition to our Auto/Integrated DevOps story.

closed

reopened

mentioned in issue #3046 (closed)

I'm a for this, just be careful when adding it to our GitLab.com story as this is the perfect tool for a malicious mind to setup and have GitLab.com DOS someone.

cc @marin for review, k6 is licensed under AGPL 3.0. We would be leveraging as we do with CodeClimate:

1. We'd provide instructions and perhaps samples on how to run run job
2. We'd archive the JSON output as an artifact
3. We'd then parse then JSON to display on the MR widget

Note that CodeClimate also appears to be AGPL 3.0: https://github.com/codeclimate/codeclimate

We are only reaching out to the outside codebase so license of k6 should not matter :)

@joshlambert Things get fuzzier when you consider Auto DevOps wouldn't just provide instructions, but run a vendored .gitlab-ci.yml. We're still not including the codebase in our codebase, but honestly I don't know the full legal ramifications.

I posted comments in https://gitlab.com/gitlab-org/gitlab-ce/issues/32841#note_36390800. @joshlambert How do the two issues relate?

The proposal setup is still too manual. It would be awesome to have a smart default, even if it just hit /. The Code Climate integration works well because it has it's own init function.

mentioned in issue gitlab-com/blog-posts#359 (moved)

changed the description

@markpundsack I will rework the meta issue to be focused on the broader goal of pre-production performance testing, of which this is a part.

I also added additional detail on the automation side in the description, here: https://gitlab.com/gitlab-org/gitlab-ee/issues/3046

Let me know what you think.

mentioned in issue #3173 (closed)

added GitLab Premium label

mentioned in issue #3307 (closed)

changed milestone to %10.4

mentioned in issue #3534 (closed)

changed the description

changed the description

mentioned in issue #3554

FYI there's taurus which supports yml configuration similar to gitlab ci http://gettaurus.org

Microsoft offers this https://www.visualstudio.com/team-services/cloud-load-testing/

@joshlambert is it going to be Deliverable for %10.4? I suppose ~Prometheus team will pick this up.

removed milestone

changed milestone to %10.5

@bikebilly no, this will be in a later release. We have a good framework in place but need to identify the tool and provide documentation and integration with AutoDevOps.

mentioned in epic &118 (closed)

added to epic &118 (closed)

changed milestone to %11.0

mentioned in epic &168

@joshlambert I removed the ~"CI/CD" label to keep boards clean, but please feel free to ping me if I can be of any help!

added Stretch label

We are currently at the tool selection phase. Candidates:

k6

Pros:

• Modern, golang
• Includes CI/CD friendly features like thresholds for throwing errors
• ES6 scripting language. Makes it usable for both load tests but also functional testing.
• Friendly JSON output which we can easily convert to our performance.json format
• Can playback a HAR file

Cons:

• AGPLv3. Not an initial blocker as we simply invoke via CI, but could be problematic later if we have goals of making this a more fully integrated service without the user managing this themselves.

Gatling

Pros:

• Apache 2.0.
• Mature, maintained.

Cons:

• DSL for tests
• Written in Scala which we do not use here

Vegeta

Pros:

• MIT
• Written in golang. Very simple, very scalable.
• JSON output

Cons:

• Arguably too simple. Configurable target list is a flat text file of URL+Headers

Proposed recommendation

k6 seems to have the most comprehensive feature set, and is easily scriptable in ES6. It should also be possible to leverage common tests from the load test to on-going functional testing as well.

We can revisit a more integrated service oriented solution if/when we get there. But for now this seems like the best tool for the job of providing users with a great load testing tool, with which we can parse the output.

Note, as part of this we should extend our existing performance section to also include results the load-testing job. I don't think we need a separate section from browser tests, so we should just support muxing the JSON results from both jobs.

changed the description

changed milestone to %11.3

Also please note that by default k6 will send a usage report every time it is run: https://docs.k6.io/docs/usage-reports. The company who is developing it does not make this very clear.

Please don't implement this feature for a specific tool. There will always be someone who doesn't use the tool you choose. In general it's as easy as:

1. run your {whatever}-test with the tool of your choosing
2. the tool (should) return 0 on succes, anything else is not succes
3. based on the output, decide what to do with it.

Note, as part of this we should extend our existing performance section to also include results the load-testing job. I don't think we need a separate section from browser tests, so we should just support muxing the JSON results from both jobs.

This appears to need UX. Adding the label.

added UX label

changed milestone to %11.6

@teemo https://gitlab.my.salesforce.com/0016100000dG6oO

added devopsmonitor + 1 deleted label

Ultimate customer who would love to see this features: https://gitlab.my.salesforce.com/0016100001UPZa4

removed Stretch + 1 deleted label

added [deprecated] Accepting merge requests + 1 deleted label

mentioned in issue #9735 (closed)

added 1st contribution devopsverify + 1 deleted label and removed devopsmonitor + 1 deleted label

added 2 deleted labels

marked #9735 (closed) as a duplicate of this issue

changed epic to &514 (closed)

added 2FA + 2 deleted labels

Talked with Andrew today to discuss today:

• https://github.com/newsapps/beeswithmachineguns
• Would Serverless / knative be leverageable here
• Or a swarm of runners for any environment to become the
• Apache Bench / other things that just point at ONE URL is a very limited test
• What you really need is a random load test against multiple/random endpoints
• The most minimal thing that would help @Andrew is at the end of a CI run, if he could post a generic metric which is help and stored somewhere (repo, Prometheus, what have you). As simple as go test -bench. To see that overtime would be amazing.

What if we implement this first: https://gitlab.com/gitlab-org/gitlab-ee/issues/1020 then build the idea of load testing over time on top of that. Would allow users to use k6 or whatever we choose, OR bring their own tool (say they have exisiting Jmeter tests)

So we've split #1020 (closed) into a few steps:

1. First #9788 (closed) which will allow for generic metrics to be reported in an open format to the merge request
2. That will enable both this issue to write those types of metrics, as well as #1020 (closed) to then graph those metrics over time.

changed milestone to %12.0

In addition to K6 there is also https://artillery.io/, their community edition is OSS.

/cc @at.ramya

/cc @sliaquat

removed 2 deleted labels

mentioned in epic &514 (closed)

A review on multiple open source load testing tools written by the creators of k6: https://blog.loadimpact.com/open-source-load-testing-tool-review

It is from Feb 2017 so some statements might not hold true any more.

mentioned in issue #10681 (closed)

mentioned in issue #10682 (closed)

changed title from Integrated Load Testing to Integrated Load Testing MVC

changed the description

changed the description

promoted to epic &952 (closed)

This was moved to epic &952 (closed) and the Artillery.io integration will be addressed in #10683 (closed)

removed direction label

removed from epic &514 (closed)

removed milestone

added Enterprise Edition label

added 1 deleted label

removed 1 deleted label