Switch from Artillery to k6
After some investigation it's been determined that Artillery will not be viable for us moving forward due to several reasons but primarily due to it's own low performance.
This started due to us finding that Artillery wasn't able to hit 200rps in our gitlab runners without maxing the CPU and required spinning up 10 separate runners running Artillery at 20rps each. This was a surprise and a problem as it skews the results and 200rps is relatively a low amount. As such, an investigation was undertaken and found the following:
- Artillery is single threaded by default and runs on nodejs. It complains of high cpu limits in our gitlab runners when trying to run at 200 rps, which in general terms isn't very high. Currently to hit this RPS we need to spin up 10 separate runners.
- Online comparisons of the tool have found it to be one of the worst performing (see below image).
- It's DSL is limited (yaml) and will start to impede us when we try to implement more realistic scenarios.
- The docs are limited and missing. For example there's no docs on the Artilley report command.
As such a replacement was sought and for this, I used the following criteria:
- It should be a very performant tool that can hit a good RPS
- It should be written in a performant and modern language
- It should have a full scripting language for writing realistic scenarios
- It should have good documentation
- It would be nice to have a tool that can output results to useful backends (e.g. grafana)
- It would be nice to an official Docker image as an option when designing CIs
For this we looked for good comparison articles of all available load test tools and found this one - https://blog.loadimpact.com/open-source-load-testing-tool-benchmarks-v2. A key finding from this comparision was Artillery is essentially the second worst performing tool on the market by some margin:
Going through the tools available I've selected k6
as the preferred replacement due to it being written in Go (engine, wanted to avoid Java and Python) and JS (scripting language). It's well reviewed, actively maintained and popular. Importantly it's magnitudes more performant than Artillery. In some POCs I was easily able to hit 200rps with the GitLab runners and in a further test was able to hit 500rps (which no environment can currently handle.
Based on all the above I'm recommending that we switch to k6 as a priority. This task is to do just that - replace Artillery, our scripting and scenarios with k6.