Analyze the performance of Semaphore, CircleCI builds compared to GitLab.com Shared Runners
Overview
We need a solid understanding along with metrics that allows us to quickly answer the questions related to how does the performance of builds on GitLab Shared Runners compare to key competitors.
Example questions:
- How does our speed of running CI tests compare to CircleCI and Semaphore? It seems they both have done a lot to run tests fast out of the box. Maybe benchmark with a Rails app based on a Cloud Native Build Pack?
- https://github.blog/2020-10-29-making-github-ci-workflow-3x-faster/
Summary
- Placeholder for summary of performance testing performed to date.
- Placeholder for insights that explains the difference in performance for CI builds on certain technology stacks.
Comparison matrix of hosted CI build VM machine types (Linux)
VM configurations | GitLab | CircleCI | Semaphore |
---|---|---|---|
1 vCPU / 3.75 GB RAM | AVAILABLE | NOT AVAILABLE | NOT AVAILABLE |
2 vCPUs / 4 GB RAM | NOT AVAILABLE | NOT AVAILABLE | AVAILABLE |
2 vCPUs / 7.5 GB RAM | NOT AVAILABLE | AVAILABLE | NOT AVAILABLE |
4 vCPUs / 8 GB RAM | NOT AVAILABLE | NOT AVAILABLE | AVAILABLE |
4 vCPUs / 15 GB RAM | NOT AVAILABLE | AVAILABLE | NOT AVAILABLE |
8 vCPUs / 16 GB RAM | NOT AVAILABLE | NOT AVAILABLE | AVAILABLE |
8 vCPUs / 32 GB RAM | NOT AVAILABLE | AVAILABLE | NOT AVAILABLE |
16 vCPUs / 64 GB RAM | NOT AVAILABLE | AVAILABLE | NOT AVAILABLE |
WIP: Performance results tables
Project = Node.js:Cartus test application
CI Job | Avg. GitLab.com duration | Avg CircleCI duration |
---|---|---|
Prebuild | 1 min 36 s | 0 min 36 s |
Build | 1 min 30 s | 1 min 6 s |
Unit Test | 2 min 42 s | 1 min 36 s |
E2E Test | 2 min 36 s | 1 min 30 s |
Performance comparison charts:
Node.js application (GitLab, CircleCI)
-
The first application chosen for the initial comparison tests with CircleCI is a Node.js application used by a prospective GitLab customer in comparing the performance of the hosted build fleets for CircleCI and GitLab. The pipelines were run on both platforms without any modifications to establish a baseline performance threshold. Therefore it is feasible that the pipelines can be optimized to improve performance. That type of analysis will need to be part of future test iterations.
-
The pipeline performance is, on average, 42.6% faster on CircleCI with a Docker executor, compared to the current GitLab Linux Runners on GitLab.com.
Node.js application (GitLab, Semaphore)
-
The first application chosen for the initial comparison tests with Semaphore is a Node.js sample application provided by Semaphore. The pipelines were run on both platforms without any modifications to establish a baseline performance threshold. Therefore it is feasible that the pipelines can be optimized to improve performance. That type of analysis will need to be part of future test iterations.
-
The pipeline performance is, on average, 48% faster on Semaphore running on an e1-standard-2 VM with Semaphore's ubuntu 1804 image, compared to the current GitLab Linux Runners on GitLab.com.