Explore performance testing of CI/CD (Pipelines, Jobs)

Explore how can we test pipelines and jobs performance in the consistent way. We can't automate registering Runners in our automated environment builder because it can't be done via API. As the first iteration we can explore how can we seed gitlabhq or horizontal projects with test data.

Update from 2024-07-24: Allow registration of runners via API (gitlab-org/gitlab#16319 - closed) is closed and needs verification if runner can be automated for test data generation. Assuming it can be automated Runner setup on our test environments, the biggest problem is that CI\CD at its core doesn't lend itself well to performance testing at all. CI\CD is async. When you schedule a job it will be picked up by a runner at the first opportunity - this may take 1 second or 1 hour. This means if we try and track job completion time or anything like that the results would be useless. In addition to this running the test at scale to exercise this endpoint will create up to tens of thousands of jobs that will directly lead to significant costs and a massive queue. Since this tool can be used by customers we need to avoid this at all costs really leaving us in a bit of a catch 22. (context)

But the bigger problem is, assuming we can automate Runner setup on our test environments, is that CI\CD at its core doesn't lend itself well to performance testing at all.

Some of the known slow and popular endpoints we should add:

API endpoint	Description	Why it should be added
GET /projects/:id/pipelines --> added ✅ web_project_pipelines.js	List project pipelines	called: very often(9.5% of the total calls), response time: fast (133.23)
GET /projects/:id/pipelines/:pipeline_id/jobs	Get a list of jobs for a pipeline	called: often (4%), response time: slow (1733.24)
GET /projects/:id/jobs/:job_id/trace	Get a trace of a specific job of a project	called: very often(32% of the total calls), response time: fast (181.39)
GET /projects/:id/jobs/:job_id	Get a single job of a project	called: often (7%), response time: not as fast(1018.9); related to Projects::JobsController#show method, which is called often(1.6%) and not very fast(PERC95=5920.13)
GET /projects/:id/jobs/:job_id/artifacts	Get the job’s artifacts zipped archive of a project	called: often (1.4%), response time: slow (1727.09)