Backend work for Pipeline Filter - Pipeline Status, Tag Name
In order to make frontend work #217617 (closed), we need to have backend side of this.
We already have status
and ref
filter in Ci::PipelinesFinder
, so we only need to pass this param from Projects::PipelinesController
and write tests.
-
the cost of pagination seems super big, as getting filtered json
for my user takes around 8s: This takes around 7s to execute: https://gitlab.com/gitlab-org/gitlab/pipelines.json?scope=all&page=1&username=ayufan- added an index, but still much more to improve.
-
this was made as select count(*)
is an expensive operation that does not scale very well with so frequently used system as pipelines -
it might be good idea to see how indexes scale, and decide if we need to add index to increase cardinality of queries that would ensure them to be fast - pipelines are mostly system-generated operations based on a common event, like a push, which are significantly more frequent than issues or merge request
- issues or merge requests are events that NEEDS to be created by user, thus amount of them is one-two-three magnitudes smaller
-
if we start filtering just by status
it will not work very well :) asstatus=success
forgitlab
will result in 100_000 pipelines likely being matched and SQL queries timing out on trying to runselect count(*)
and this in the past was a significant performance problem -
I think we should do consider making next page pagination on much smaller limit than currently configured to have significantly better query times - I don't understand why we don't have pagination header on: https://gitlab.com/gitlab-org/gitlab/pipelines.json?scope=all&page=1, but we have on https://gitlab.com/gitlab-org/gitlab/pipelines.json?scope=all&page=1&username=ayufan. I would expect both of them to have just next-page pagination :)
- I think as for pagination everything is fine here. There’s something else in play that decides whether to use next-page vs full pagination. It just seems to not work super well on this page, this is why high filtering duraiton
🙂 - It is likely a result of
lib/gitlab/pagination/offset_pagination.rb
. I wish we could make this limit of 10k to be smaller, to be 1k to improve the performance of this page. - in the past we always truncated counters and introduced next page pagination only
- truncated counters allow to have a predictable performance as we limit amount of data being processed
- truncated counters is also based on assumption that user do not care about very old data, data older than some threshold, like 100-1000 pipelines
Edited by Furkan Ayhan