Find the job in a pipeline that contains the output for a particular test
Problem to solve
While trying to investigate a truth claim for https://gitlab.com/gitlab-org/gitlab-ce/issues/58956 , I found myself needing to check a pipeline on master, to see if a spec in it behaves identically to a pipeline on a branch or not.
So I have these two pipelines:
- Branch: https://gitlab.com/gitlab-org/gitlab-ce/pipelines/51771782
- Master: https://gitlab.com/gitlab-org/gitlab-ce/pipelines/51792482
In the branch,
rspec-pg 24 50 fails, and we suspect it failed due to the behaviour of a specific test (file). In the master pipeline, I want to find the
rspec job containing that test (file) so I can see whether the test run was truncated or not.
Problem is, to do that, my only recourse right now is to go through all 50 rspec jobs looking for that string in the build logs.
Developers for sure
In our case, we have junit-compatible test output, and so gitlab knows - somehow - which tests were run. However, I expect that's aggregated in a later phase of the pipeline, so doesn't have a link to the original job the spec was run in. So it's unlikely to help us here.
A more general solution might be to allow build logs to be searchable. Then I could just put the spec file name into a search box in the pipeline, and have it show me the matching jobs.
Obviously, this is quite expensive. We could implement it only for the scope of a single pipeline in CE (it's unlikely to be more expensive than
git grep in general), and then use elasticsearch to permit better, faster, and even group or global, search of build log contents, in EE.
Build logs are quite large and the number of builds a project has tends towards infinity, so we should perhaps consider only making some of them searchable. This is particularly a concern in the elasticsearch case - for CE, we don't have to build an index, so supporting search on arbitrary pipelines is free, as long as you can only search one pipeline at a time.
Permissions and Security
Build logs are potentially privileged if the project or pipelines are private or internal. Any search interface will have to be aware of this.
There's also a risk that this will make mistakes more discoverable - if you can search for a string like "secret token: ", then people undoubtedly will. I don't think that's a great argument against implementing it, but it is something to consider.
doc/user/project/pipelines will need enhancing to include the new functionality.
We'll probably want to add something to the search documentation if we go that route, too:
What does success look like, and how can we measure that?
If I want to find a specific test or string in an arbitrary pipeline, I now can, as if by magic ;)
Links / references
@jramsay @brendan here's a feature proposal that's a little difficult for me to categorise. Pipelines are definitely ~Verify, but is searching them ~Create? I've erred on the side of "no" but do feel free to change if you think that's wrong.