Use GitLab logs in tests
Problem
Sometimes GitLab's logs contain the solution to a problem we encounter in tests, or when troubleshooting tests.
For example, the only record of some async events is in the sidekiq logs. One case is when a project is indexed. Another is when a project's repository storage it moved. If the tests could access those logs it would make the tests more efficient and reliable than waiting for an API or UI change that may never happen.
And when we're troubleshooting tests, we often need to search through the logs to find an entry related to a failure. It would be better if we could capture and expose all the relevant logs with the failed tests so that we don't have to go searching.
Proposal
Let the test framework access GitLab's logs.
We'll need to access the logs in a way that's compatible with running tests locally and in CI. That includes:
- Running GitLab locally using GDK where logs are written to files and we have access to the filesystem
- Running GitLab locally or in CI using omnibus-gitlab where the instance is in a different container to the tests
- gitlab.com and staging.gitlab.com and where we don't have access to the filesystem, but where logs are available via kibana
Option 1
- Use ELK to capture and expose logs when testing against GDK or omnibus-gitlab
- Access logs the same way in all environments - via kibana
Cons: ELK is another dependency that will add complexity to our testing environments
Option 2
- Provide a common interface for tests to access logs, backed by different adapters for logs in files vs. logs in kibana
Cons: More complex test framework code to handle the different log sources
I think option 2 might be the better option because even though the test framework code would be more complex, it probably won't require more effort than managing an ELK stack locally and in CI.