Add tooling to periodically query Thanos
This adds tooling to periodically query Thanos and upload the query results into a Google Cloud Storage bucket.
The script to perform these operations is
scripts/perform-periodic-thanos-queries.rb. See it's --help output
for information about how to use it.
The script evaluates all files in
periodic-thanos-queries/*.queries.jsonnet and uses the result of
that to build requests to perform against thanos.GitLab.net.
The result of the requests is stored in the periodic-query-results
directory. Using the format defined in the README.
The queries are validated in jsonnet at evaluation time. This happens
in the test-periodic-queries CI job. The queries are not run against
a Prometheus server or stored anywhere at this time. So we have not
checked if the queries are actually valid.
For our own infrastructure, the CI job publish-periodic-queries
added here can run on specific runner that has access to Thanos. This
job can upload to a GCS bucket.
One query failing should not block the other queries from executing.
Output of the CI job doing a full run including uploading to GCP: https://ops.gitlab.net/reprazent/runbooks/-/jobs/4001892
The GCP bucket where those files end up: https://console.cloud.google.com/storage/browser/thanos-periodic-queries-dev