I can't find Sidekiq-on-K8s logs in GCS
Problem
I wanted to get some older (no longer in Elasticsearch) Sidekiq logs for @rpereira2: scalability#957
I saw that gitlab-gprd-logging-archive/sidekiq was populated, so I went to follow https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/logging/logging_gcs_archive_bigquery.md#accessing-fields-that-cant-be-loaded-due-to-invalid-characters. But I noticed that I don't get all Sidekiq shards, and I think I only get the ones on VMs. Running this in BigQuery:
SELECT JSON_EXTRACT_SCALAR(json, "$.jsonPayload['shard']") as shard, COUNT(*) FROM `gitlab-production.smcgivern.container_registry_migration_pre` GROUP BY shard ORDER BY shard ASC;Gives:
| Row | shard | f0_ |
|---|---|---|
| 1 | null | 198 |
| 2 | catchall | 50840499 |
| 3 | catchnfs | 56131 |
| 4 | default | 6575744 |
And the jobs I want are from the database-throttled shard: https://thanos.gitlab.net/graph?g0.range_input=3d&g0.end_input=2021-03-18%2000%3A00&g0.moment_input=2021-03-25%2012%3A56%3A08&g0.max_source_resolution=0s&g0.expr=sum%20by%20(shard)%20(increase(sidekiq_jobs_completion_seconds_count%7Bqueue%3D%22background_migration%22%2C%20env%3D%22gprd%22%7D%5B1h%5D))&g0.tab=0
Desired outcome
Someone tells me where these logs are
Acceptance criteria
-
...