In order to enable the completion of the [free saas user top initiative](https://gitlab.com/gitlab-com-top-initiatives/free-saas-user-efficiency/free-saas/-/issues/65) we need to add the following data sources to out data warehouse in [snowflake](https://about.gitlab.com/handbook/business-technology/data-team/platform/#data-warehouse) | Source | Data Team Issue | Engineering Issue | |---|---|---| | HA proxy | https://gitlab.com/gitlab-data/analytics/-/issues/11584 | https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/12980 | | Container Registry Logs | https://gitlab.com/gitlab-data/analytics/-/issues/11752 | https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/12990 | It has been proposed that we use Thanos as an intermediary to both of these sources since we have a working [data pipeline](https://gitlab.com/gitlab-com/runbooks/blob/d3b03bd2aff20865ba0ae3f96c9d38e3209b4e15/periodic-thanos-queries/README.md) through [CI jobs in `ops.gitlab.net`](https://ops.gitlab.net/gitlab-com/runbooks/-/pipeline_schedules/181/edit) and a [GCS bucket](https://console.cloud.google.com/storage/browser/periodic-queries;tab=objects?pli=1&prefix=&forceOnObjectsSortingFiltering=false) that is successfully extracting data into Snowflake. This method has been validated for HA Proxy as a live [Grafana dashboard](https://dashboards.gitlab.net/d/general-egress_ingress/general-network-ingress-egress-overview?viewPanel=23&orgId=1&refresh=5m) already exists using [prometheus queries](https://gitlab.com/gitlab-data/analytics/-/issues/11584#note_862457561). @rnienaber [recommended](https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/12990#note_836998493) that we use the same method for the container registry log data, but it is not clear yet if these data even exist in Thanos. ## Next Steps needed for HA Proxy Data 1. Add topic for the prometheus metrics to the pipeline linked above :point_right: https://gitlab.com/gitlab-com/runbooks/-/merge_requests/4481 :point_right: https://gitlab.com/gitlab-com/runbooks/-/merge_requests/4505 ## Next Steps for Container Registry logs. 1. Create prometheus metrics for container registry logs 1. Add topic for the prometheus metrics to the pipeline linked above ## Final notes We might still need additional validation and guidance from @lmai1 and others in order to create the prometheus metrics appropriately. We're currently validating this by looking at static data for the registry logs that are in bigQuery