Push grafana logs to loging cluster

Summary

We currently do not have logs for grafana available, which makes it difficult to diagnose issues. In particular when combined with relatively poor observability of the cloudsql database, it becomes hard to debug database issues.

We should set up pubsub topic, pubsubbeat, and indices for the logs from grafana pods, and push them to the nonprod-log cluster.

Related Incident(s)

• production#6639 (closed)

Originating issue(s): production#6639 (closed)

Desired Outcome/Acceptance Criteria

Grafana logs are available.

Associated Services

Corrective Action Issue Checklist

• Link the incident(s) this corrective action arose out of
• Give context for what problem this corrective action is trying to prevent from re-occurring
• Assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4')
• Assign a priority (this will default to 'priority::4')