Skip to content

Add logging to process-sync-events service

Furkan Ayhan requested to merge 346786-log-sync-events into master

What does this MR do and why?

With this, we can track;

  • how many events we process
  • queue depth, event backlog

Log example;

{
  "severity": "INFO",
  // ...
  "class": "Projects::ProcessSyncEventsWorker",
  "jid": "291154d5221998bd27c7441a",
  // ...
  "extra.projects_process_sync_events_worker.total_events": 4,
  "extra.projects_process_sync_events_worker.consumable_events": 4,
  "extra.projects_process_sync_events_worker.processed_events": 4,
  // ...
}

Related to: https://gitlab.com/gitlab-org/gitlab/-/issues/346786#note_821346429

Database

Prepare:

INSERT INTO namespaces_sync_events (namespace_id) SELECT id FROM namespaces LIMIT 100000;
VACUUM ANALYZE namespaces_sync_events;

Option 1:

SELECT COUNT(id) FROM namespaces_sync_events;

https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/8316/commands/29414

Option 2:

SELECT COALESCE(MAX(id) - MIN(id) + 1, 0) AS upper_bound_count FROM namespaces_sync_events;

https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/8316/commands/29415

We choose Option 2.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Furkan Ayhan

Merge request reports