Skip to content

Store correlation ID with import state & expose via API

Matthias Käppler requested to merge 210522-import-status-correlation-id into master

What does this MR do?

In #210522 (closed) we are looking to expose more information about import status and possible failures via the REST API to facilitate diagnostics in case of problems.

This MR is an MVC toward that goal by capturing and exposing the correlation_id for each attempted import.

The correlation ID is useful for two reasons:

  • To correlate exceptions with import executions via sidekiq or otherwise (we sometimes have to run imports manually via rake).
  • To find all ImportFailure records related to a particular import. Theoretically, at least currently, the project ID should suffice to tie these records together, but since we are looking to allow incremental imports in the future, there might actually be multiple failure sets that exist for a single project, and we would likely only want to know those of the latest import attempt. Since correlation IDs are unique per job, and moreover not ephemeral like jids, they seem like a good choice.

Implementation wise:

  • add a correlation_id attribute to ProjectImportState
    • I did not index it because there is currently a 1-1 relationship between projects & state and even in the future we wouldn't expect to have more than a handful of import attempts per project
  • save the current correlation ID whenever an import starts
  • GET + POST endpoints for imports now render the correlation_id

Example:

{
  "id": 39,
  "description": "Ut soluta dignissimos sapiente atque nam eius perferendis.",
  "name": "test-import",
  "name_with_namespace": "Administrator / test-import",
  "path": "test-import",
  "path_with_namespace": "root/test-import",
  "created_at": "2020-04-03T12:48:12.314Z",
  "import_status": "started",
  "correlation_id": "mezklWso3Za"
}

Migration output:

$ bin/rake db:migrate:down VERSION=20200402124802
== 20200402124802 AddCorrelationIdToProjectImportState: reverting =============
-- remove_column(:project_mirror_data, :correlation_id_value)
   -> 0.0014s
== 20200402124802 AddCorrelationIdToProjectImportState: reverted (0.0083s) ====

$ bin/rake db:migrate:up VERSION=20200402124802
== 20200402124802 AddCorrelationIdToProjectImportState: migrating =============
-- add_column(:project_mirror_data, :correlation_id_value, :string, {:limit=>128})
   -> 0.0009s
== 20200402124802 AddCorrelationIdToProjectImportState: migrated (0.0060s) ====

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Tested locally via curl.

There seem to be essentially zero tests for grape entities -- I didn't add one either. Not sure if that's just broken windows or whether we specifically decided not to test these?

Edited by 🤖 GitLab Bot 🤖

Merge request reports