Part of https://gitlab.com/groups/gitlab-org/-/work_items/21428 (Import Group metrics and observability initiative). Phase 1: Import observability foundation. ## Summary This issue tracks adding Internal Events for the import/migration lifecycle across all importers (standard importers like GitHub/Bitbucket, and Direct Transfer/Offline Transfer). **Before writing any code**, we want the Data team / Analytics Instrumentation team to review the event list below (names, identifiers, and properties), since some of these choices, like which property gets the `value`/`label`/`property` built-in slot when an event has more candidates than slots, are judgment calls we'd rather validate before they're locked in by an event definition. ## Terms used below A few things worth knowing before reading the table, since the properties below assume them: - **`namespace` means two different things depending on the event.** On project events, `namespace` is the project's *parent* group. On group events, `namespace` *is* the group itself, there's no separate "parent" identifier for a group, since a Group already is a Namespace in GitLab's data model. - **Direct Transfer and Offline Transfer** are the same underlying mechanism (`BulkImport`/ `BulkImports::Entity`) and share every event in this document. Offline Transfer is Direct Transfer's data format exported to a file and imported later or elsewhere, for example when there's no live network path between the two instances. `import_type` tells them apart: `gitlab_migration` vs. `gitlab_migration_offline`. - **`import_source`**: the source host and, where applicable, path (e.g. `gitlab.com/gitlab-org`). - **`request_channel`**: API, UI, or Congregate. Congregate is a Professional Services migration tool that itself calls the API, identified by its user-agent. - **`fidelity_rate`**, **`source_hosting`**, **`same_instance`**, **`parent_entity_id`**: covered in detail in "Field notes" below; each is only used once it's introduced in the table. ## Context for reviewers GitLab has two fairly different import mechanisms, and the event design below treats them differently on purpose. If you're not familiar with GitLab's importers, here's the short version. **Standard importers** (GitHub, Bitbucket, Bitbucket Server, Gitea, Git URL, Manifest) are project-only: there's no group to migrate, so they only ever create a `Project` record, tracked by `ProjectImportState` through `started → finished/failed/canceled`. File-based export/import can migrate either a project or a group, but a group import there uses a separate model, `GroupImportState`, with a smaller state machine (`started → finished/failed`, no `canceled`). Whichever model is involved, one import creates one project/group record, tracked by one state machine. ```mermaid sequenceDiagram participant User participant Project as Project + ProjectImportState User->>Project: Start import (e.g. from GitHub) Note over Project: Project record created immediately Note over Project: namespace_id and project_id available from here on Project->>Project: started (start_project_import) Project->>Project: importing... Project->>Project: finished / failed / canceled / timeout Note over Project: finish / fail / cancel / timeout_project_import ``` **Direct Transfer and Offline Transfer** (GitLab-to-GitLab migration) work differently, and this is the part worth understanding before reading the event table: - A user submits one migration request, which can include **several unrelated top-level groups/projects at once** (the multi-select migration wizard). This creates one `BulkImport` record. - Each of those top-level items becomes a `BulkImports::Entity`. Migrating a group also migrates everything under it, so a single top-level entity can have **nested child entities** (subgroups, projects inside that group), linked back to their parent entity. - Each `BulkImports::Entity` creates its own `Project` or `Group` record, but not immediately. That record only appears partway through the entity's processing, once a specific pipeline step runs. Some of the events below can't be tied to a project/namespace until that step has happened. ```mermaid graph TD U[User submits one migration request] --> BI[BulkImport record] BI --> E1[Entity: top-level Group A] BI --> E2[Entity: top-level Project B] E1 --> E3[Entity: nested subgroup] E3 --> E4[Entity: nested project] E1 -. creates .-> G1[Group record] E3 -. creates .-> G2[Group record] E4 -. creates .-> P2[Project record] E2 -. creates .-> P1[Project record] ``` Because of this, Direct Transfer/Offline Transfer needs events at more than one level: - One set fires per **`BulkImport`** (the whole migration request): was it requested, and did the whole job finish, including every entity under it? - For each individual group/project in that request (top-level or nested), we fire **two** events for the same transition: `start_project_import`/`finish_project_import`/etc. (the same events every other importer uses, carrying the full set of properties: duration, failures, fidelity), and a second, much smaller `start_gitlab_migration_entity`/`finish_gitlab_migration_entity`/etc. that exists only to carry `parent_entity_id`, the pointer needed to tell a top-level entity from a nested one. That's deliberate duplication: it means most analysis can treat every importer the same way, while the handful of hierarchy-aware questions (re-attempts, mainly) have a dedicated, minimal event to answer them. The diagrams above show structure (what creates what); they don't show the order events actually fire in, or when `namespace_id`/`project_id` become available on each one. For a single Direct Transfer/Offline Transfer entity: ```mermaid sequenceDiagram participant BI as BulkImport participant E as Entity participant D as Destination record Note over BI: User submits a migration request BI->>BI: start_gitlab_migration (once per top-level namespace) Note over BI: identifiers: user only, no namespace/project yet BI->>E: entity.start! (before any pipeline runs) E->>E: start_gitlab_migration_entity Note over E: identifiers: user (+ namespace only when entity_type is project), never project alt Loader pipeline creates the destination E->>D: ProjectPipeline's / GroupPipeline's loader runs D->>D: start_project_import / start_group_import Note over D: namespace_id and project_id both available from here on D->>D: relation pipelines run (issues, merge requests, notes, ...) D->>D: finish_project_import / finish_group_import Note over D: same identifiers, plus duration/failure/fidelity properties E->>E: finish_gitlab_migration_entity else Entity fails before a destination exists E->>E: fail_gitlab_migration_entity only Note over E: no project/namespace to attach, identifiers omitted end Note over BI: once every entity in the request reaches a terminal state BI->>BI: finish_gitlab_migration Note over BI: identifiers: user only, still no namespace/project ``` For standard importers, there's no split: `Project`/`Group` already exists before `start_project_import` fires, so `namespace_id`/`project_id` are there from the very first event. ## What we want to achieve - Compare importer usage and mix (file-based, Direct Transfer, Offline Transfer, GitHub, etc.), how migrations were requested (API, UI, or Congregate), and, for GitHub today with more importers later, whether the source was SaaS or self-managed. - Tell an attempt apart from a successful creation (`start_*` vs. `finish`/`fail`/`cancel`/`timeout_*`), instead of one usage number that conflates growth with retries. - Separate a clean success from a partial one: a migration that "succeeded" but silently dropped content is a worse outcome than a clean failure, and today we can't distinguish them. - Break down *why* imports fail, grouped by relation and error type, instead of one opaque failure counter. - Measure import fidelity (a proxy for silently dropped content) per importer. - Measure migration duration at two levels: how long one project/group took, and how long the whole migration job took end to end, against how much content it moved. - Measure re-import attempts, both per destination namespace and per migration job, including a signal for large migrations split into many small batches ("waves"). ## Proposed dashboards These give the Data team something concrete to check the event list against, instead of just the goals above: 1. **Importer usage**: counts of `start_*` vs. `finish`/`fail`/`cancel`/`timeout_*` per `import_type` over time, to see adoption trend. Events: `start`/`finish`/`fail`/`cancel`/`timeout_project_import`, and the group equivalents. 2. **Reliability**: a stacked bar by `import_type` over time, split into clean (`finish_*` with `failure_count = 0`), partial (`finish_*` with `failure_count > 0`), and failed (`fail_*`). Events: `finish`/`fail_project_import`, and the group equivalents. 3. **Failure reasons**: top failing relations and exception types, to prioritize fixes. Events: `fail_import_relation`. 4. **Import fidelity trend**: average/median `fidelity_rate` by `import_type` over time, flagging importers or periods where content silently drops. Events: `finish_project_import`, `finish_group_import`, `finish_gitlab_migration`, `finish_gitlab_migration_entity`. 5. **Duration vs. size**: `duration_seconds` against `imported_objects_count`, to baseline performance and spot imports far slower than their size predicts. Events: `finish_project_import`/`finish_group_import` (per entity), `finish_gitlab_migration` (whole job). 6. **Re-attempts**: sources with more than one attempt in a rolling window. Standard importers and file-based group import only have one level, since every import there is already a self-contained unit: `finish`/`fail`/`timeout`/`cancel_project_import`, and the group equivalents. Direct Transfer/Offline Transfer gets two views from the same events (`finish`/`fail`/`timeout`/`cancel_gitlab_migration_entity`), depending on the filter: unfiltered catches a re-attempt of one specific nested subgroup/project within an otherwise-successful migration; filtered to `parent_entity_id: null`, it's the whole top-level namespace being re-run, as described in "Re-attempts" below. 7. **Request channel**: a breakdown of API vs. UI vs. Congregate by `import_type` over time. Events: `start_project_import`, `start_group_import`, `start_gitlab_migration`. 8. **Source hosting**: SaaS vs. self-managed by `import_type` over time. Events: `start_project_import`, though only GitHub/Gitea actually populate `source_hosting`; other importers fire the same event without it. 9. **Waves**: distribution of `top_level_entities_count` per request, to spot large PS-run migrations split into many small batches. Events: `start_gitlab_migration`. 10. **Subscription plan of imported top-level namespaces**: which plan (free/premium/ultimate) the top-level group/project being imported lands in, to connect import activity to account tier. Standard importers and file-based group import are already one top-level unit per `finish_project_import`/ `finish_group_import`, so this is a direct breakdown by `import_type`. Direct Transfer/Offline Transfer needs `finish_gitlab_migration_entity` filtered to `parent_entity_id: null` instead, the same whole-namespace view as "Re-attempts," so a migration with many nested subgroups/projects counts its plan once, not once per nested entity. 11. **Failures before a destination existed**: trend of Direct Transfer/Offline Transfer entities that started, then failed, were canceled, or timed out before the loader pipeline ever created their `Project`/`Group` record. These entities are invisible to the "Reliability" dashboard above, since `fail_project_import`/`cancel_project_import`/`timeout_project_import` can't fire without a `project`/`namespace` to attach. Events: `start_gitlab_migration_entity` minus (`start_project_import` + `start_group_import`) for `gitlab_migration`/`gitlab_migration_offline`, or equivalently, `fail_gitlab_migration_entity`/`cancel_gitlab_migration_entity`/`timeout_gitlab_migration_entity` rows where identifiers are absent. ## Questions for the Data team Answered in [this thread](https://gitlab.com/gitlab-org/gitlab/-/work_items/617884#note_3709606580): - Several `finish_*` events have more numeric properties than there are `value` slots. For example, `finish_gitlab_migration` has `duration_seconds`, `total_failure_count`, `imported_objects_count`, and `fidelity_rate`, but only one can use the built-in `value` column. Is there a preferred one to promote, based on what's actually queried often? > `@ankit.panchal`: Extra numeric properties can be sent as [additional_properties](https://docs.gitlab.com/development/internal_analytics/internal_event_instrumentation/quick_start/#additional-properties) > instead of forcing one into `value`. We're keeping all of them (`duration_seconds`, `total_failure_count`, > `imported_objects_count`, `fidelity_rate`) as plain additional_properties on events where there's more > than one, rather than promoting any single one. - `import_source` (the source host/path) may need hashing or bucketing rather than being sent raw, per the Data Classification Standard. Can you confirm what's acceptable here? > `@ankit.panchal`: Confirmed: `import_source` counts as Red data and must not be sent raw to Snowplow. It > will be hashed the same way Analytics Instrumentation hashes source identifiers for frontend events. > `namespace_id`/`project_id` can still be attached raw. - Several events use extra attributes, would it be better to send more events than adding more attributes to them? > `@ankit.panchal`: No fixed rule on the number of extra attributes as long as they aren't nested, though > 10+ would be a smell. Our highest count today is 5 (on `finish_project_import` and > `finish_gitlab_migration`), comfortably within that. - Are the events enough to achieve [Phase 2](https://gitlab.com/groups/gitlab-org/-/work_items/21428#phase-2---activation-measurement-time-to-first-use-2-3-milestones) and [Phase 3](https://gitlab.com/groups/gitlab-org/-/work_items/21428#phase-2---activation-measurement-time-to-first-use-2-3-milestones) mentioned in the epic? > `@ankit.panchal`: Yes: as long as `project_id` and `ultimate_parent_namespace_id`/`unique_instance_id` > are attached, import events can be correlated with other GitLab events on the same namespace/project to > build activation funnels. Both are guaranteed to be available from `start_project_import` onward, through > all its terminal events (`finish_project_import`, `fail_project_import`, `cancel_project_import`, > `timeout_project_import`), for every importer including Direct Transfer/Offline Transfer. ## Events | Event | Fires from | Identifiers | Properties | Notes | |---|---|---|---|---| | `start_gitlab_migration` | `BulkImport` → `started` | user | `request_channel`, `top_level_entities_count` (value), `same_instance` | One request can include several top-level entities. No `import_source`, see "Re-attempts" below | | `finish_gitlab_migration` | `BulkImport` → `finished` | user | `migrated_entities_count`, `total_failure_count`, `duration_seconds`, `imported_objects_count`, `fidelity_rate` | Aggregated across every entity in the request. Fires even if some entities failed, that's what `total_failure_count` is for; only *every* entity failing produces `fail_gitlab_migration` instead | | `fail_gitlab_migration` / `cancel_gitlab_migration` / `timeout_gitlab_migration` | `BulkImport` → failed/canceled/timeout | user | `duration_seconds`, `total_failure_count` | `fail_gitlab_migration` specifically only fires when every entity in the request failed | | `start_gitlab_migration_entity` | `entity.start!`, before any pipeline runs | user, namespace (only when it's a project entity; never `project`) | `entity_type` (label), `import_source` (property), `parent_entity_id` | Direct Transfer/Offline Transfer only. Fires earlier than `start_project_import` for the same entity, see "Attempts vs. destinations created" below | | `finish_gitlab_migration_entity` | entity reaches `finished` | user, namespace (+ project) | `entity_type` (label), `import_source` (property), `parent_entity_id` | Deliberately minimal, see below | | `fail_gitlab_migration_entity` / `cancel_gitlab_migration_entity` / `timeout_gitlab_migration_entity` | entity fails/canceled/times out | user, namespace (+ project); may be absent if the record was never created | `entity_type` (label), `import_source` (property), `parent_entity_id` | Can fire before a `Project`/`Group` record exists at all | | `start_project_import` | `ProjectImportState` → `started`, or when Direct Transfer's entity creates the project record | user, project, namespace | `import_type` (label), `import_source` (property), `request_channel`, `source_hosting` (GitHub, Gitea) | Every importer, including Direct Transfer/Offline Transfer | | `finish_project_import` | `ProjectImportState` → `finished`, or when the Direct Transfer entity finishes | user, project, namespace | `import_type` (label), `import_source` (property), `failure_count`, `duration_seconds`, `imported_objects_count`, `fidelity_rate`, `source_hosting` (GitHub, Gitea) | | | `fail_project_import` | `ProjectImportState` → `failed`, or when the Direct Transfer entity fails | user, project, namespace | `import_type` (label), `import_source` (property), `source_hosting` (GitHub, Gitea) | | | `cancel_project_import` | `ProjectImportState` → `canceled`, or when the Direct Transfer entity is canceled | user, project, namespace | same as `fail_project_import` | | | `timeout_project_import` | `ProjectImportState` → timeout, or when the Direct Transfer entity times out | user, project, namespace | same as `fail_project_import` | `ProjectImportState` has no timeout state yet for standard importers, so this needs adding first; Direct Transfer already has one | | `start_group_import` / `finish_group_import` / `fail_group_import` | `GroupImportState` → `started`/`finished`/`failed`, or the Direct Transfer equivalent | user, namespace | same as the project events above, minus `source_hosting` (GitHub/Gitea-only, never applies to a group) | File-based group import and Direct Transfer/Offline Transfer only. GitHub/Bitbucket/Bitbucket Server/Gitea don't import groups at all | | `cancel_group_import` / `timeout_group_import` | Direct Transfer entity canceled/timed out | user, namespace | same as `fail_project_import`, minus `source_hosting` | Direct Transfer/Offline Transfer only: `GroupImportState` has no `canceled` or `timeout` state, so file-based group import can never produce these | | `fail_import_relation` | one or more relations (issues, MRs, etc.) fail within an import, grouped | user, project, namespace (may be absent) | `import_type` (label), `exception_class` (property), `failure_count` (value), `relation_key`, `import_source`, `parent_entity_id` (Direct Transfer/Offline Transfer only) | Fires once per (relation, error type) group, not once per failure, so an entity with 500 failed issues and 3 failed merge requests produces 2 events, not 503 | ## Field notes A few properties need more explanation than fits in a table cell. ### Re-attempts Why track this at all: a raw usage number for imports can be misleading on its own. High usage from the same source could mean healthy adoption, or it could mean the same migration is being retried over and over because something's broken. Since retries inflate usage without adding value, being able to tell "one customer, five attempts" apart from "five different customers" matters before any of the other metrics in this issue are meaningful. Repeated attempts don't all mean the same thing, though. From what's come up discussing this initiative, we expect several distinct explanations to show up in the data, not just one: - Genuine failures: the migration hit a bug or a data problem, and the user tried again. This is the reliability story, and the one we'd most want to reduce. - Dry runs and testing: Professional Services validating settings before a real cutover, or Sales running a proof-of-value. Not a reliability problem, and treating it like one would be misleading. - Backup/DR workflows: some users may be running imports repeatedly as an ad hoc backup mechanism, unrelated to migrating anywhere. - Large migrations split into waves: Professional Services sometimes runs one big migration as many smaller batches on purpose, which looks like "repeated attempts" but is really one migration, coordinated. `top_level_entities_count` on `start_gitlab_migration` is one signal for this. - Same-instance moves: reorganizing groups within one GitLab instance isn't a migration at all. `same_instance` exists specifically to filter these out. Measuring re-attempts relies on `import_source`, the source host and path of the group or project being imported: seeing the same `import_source` more than once means a re-attempt. Comparing the outcomes across those occurrences (`finish_project_import`/`fail_project_import`/`timeout_project_import`/ `cancel_project_import`, and the group equivalents) can hint at why: a failure followed by a success suggests a bug got fixed, while a string of clean successes looks more like testing or a backup routine than reliability trouble. Pairing this with `fidelity_rate` also shows whether later attempts improve or get worse. For Direct Transfer/Offline Transfer, the same events (`finish_gitlab_migration_entity`/ `fail_gitlab_migration_entity`/`timeout_gitlab_migration_entity`/`cancel_gitlab_migration_entity`) give two different views, depending on whether they're filtered: - Filtered to `parent_entity_id: null`: whole-namespace re-attempts, so a migration with many nested subgroups and projects counts as one attempt, not one per entity. This is what answers the same "was this customer's migration retried" question the standard-importer bullet above answers. - Unfiltered: granular re-attempts, catching a retry of one specific nested subgroup or project within an otherwise-successful migration, for example re-running just the one component that failed rather than the whole namespace. For example, migrating `gitlab.com/gitlab-org` (a group with 50 subgroups/projects) produces one `finish_gitlab_migration_entity` with `parent_entity_id: null` and `import_source: gitlab.com/gitlab-org`, plus 49 more with `parent_entity_id` set to that top-level entity's id. Filtering to `parent_entity_id: null` collapses this down to one attempt, not 50. Migrating the same group again later produces a second `parent_entity_id: null` row with the same `import_source`, correctly counting as 2 attempts of the same group. `*_gitlab_migration`, the job-level events, don't have `import_source`: a single request can touch several unrelated top-level namespaces on the same source instance, so there's no single path to attach, and they aren't used for this. ### Attempts vs. destinations created `start_gitlab_migration_entity` fires the moment `BulkImports::ProcessService` calls `entity.start!`, before any pipeline has run. `start_project_import`/`start_group_import` wait until the entity's pipeline actually creates the `Project`/`Group` record. That gap is intentional: an entity that fails before its pipeline ever creates a destination still gets a `start_gitlab_migration_entity`, but never gets a `start_project_import`. So `start_gitlab_migration_entity` counts every attempt, and `start_project_import` counts attempts that got far enough to create something. It's also why `start_gitlab_migration_entity` can only have `namespace` as an identifier when the entity is a project (that namespace is the pre-existing parent the project will be created in, picked before the migration started, so it doesn't depend on the entity's own record existing yet) and never for a group entity (there, `namespace` would be the group's own record, which doesn't exist this early). ### Fidelity rate A migration can "succeed" and still silently drop content. `fidelity_rate` is a proxy for this: the share of source content that actually landed in GitLab, based on counters the importers already keep (source/fetched/imported counts per relation): ``` fidelity_rate = (sum of imported records across relations) / (sum of source records across the same relations) ``` The sums are taken across relations first, then divided once, rather than averaging each relation's own ratio. For example, if `issues` has 2 records and both fail to import, while `merge_requests` has 10,000 records with 9,900 imported: - Averaging each relation's ratio: `(0% + 99%) / 2 = 49.5%`, which understates how well the migration actually went. - Summing first: `(0 + 9,900) / (2 + 10,000) ≈ 99.0%`, which correctly reflects that almost everything came through. ### Request channel `request_channel` records whether a migration was started through the API, the UI, or Congregate (a Professional Services tool that itself uses the API, identified by its user-agent). It's on `start_project_import` (every importer, including Direct Transfer/Offline Transfer) and `start_group_import` (file-based group import and Direct Transfer/Offline Transfer, the only importers that fire it), so "how were migrations started" is one query grouped by `import_type`, using whichever of the two events applies. For Direct Transfer/Offline Transfer, it's *also* on the job-level `start_gitlab_migration` (one value per request, not repeated per entity). If you need job-level counts instead, so a 10-entity migration doesn't count as 10 separate requests, use `start_gitlab_migration` for that instead. It's deliberately not also on `start_gitlab_migration_entity`: two places is already enough, and that event only carries what `start_project_import`/`start_group_import` doesn't. ### Duration and size Two different durations are tracked, and they answer different questions: - `duration_seconds` on `finish_project_import` (every importer) or `finish_group_import` (file-based group import and Direct Transfer/Offline Transfer) is how long one project or group took to import. - `duration_seconds` on `finish_gitlab_migration` is how long the *entire migration request* took, including every entity in it and any time spent queued. This is the number that answers "how long did the customer's migration take." `imported_objects_count` is the total number of records imported. It's sent as a raw number rather than a pre-computed size bucket (small/medium/large), so bucket boundaries can be adjusted later in Tableau without a code change. Duration on its own doesn't say much: a project with 50,000 issues will always take longer to import than one with 50, regardless of how well the import is going. `duration_seconds` and `imported_objects_count` need to be read together, not separately, for example as duration per object, or by comparing durations only within the same size bucket. A 10-minute import isn't slow or fast on its own; that depends entirely on how much it moved. ### Failures Two separate signals cover different questions about failure: - `failure_count` / `total_failure_count`: how many things failed, at the entity level and the whole-request level. - `fail_import_relation`: *why* things failed, grouped by which relation (issues, MRs, etc.) and which error type, so we can tell which errors are contributing to fidelity/reliability degradation. Two properties carry that grouping: - `relation_key`: the name of the relation that failed, one of the model/relation names GitLab imports (`issues`, `merge_requests`, `notes`, and so on), the same vocabulary `imported_objects_count` and `fidelity_rate` are summed over. - `exception_class`: the Ruby exception class that was raised (for example `ActiveRecord::RecordInvalid`), not the exception message. Cardinality is bounded by however many distinct exception types the import pipelines actually raise, expected to be a few dozen, not open-ended, and a class name shouldn't carry customer data the way a message can (a message might quote the invalid record's content, a URL, or a token). That's why `exception_class` is what gets sent instead of `exception_message`. ### Migration hierarchy (`parent_entity_id`) Direct Transfer/Offline Transfer entities can be nested (a subgroup or project inside a migrated group). `parent_entity_id` records which entity a nested one belongs to, and is `null` for a top-level entity. This is what lets us tell "this specific destination was imported before" apart from "this was one small piece of a larger migration" when measuring repeat attempts (see "Re-attempts" above). ### Source hosting `source_hosting` records whether the source was a SaaS product (`saas`) or a self-managed instance (`self_managed`), for example GitHub.com vs. a customer's own GitHub Enterprise Server, or gitea.com vs. a self-hosted Gitea instance. It's GitHub and Gitea only for now, but named generically because the same distinction will likely apply to other importers later. ### Same instance `same_instance` flags when a Direct Transfer/Offline Transfer migration's source and destination are the same GitLab instance. This matters because that's a reorganization (moving groups around within one installation), not a customer migrating onto GitLab. ## References - Epic: https://gitlab.com/groups/gitlab-org/-/work_items/21428 - Draft MR https://gitlab.com/gitlab-org/gitlab/-/merge_requests/236324