Extend request_channel tracking to remaining import types
What does this MR do and why?
A previous MR (!256773 (merged)) added request_channel (api / ui / congregate) to the start_project_import and start_group_import events. This lets us see how an import was started via the UI, the API, or via Congregate. That MR covered Direct Transfer, Offline Transfer, and importers built on Import::BaseService (GitHub, Gitea, Bitbucket, Bitbucket Server, Fogbugz).
This MR covers the rest of the import types:
- File-based project import (upload, remote URL, S3)
- Manifest import
- Bitbucket Cloud import from the UI
- Git repository by URL, for both new projects and retrying a failed import
- File-based group import
request_channel is still only sent on start events, and channel detection is unchanged. Web controllers pass :ui directly, and API endpoints call Gitlab::Import::RequestChannel.detect(request), which returns :congregate if the User-Agent starts with GitLabApiClient, or :api otherwise. There is a follow up issue to replace User-Agent sniffing for congregate detection: Replace User-Agent sniffing for Congregate dete... (#630340)
Implementation
| Entry point | How the channel is captured |
|---|---|
Import::GitlabProjectsController (UI upload) |
Passes request_channel: :ui to Projects::GitlabProjectsImportService, which stashes it before calling Projects::CreateService. |
POST /projects/import, /projects/remote-import, /projects/remote-import-s3 |
Pass detect(request) to Import::GitlabProjects::CreateProjectService (new request_channel: keyword, defaults to :api), which forwards it to Projects::GitlabProjectsImportService (new keyword, defaults to nil so templates and the rake import task keep working as before). |
Import::ManifestController and Import::BitbucketController (UI) |
Stash :ui in the controller before calling their project creator. Bitbucket via the API was already covered through Import::BitbucketService. |
ProjectsController#create with an import URL |
Stashes :ui. |
POST /projects and POST /projects/user/:user_id with import_url |
Stash detect(request) through a small helper. |
Retry endpoints: Projects::ImportsController#create (:ui) and POST /projects/:id/import/git (detect) |
Set import_state.request_channel directly before calling .schedule. The import state usually already exists in these cases, so its after_create callback (see below) won't run. |
Import::GitlabGroupsController (:ui) and POST /groups/import (detect) |
Pass request_channel: to Groups::ImportExport::ImportService (new keyword, defaults to :api), which writes it to GroupImportState before enqueuing GroupImportWorker. |
"Stashing" the channel means putting it in the request store. ProjectImportState's after_create callback then copies it into Redis, so it's saved before the import job is enqueued. This matters because the job can start running before Projects::CreateService returns.
GroupImportState gets the same treatment as ProjectImportState: new request_channel= / request_channel methods backed by a Redis key (group_import_request_channel_<group_id>). start_group_import now includes request_channel in its payload, using .compact so it's left out when there's no value. Finish and fail events still don't carry it.
The event YAMLs already declare request_channel, so no YAML changes are needed here.
Note for reviewers: this needs an analytics instrumentation review because the start_group_import payload is changing. Also, some of the controller and request specs for this change can't be run in the author's local environment, so CI is the check for those.
Differences from the issue
- The issue says
GroupImportState#request_channelalready exists. It doesn't; this MR adds it. - The Bitbucket Cloud UI controller is in scope, since it was listed as a follow-up in the previous MR, even though it's not mentioned in this issue.
app/controllers/import/url_controller.rbonly validates URLs. Git-by-URL imports actually go throughProjectsController#create,POST /projects, and the retry endpoints.- The group import UI controller is
Import::GitlabGroupsController.groups/import_export_controller.rb, as named in the issue, doesn't exist. - The issue suggests setting
import_state.request_channelafter the record is created. For new projects, that's too late, since the import job can start before creation finishes. Instead, the channel is stashed before creation, following the same pattern as the previous MR.
References
Parent: #617884 Closes #630178 (closed) Previous MR: !256773 (merged) Congregate header follow-up: #630340
Screenshots or screen recordings
Not applicable. This is a backend-only change with no UI.
How to set up and validate locally
Same setup as the previous MR:
- Enable Snowplow Micro in GDK:
gdk config set snowplow_micro.enabled true, thengdk reconfigure && gdk restart. - Watch events at http://gdk.test:9091/micro/good.
- Start imports through each entry point and check the events:
- UI: GitLab export upload, Manifest, Bitbucket Cloud, Repository by URL, group export upload.
- API: use curl, e.g.
POST /api/v4/projects/import,POST /api/v4/projectswithimport_url,POST /api/v4/groups/import. - Congregate: repeat the API calls with
-H "User-Agent: GitLabApiClient".
- Confirm
start_project_importandstart_group_importcarryrequest_channelasui,api, orcongregatedepending on how the import was started, and that finish/fail events don't carry it.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Related to #630178 (closed)