Test if Direct transfer is possible on the instance with minimal requirements for running GitLab
In the docs, it’s mentioned the minimum requirements to run GitLab. We should test Direct Transfer in an instance with such configuration.
If DT in this case doesn't work, find minimum requirements for instance for DT and document it.
Designs
- Show closed items
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- Magdalena Frankiewicz added groupimport and integrate label
added groupimport and integrate label
- 🤖 GitLab Bot 🤖 added devopsmanage sectioncore platform labels
added devopsmanage sectioncore platform labels
added Category:Importers Importer:Direct Transfer labels
- Magdalena Frankiewicz added workflowrefinement label
added workflowrefinement label
- Magdalena Frankiewicz added ready for next refinement label
added ready for next refinement label
- Magdalena Frankiewicz added backend documentation labels
added backend documentation labels
- Magdalena Frankiewicz added to epic &9758
added to epic &9758
- Author Developer
@wortschi if this would fit in 16.11 after user mapping and air-gapped issues are added, and it doesn't require too much work, let's add it to 16.11, WDYT?
Collapse replies - Maintainer
I suppose we'd need to setup another instance similar to https://gitlab.com/gitlab-org/gitlab/-/issues/438016+ but with the latest GitLab version running and with the minimal requirements. @carlad-gl WDYT?
- Maintainer
Yes we would need to set up test instances. Would @nprabakaran be able to help with this?
- Developer
Thank you for tagging me here. Yes, I would be able to set this up, but I would need your guidance on the setup @carlad-gl as you have the experience of doing this recently
To understand more on this, I have few questions:
- Do we want to test the DT import from the gitlab.com to this intance (running on latest gitlab version with minimal requirements)
- Are we looking to import the same large test project we used for the DT from 16.2 to 16.8 issue
- What is the outcome we are expecting out of this testing, is it the DT import metrics or we also want to see the performance of the import process
@wortschi Can you please provide your thoughts the above questions? Thank you!
- Maintainer
@nprabakaran Thanks for the great questions:
- Do we want to test the DT import from the gitlab.com to this intance (running on latest gitlab version with minimal requirements)
I think we want to use
gitlab.com
as the source and the instance with minimal requirements as the target. @m_frankiewicz Can you confirm?- Are we looking to import the same large test project we used for the DT from 16.2 to 16.8 issue
Yeah we can do that
- What is the outcome we are expecting out of this testing, is it the DT import metrics or we also want to see the performance of the import process
@m_frankiewicz Since you created the issue, can you please answer this question? I suppose we want to find out if DT is causing performance problems with minimal requirements?
- Author Developer
I think we want to use
gitlab.com
as the source and the instance with minimal requirements as the target.Good question, on which instance, source or destination, is there more load?
Perhaps we would want to check both ways. Once we have the instance we could do migration both ways, right?
Are we looking to import the same large test project we used for the DT from 16.2 to 16.8 issue
Yeah, the test should be with large enough data set. Hope testing with this project makes it easier?
I suppose we want to find out if DT is causing performance problems with minimal requirements?
Yes, exactly.
- Maintainer
Perhaps we would want to check both ways. Once we have the instance we could do migration both ways, right?
That's good idea
- Maintainer
I would need your guidance on the setup @carlad-gl as you have the experience of doing this recently
Absolutely. I also added a page to our runbook with info about setting up sandbox instances. If there are any gaps in that guide I can definitely help out
- Magdalena Frankiewicz changed milestone to %16.11
changed milestone to %16.11
- Magdalena Frankiewicz added typemaintenance label
added typemaintenance label
- Magdalena Frankiewicz mentioned in issue gitlab-org/manage/general-discussion#17668
mentioned in issue gitlab-org/manage/general-discussion#17668
- Magdalena Frankiewicz added Stretch milestonep4 labels
added Stretch milestonep4 labels
- Martin Wortschack changed milestone to %17.0
changed milestone to %17.0
- Maintainer
@m_frankiewicz Moving to %17.0 as discussed
- Martin Wortschack mentioned in issue #454829 (closed)
mentioned in issue #454829 (closed)
- Magdalena Frankiewicz removed milestonep4 label
removed milestonep4 label
- Magdalena Frankiewicz removed Stretch label
removed Stretch label
- Magdalena Frankiewicz changed milestone to %17.1
changed milestone to %17.1
- Magdalena Frankiewicz mentioned in issue gitlab-org/manage/general-discussion#17672
mentioned in issue gitlab-org/manage/general-discussion#17672
- Developer
The current minimal requirements is setup for 500 users but the setup is going to be changed to 1000 users as the minimal requirements.
So the Testing for this needs to be re-done with 1k users
Collapse replies - Developer
1K environment is setup is done in GCP and Direct transfers of project using https://gitlab.com/gitlab-org/quality/performance/-/blob/main/docs/environment_prep.md was successful
I am carrying out other API and UI transfers with the projects in https://gitlab.com/gitlab-migration-large-import-test
- Author Developer
@nprabakaran how is this going? Any updates here?
- Developer
@m_frankiewicz This is completed and the DT results were successful. I raised one issue from this testing #460238 (closed)
Status of the Test:
- I have setup 1k ref arch instances in my sandbox environment (http://34.91.153.18/)
- I triggered a direct transfer import of gitlab-migration-large-import-test/single-project/kubernetes to the 1k instance using the UI.
- After 24 hours it timed-out and the migration was not complete.
- It did not migrate the issues
- It did not migrate the MR completely
- Stats:
- Milestone51/51
- Repository1/1
- Label202/202
- Group1/1
- Design1/1
- Auto DevOps1/1
- commit_notes5/5
- Board1/1
- Service Desk1/1
- Container expiration policy1/1
- Push Rule1/1
- Project feature1/1
- Merge request75344/75408
- releases633/633
- I've triggered another import, this time of the smaller project gitlab-migration-large-import-test/gitaly-copy/gitaly-complete only using the api:
[12:20:42] dev/gdk [main]$ SOURCE_GROUP=gitlab-migration-large-import-test/gitaly-copy DESTINATION_PATH=gitaly-complete curl --request POST \ --url "http://34.91.153.18/api/v4/bulk_imports" \ --header "content-type: application/json" \ --header "PRIVATE-TOKEN: xxxx" \ --data "{ \"configuration\": { \"url\": \"https://gitlab.com/\", \"access_token\": \"xxxx\" }, \"entities\": [ { \"source_full_path\": \"$SOURCE_GROUP\", \"source_type\": \"group_entity\", \"destination_slug\": \"$DESTINATION_PATH\", \"destination_namespace\": \"\", \"migrate_projects\": true } ] }" {"id":5,"status":"created","source_type":"gitlab","created_at":"2024-05-28T21:25:30.478Z","updated_at":"2024-05-28T21:25:30.478Z","has_failures":false}%
- This import was successful with the following stats:
- Service Desk1/1
- releases1/1
- Repository1/1
- Label554/554
- Milestone132/132
- CI pipeline20870/20870
- Issue5288/5288
- Push Rule1/1
- commit_notes3394/3394
- Project feature1/1
- Protected Branch3/3
- Board3/3
- Container expiration policy1/1
- Merge request6520/6520
- uploads4399/4399
- Design1/1
- Auto DevOps1/1
- pipeline_schedules1/1
- Group1/1
With the above we can say that Direct transfer is possible on the instance with minimal requirements for running GitLab which is 1k
Edited by Nivetha Prabakaran - Author Developer
@nprabakaran I cannot see a link to docs update MR here, could you share it again please?
- Developer
This is the MR link : !154538
- Developer
I also did a migration of a small project and the DT migration was successful. I will be adding this also in the above MR
- Repository 1/1
- Group 1/1
- Label 7/7
- Milestone 76/76
- Issue 598/598
- Merge request 1049/1049
- Service Desk 1/1
- releases 211/211
- Push Rule 1/1
- Project feature 1/1
- Board 1/1
- Container expiration policy 1/1
- Auto DevOps 1/1
- commit_notes 104/104
- CI pipeline 12491/12491
- Design 1/1
Edited by Nivetha Prabakaran
- Magdalena Frankiewicz assigned to @nprabakaran
assigned to @nprabakaran
- Martin Wortschack removed ready for next refinement label
removed ready for next refinement label
- Magdalena Frankiewicz added workflowin dev label and removed workflowrefinement label
added workflowin dev label and removed workflowrefinement label
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#17901 (closed)
mentioned in issue gitlab-org/quality/triage-reports#17901 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#18066 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18066 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#18168 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18168 (closed)
- Nivetha Prabakaran mentioned in merge request !154538
mentioned in merge request !154538
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#18252 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18252 (closed)
- 🤖 GitLab Bot 🤖 changed milestone to %17.2
changed milestone to %17.2
- 🤖 GitLab Bot 🤖 added missed:17.1 label
added missed:17.1 label
- Developer
The context of this issue now includes testing of the source environment being on minimal requirements as well, so the testing needs to be done with this setup.
Collapse replies - Developer
Reg this MR !154538
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#18360 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18360 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#18441 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18441 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#18637 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18637 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#18724 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18724 (closed)
- 🤖 GitLab Bot 🤖 changed milestone to %17.3
changed milestone to %17.3
- 🤖 GitLab Bot 🤖 added missed:17.2 label
added missed:17.2 label
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#18806 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18806 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#18918 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18918 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#19089 (closed)
mentioned in issue gitlab-org/quality/triage-reports#19089 (closed)
- Maintainer
@nprabakaran Can you please share an update on the progress here? What's been done, what is left?
Collapse replies - Developer
Hi @wortschi, I did multiple DT from source on minimal requirements and target also being on minimal requirements. But both the times, the DT was only partially completed.
Details of the DT Source the count of CI pipelines is 12491
In target, in both the DT, the CI pipelines were not migrated completely
I need some guidance to figure out why the DT was partially completed and whats the error. Can you please pair someone here to figure this out?
Thanks!
Edited by Nivetha Prabakaran - Maintainer
@nprabakaran Thank you for the update.
I need some guidance to figure out why the DT was partially completed and whats the error. Can you please pair someone here to figure this out?
Do you have access to the logs? If so, can you share them?
- Developer
I do have access to the environment and logs in the https://console.cloud.google.com, which logs are we looking for here?
- Maintainer
@nprabakaran I think the logs in google cloud would be helpful. Are there any errors logged for the time of the migration?
- Developer
Thank you @wortschi, These are logs from the target instance in GCP downloaded-logs-20240813-231043.json but I dont see any errors logged in these.
Edited by Nivetha Prabakaran
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#19201 (closed)
mentioned in issue gitlab-org/quality/triage-reports#19201 (closed)
- 🤖 GitLab Bot 🤖 changed milestone to %17.4
changed milestone to %17.4
- 🤖 GitLab Bot 🤖 added missed:17.3 label
added missed:17.3 label
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#19287 (closed)
mentioned in issue gitlab-org/quality/triage-reports#19287 (closed)