CI Job Token Allowlist migration rake task
What does this MR do and why?
Rake task to migrate authentications log to job... (#514385 - closed)
This MR adds the rake task to migrate and compact the authentications log to job token allowlist. The rake task can be execute with the following options:
Task: rake ci:job_tokens:allowlist:autopopulate_and_enforce
-
PREVIEWperforms a dry-run and outputs the steps that it will take, but no data is changed -
ONLY_PROJECT_IDSperforms the migration for only the supplied project ids -
EXCLUDE_PROJECT_IDSperforms the migration for all projects except for the supplied project ids
Examples:
rake ci:job_tokens:allowlist:autopopulate_and_enforce PREVIEW=truerake ci:job_tokens:allowlist:autopopulate_and_enforce PREVIEW=true ONLY_PROJECT_IDS=2,3rake ci:job_tokens:allowlist:autopopulate_and_enforce PREVIEW=true EXCLUDE_PROJECT_IDS=2,3rake ci:job_tokens:allowlist:autopopulate_and_enforcerake ci:job_tokens:allowlist:autopopulate_and_enforce ONLY_PROJECT_IDS=2,3rake ci:job_tokens:allowlist:autopopulate_and_enforce EXCLUDE_PROJECT_IDS=2,3
See How to set up and validate locally below for examples of the task output.
References
Please include cross links to any resources that are relevant to this MR. This will give reviewers and future readers helpful context to give an efficient review of the changes introduced.
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
| Before | After |
|---|---|
How to set up and validate locally
-
Create sample authorization log records in the rails console:
project1 = Project.first project2 = Project.second project3 = Project.third project4 = Project.fourth project5 = Project.fifth Ci::JobToken::Authorization.create(accessed_project: project1, origin_project: project2, last_authorized_at: 1.day.ago) Ci::JobToken::Authorization.create(accessed_project: project1, origin_project: project3, last_authorized_at: 1.day.ago) Ci::JobToken::Authorization.create(accessed_project: project2, origin_project: project1, last_authorized_at: 1.day.ago) Ci::JobToken::Authorization.create(accessed_project: project2, origin_project: project3, last_authorized_at: 1.day.ago) Ci::JobToken::Authorization.create(accessed_project: project3, origin_project: project4, last_authorized_at: 1.day.ago) Ci::JobToken::Authorization.create(accessed_project: project3, origin_project: project5, last_authorized_at: 1.day.ago) Ci::JobToken::Authorization.create(accessed_project: project4, origin_project: project2, last_authorized_at: 1.day.ago) Ci::JobToken::Authorization.create(accessed_project: project4, origin_project: project3, last_authorized_at: 1.day.ago) -
Get the IDs for each project:
[project1, project2, project3, project4, project5].pluck(:id) => [1, 2, 3, 4, 5] -
Run the rake task in preview mode:
bundle exec rake ci:job_tokens:allowlist:autopopulate_and_enforce PREVIEW=trueThe output should look similar to below. The total number may be different depending on how many projects exist in your GDK.
########## # # PREVIEW MODE ENABLED # ########## Migrating project(s) in preview mode... Would have migrated project id: 1. Would have migrated project id: 2. Would have migrated project id: 3. Would have migrated project id: 4. Would have migrated project id: 5. Would have migrated project id: 6. Would have migrated project id: 7. Would have migrated project id: 8. Would have migrated project id: 9. Would have migrated project id: 10. Would have migrated project id: 11. Would have migrated project id: 12. Would have migrated project id: 13. Would have migrated project id: 14. Would have migrated project id: 15. Would have migrated project id: 16. Would have migrated project id: 17. Would have migrated project id: 18. Would have migrated project id: 19. Would have migrated project id: 20. Migration complete in preview mode. -
Run the rake task in preview mode with
ONLY_PROJECT_IDSbundle exec rake ci:job_tokens:allowlist:autopopulate_and_enforce PREVIEW=true ONLY_PROJECT_IDS=2,3The output should look similar to this:
########## # # PREVIEW MODE ENABLED # ########## Migrating project(s) in preview mode... Would have migrated project id: 2. Would have migrated project id: 3. Migration complete in preview mode. -
Run the rake task in preview mode with too many
ONLY_PROJECT_IDSbundle exec rake ci:job_tokens:allowlist:autopopulate_and_enforce PREVIEW=true ONLY_PROJECT_IDS=1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287,288,289,290,291,292,293,294,295,296,297,298,299,300,301,302,303,304,305,306,307,308,309,310,311,312,313,314,315,316,317,318,319,320,321,322,323,324,325,326,327,328,329,330,331,332,333,334,335,336,337,338,339,340,341,342,343,344,345,346,347,348,349,350,351,352,353,354,355,356,357,358,359,360,361,362,363,364,365,366,367,368,369,370,371,372,373,374,375,376,377,378,379,380,381,382,383,384,385,386,387,388,389,390,391,392,393,394,395,396,397,398,399,400,401,402,403,404,405,406,407,408,409,410,411,412,413,414,415,416,417,418,419,420,421,422,423,424,425,426,427,428,429,430,431,432,433,434,435,436,437,438,439,440,441,442,443,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,461,462,463,464,465,466,467,468,469,470,471,472,473,474,475,476,477,478,479,480,481,482,483,484,485,486,487,488,489,490,491,492,493,494,495,496,497,498,499,500,501,502,503,504,505,506,507,508,509,510,511,512,513,514,515,516,517,518,519,520,521,522,523,524,525,526,527,528,529,530,531,532,533,534,535,536,537,538,539,540,541,542,543,544,545,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,565,566,567,568,569,570,571,572,573,574,575,576,577,578,579,580,581,582,583,584,585,586,587,588,589,590,591,592,593,594,595,596,597,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,615,616,617,618,619,620,621,622,623,624,625,626,627,628,629,630,631,632,633,634,635,636,637,638,639,640,641,642,643,644,645,646,647,648,649,650,651,652,653,654,655,656,657,658,659,660,661,662,663,664,665,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,733,734,735,736,737,738,739,740,741,742,743,744,745,746,747,748,749,750,751,752,753,754,755,756,757,758,759,760,761,762,763,764,765,766,767,768,769,770,771,772,773,774,775,776,777,778,779,780,781,782,783,784,785,786,787,788,789,790,791,792,793,794,795,796,797,798,799,800,801,802,803,804,805,806,807,808,809,810,811,812,813,814,815,816,817,818,819,820,821,822,823,824,825,826,827,828,829,830,831,832,833,834,835,836,837,838,839,840,841,842,843,844,845,846,847,848,849,850,851,852,853,854,855,856,857,858,859,860,861,862,863,864,865,866,867,868,869,870,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937,938,939,940,941,942,943,944,945,946,947,948,949,950,951,952,953,954,955,956,957,958,959,960,961,962,963,964,965,966,967,968,969,970,971,972,973,974,975,976,977,978,979,980,981,982,983,984,985,986,987,988,989,990,991,992,993,994,995,996,997,998,999,1000,1001The output should look like to this:
########## # # ERROR: ONLY_PROJECT_IDS and EXCLUDE_PROJECT_IDS must contain less than 1000 items, try again. # ########## -
Run the rake task in preview mode with
EXCLUDE_PROJECT_IDSbundle exec rake ci:job_tokens:allowlist:autopopulate_and_enforce PREVIEW=true EXCLUDE_PROJECT_IDS=2,3The output should look similar to this. Again, this will vary based on number of project, but the important part is that project ids 2 and 3 are not shown.
########## # # PREVIEW MODE ENABLED # ########## Migrating project(s) in preview mode... Would have migrated project id: 1. Would have migrated project id: 4. Would have migrated project id: 5. Would have migrated project id: 6. Would have migrated project id: 7. Would have migrated project id: 8. Would have migrated project id: 9. Would have migrated project id: 10. Would have migrated project id: 11. Would have migrated project id: 12. Would have migrated project id: 13. Would have migrated project id: 14. Would have migrated project id: 15. Would have migrated project id: 16. Would have migrated project id: 17. Would have migrated project id: 18. Would have migrated project id: 19. Would have migrated project id: 20. Migration complete in preview mode. -
Run the rake task in preview mode with
ONLY_PROJECT_IDSandEXCLUDE_PROJECT_IDSbundle exec rake ci:job_tokens:allowlist:autopopulate_and_enforce PREVIEW=true ONLY_PROJECT_IDS=1,5 EXCLUDE_PROJECT_IDS=2,3The output should look like this:
########## # # ERROR: ONLY_PROJECT_IDS and EXCLUDE_PROJECT_IDS cannot both be set, try again. # ########## -
Run the rake task in regular mode, introduce a forced exception to show the error summary:
In the file:
app/services/ci/job_token/autopopulate_allowlist_service.rb, add the following afterdef unsafe_execute!:raise Gitlab::Utils::TraversalIdCompactor::CompactionLimitCannotBeAchievedErrorSave the file, then run the rake task:
bundle exec rake ci:job_tokens:allowlist:autopopulate_and_enforceThe output should look like this:
Migrating project(s)... Error migrating project id: 1, error: Gitlab::Utils::TraversalIdCompactor::CompactionLimitCannotBeAchievedError Error migrating project id: 2, error: Gitlab::Utils::TraversalIdCompactor::CompactionLimitCannotBeAchievedError Error migrating project id: 3, error: Gitlab::Utils::TraversalIdCompactor::CompactionLimitCannotBeAchievedError Error migrating project id: 4, error: Gitlab::Utils::TraversalIdCompactor::CompactionLimitCannotBeAchievedError Error migrating project id: 5, error: Gitlab::Utils::TraversalIdCompactor::CompactionLimitCannotBeAchievedError Migration complete. Summary: 0 project(s) successfully migrated, 5 error(s) reported. The following 5 project id(s) failed to migrate: 1, 2, 3, 4, 5Remove the code change for the rest of the examples.
-
Run the rake task in regular mode, which performs the migrations:
bundle exec rake ci:job_tokens:allowlist:autopopulate_and_enforceThe output should look like this:
Migrating project(s)... Migrated project id: 1. Migrated project id: 2. Migrated project id: 3. Migrated project id: 4. Migrated project id: 5. Migrated project id: 6. Migrated project id: 7. Migrated project id: 8. Migrated project id: 9. Migrated project id: 10. Migrated project id: 11. Migrated project id: 12. Migrated project id: 13. Migrated project id: 14. Migrated project id: 15. Migrated project id: 16. Migrated project id: 17. Migrated project id: 18. Migrated project id: 19. Migrated project id: 20. Migration complete. Summary: 20 project(s) successfully migrated, 0 error(s) reported.Run it again to verify the projects have been migrated:
bundle exec rake ci:job_tokens:allowlist:autopopulate_and_enforceThe output will be:
Migrating project(s)... Migration complete. Summary: 0 project(s) successfully migrated, 0 error(s) reported.
Database Review
ProjectCiCdSetting Load
Select ProjectCiCdSettings in batches where inbound_job_token_scope_enabled is FALSE
SELECT
"project_ci_cd_settings".*
FROM
"project_ci_cd_settings"
WHERE
"project_ci_cd_settings"."inbound_job_token_scope_enabled" = FALSE
ORDER BY
"project_ci_cd_settings"."id" ASC
LIMIT
1000https://console.postgres.ai/gitlab/gitlab-production-main/sessions/36063/commands/111226
Project Load
Select Projects for a given set of ids
SELECT
"projects"."id",
"projects"."name",
"projects"."path",
"projects"."description",
"projects"."created_at",
"projects"."updated_at",
"projects"."creator_id",
"projects"."namespace_id",
"projects"."last_activity_at",
"projects"."import_url",
"projects"."visibility_level",
"projects"."archived",
"projects"."merge_requests_template",
"projects"."star_count",
"projects"."merge_requests_rebase_enabled",
"projects"."import_type",
"projects"."import_source",
"projects"."avatar",
"projects"."approvals_before_merge",
"projects"."reset_approvals_on_push",
"projects"."merge_requests_ff_only_enabled",
"projects"."issues_template",
"projects"."mirror",
"projects"."mirror_last_update_at",
"projects"."mirror_last_successful_update_at",
"projects"."mirror_user_id",
"projects"."shared_runners_enabled",
"projects"."runners_token",
"projects"."build_allow_git_fetch",
"projects"."build_timeout",
"projects"."mirror_trigger_builds",
"projects"."public_builds",
"projects"."pending_delete",
"projects"."last_repository_check_failed",
"projects"."last_repository_check_at",
"projects"."only_allow_merge_if_pipeline_succeeds",
"projects"."has_external_issue_tracker",
"projects"."repository_storage",
"projects"."request_access_enabled",
"projects"."has_external_wiki",
"projects"."repository_read_only",
"projects"."lfs_enabled",
"projects"."description_html",
"projects"."only_allow_merge_if_all_discussions_are_resolved",
"projects"."repository_size_limit",
"projects"."service_desk_enabled",
"projects"."printing_merge_request_link_enabled",
"projects"."auto_cancel_pending_pipelines",
"projects"."cached_markdown_version",
"projects"."last_repository_updated_at",
"projects"."ci_config_path",
"projects"."disable_overriding_approvers_per_merge_request",
"projects"."delete_error",
"projects"."storage_version",
"projects"."resolve_outdated_diff_discussions",
"projects"."remote_mirror_available_overridden",
"projects"."only_mirror_protected_branches",
"projects"."pull_mirror_available_overridden",
"projects"."jobs_cache_index",
"projects"."external_authorization_classification_label",
"projects"."mirror_overwrites_diverged_branches",
"projects"."external_webhook_token",
"projects"."pages_https_only",
"projects"."packages_enabled",
"projects"."merge_requests_author_approval",
"projects"."pool_repository_id",
"projects"."runners_token_encrypted",
"projects"."bfg_object_map",
"projects"."detected_repository_languages",
"projects"."merge_requests_disable_committers_approval",
"projects"."require_password_to_approve",
"projects"."max_pages_size",
"projects"."max_artifacts_size",
"projects"."pull_mirror_branch_prefix",
"projects"."remove_source_branch_after_merge",
"projects"."marked_for_deletion_at",
"projects"."marked_for_deletion_by_user_id",
"projects"."suggestion_commit_message",
"projects"."autoclose_referenced_issues",
"projects"."project_namespace_id",
"projects"."hidden",
"projects"."organization_id"
FROM
"projects"
WHERE
"projects"."id" IN (
[MASKED] # list of project ids
)
ORDER BY
"projects"."id" ASC
LIMIT
1000https://console.postgres.ai/gitlab/gitlab-production-main/sessions/36063/commands/111227
Update ProjectCiCdSetting
UPDATE
"project_ci_cd_settings"
SET
"inbound_job_token_scope_enabled" = TRUE
WHERE
"project_ci_cd_settings"."id" = [MASKED] # project_idhttps://console.postgres.ai/gitlab/gitlab-production-main/sessions/36063/commands/111229
Edited by Darby Frey