Skip to content

Monorepo Refactor Preliminary Separation and Decoupling Tasks

Description

Pending reorganization of the sub-epics (&654 (closed), &283 (closed), etc.) based on the new phased approach, we are proceeding with some preliminary tasks to cleanup, DRY-up, simplify and decouple the current top-level build.

This iterative approach will make future work on the monorepo Separation (&654 (closed)) and Isolation (&655 (closed)) epics faster, less risky, and easier to reason about.

The end result of this work will be multiple small, independent builds and deploys which can be further separated and isolated going forward.

NOTE: Some of the task descriptions below may be out-of-date or incomplete. See the linked issue for the latest information

Tasks

  • (!56731 (merged)) Rewrite sitemap to be composed of index of different sitemaps, to prepare for independent deploys: https://www.sitemaps.org/protocol.html#index
    • Each builds job builds its own sitemap based on the files it builds, and main sitemap includes them all.
    • Allows deletion of bin/combine-sitemaps complexity and coupling from deploy process.
    • Remember to remove hardcoded handling of sitemap.xml from extensions/monorepo.rb.
  • (!56764 (merged)) Remove unnecessary templates hardcoded handling in extensions/monorepo.rb.
  • (!56765 (merged)) Extract .review-replace-urls job logic to scripts/review-replace-urls, to simplify CI logic and support refactoring review/master deploy logic out of CI file into a single common deploy script.
    • NOTE: This does introduce some extra time to the review-prep-bucket and review jobs, by switching from GIT_STRATEGY: none to GIT_STRATEGY: "fetch". But this is in anticipation of other tasks below: moving the deploy process to a script, and running it as part of the individual partial build jobs, which will have already cloned the repo. After that is all complete, the review job will be removed entirely, and the review-prep-bucket will only be run once per-pipeline on the first run.
  • (!56769 (merged)) Refactor review/master deploy logic out of CI file into a single common script.
    • Introduce DEPLOY_TYPE env var with value of review or production.
    • Introduce DEPLOY_CLEANUP_OLD_DELETED_FILES env var which controls whether rsync deletion flag is used.
    • Introduce scripts/deploy script which includes logic from .common-gcp-setup, .review-rsync, and .deploy-rsync with appropriate commands executed based on above env vars.
    • Use scripts/deploy script from review-prep-bucket, review, and deploy jobs.
  • (!57327 (merged)) Add some informative logging to scripts/deploy, to indicate where it's deploying to.
  • (!56889 (merged)) Add deploy-cleanup-old-deleted-files scheduled job
    • Supports independent per-site production deploys by removing need for --delete flag on master GCP deploy rsync.
    • Introduces a separate "deploy-cleanup-old-deleted-files" scheduled job which does a full build of everything and deploys with the DEPLOY_CLEANUP_OLD_DELETED_FILES flag set.
    • This will use all artifacts from partial build jobs to ensure everything is still cleaned up, even if partial build jobs are added/deleted/refactored, and when top-level paths of the deploy are deleted/renamed.
    • This also resolves #7661 (closed)
    • (!58219 (merged)) Delete manual job to perform cleanup, it has unintended side effect of making a deploy to prod button show up, which could be incorrectly used to deploy an old version to production. See slack thread.
    • (!58397 (merged)) Delete review-deploy-cleanup-old-deleted job, it's unnecessary (can just stop and restart the review env if needed).
  • (!57276 (merged)) Refactor/reorganize/comment .gitlab-ci.yml config file.
  • (!57267 (merged)) Refactor build-handbook job to handbook-build-and-deploy, to leverage above changes and do a partial deploy of handbook as part of the job. This will serve as a template/example for continuing to refactor PartialBuild into individual jobs.
  • (!57449 (merged)) Move check-edit-links job to a separate post-deploy stage (or some other name based on standards from gitlab pipelines), and only have it run if changes to the handbook or partials/includes are made.
  • (!57451 (merged)) Refactor external asset pipeline handling
    • Remove rollup/pipeline.sh from being called by every job
    • Change the external_pipeline in main config to only be called outside of CI (only for local builds and development mode).
    • Create new dedicated js-bundle-build-and-deploy job which just directly calls rollup.js and deploys tmp/frontend/javascripts/ to public/javascripts and publish it as an artifact.
    • Remember to add as a dependency to review-prep-bucket and deploy-cleanup-old-deleted-files jobs.
    • (!57585 (merged)) Fix minification which was broken in original MR
  • (!57938 (merged)) Delete review-prep-bucket job. It is not as necessary anymore now that we are migrating to having each partial build job deploy its own assets, and has the possibility of causing problems if it overwrites new MR files due to a race condition because it is in the same build stage.
  • [-] Avoid having multiple CI jobs for each dedicated partial build/deploy job. Modify scripts/deploy.rb to contain the logic to switch accordingly. This isn't possible, because GitLab CI does not provide support for dynamically setting environment: values. We could eliminate the -build-only jobs by reusing one of the build-and-*-deploy jobs and passing a flag to the deploy script to make it a no-op, but this is would be more confusing than it currently is.
  • Iteratively refactor CI PartialBuild to move IMAGES and ASSETS builds into into individual build jobs (as has already been done for the handbook by this point). Note that this does NOT include team/pet images which are built as part of the COMPANY PartialBuild section
    • Note that some of the items below which have been strikethrough have been deferred, due to additional complexity.
      • REASON 1: Due to things like the coupling of company partials to team/pet images (see slack thread), it's not currently easy/possible to get some easy wins on total build time reduction by splitting into several small dedicated jobs (primarily because company is one of the longest-running ones, and it's not trivial to split up).
      • REASON 2: Because of the current architecture (and need to still have a common scheduled job which deletes old deployed files which have been deleted from the repo), each dedicated partial build has to have three separate jobs. If we split out everything, that would create many more jobs and make the ci config more confusing, so deferring that until we can revisit perhaps better approaches after the marketing monorepo move is complete.
    • Improvements from this refactor:
      • Supports individual build and deploy jobs for images and assets to be more loosely coupled and cohesive. This will make them easier to split up into separate sites in the future, but in the meantime still share middleman dependencies without having to explicitly declare them via data/monorepo.yml shared_files.
      • Increase performance and eliminate current build bottlenecks: Individual build jobs can be dependent upon modified files where possible (i.e. where there are no implicit dependencies via Middleman, e.g. images). See average job durations at the bottom of https://app.periscopedata.com/app/gitlab/561277/WIP:-www.gitlab.com-CI-CD-Pipeline-Metrics. Unfortunately, this will be more complex than initially anticipated, for reasons listed above.
      • Partial deploy can be performed as part of the build job, getting changes available on prod or review apps faster, and saving time/cost by eliminating need for dedicated deploy/review job. However, artifacts for each job must still be published, for the purposes of the review-prep-bucket and deploy-cleanup-old-deleted-files jobs.
      • Allows elimination of custom "blog post only" pipeline build without regressing on time-to-reviewability for blog post MRs. This is because they will no longer be bottlenecked on all other non-blog-post jobs finishing, they will deploy as soon as they are finished building.
    • (!57754 (merged) and !57794 (merged)) Split build 7/9 ("assets") into separate dedicated job
      • This first dedicated job will introduce extensions/destination_path_regexes_filter. This will use DESTINATION_PATH_REGEXES, which is a comma-delimited list of regexes which will be matched against paths under source (Middleman "resource.destination_path") to determine what will be built. The extension will be very similar to the manipulate_resource_list in extensions/monorepo.rb.
      • Update scripts/deploy to use DESTINATION_PATH_REGEXES. This is a comma-separated list of top-level paths to be partially deployed. Will need to introduce a loop with a separate rsync command for each path.
    • Split build 1/9 ("images") into separate dedicated job
      • (!57941 (merged)) Move devops-tools/azure_devops/images under top-level images directory
      • (!57943 (merged)) Switch from having middleman process them and just rsync the whole images directory to the public dir to be passed as an artifact.
    • [-] (!57568 (closed)) Split build 8/9 ("company") out of PartialBuild to separate dedicated jobs.
    • [-] Split build_proxy_resource out of PartialBuild into separate dedicated job(s) (This was done in !58221 (merged))
      • Try to understand why build_proxy_resource takes so much longer on a branch than on master. Something with caching?
    • [-] Split other remaining PartialBuild parts into dedicated jobs.
    • [-] Move build 9/9 ("all others") to a dedicated job and delete/rename PartialBuild.
      • This will have a big list of DESTINATION_PATH_REGEXES listing everything else. This will need to be kept updated when any new top-level directories are added.
      • Should also be accompanied/preceded by a cleanup to move all files out of top-level source directory except ones which must be there.
  • (!58092 (merged)) Add support for handbook sub-site build to properly serve assets and images in development mode.
  • (!58205 (merged)) Remove extra job worker slots from PartialBuild.
  • (!58221 (merged)) Change the proxy resource build jobs to do independent deploys as well.
  • (!58415 (merged) !58399 (merged)) Change the remaining PartialBuild build jobs to do independent deploys as well.
    • Remove review and deploy jobs, they are no longer needed at this point.
    • Update job names, use marketing and review/prod terminology, get rid of branch references, get rid of master references other than ones referring to actual master branch.
    • Reorganize/rename/add build stages to be more descriptive
    • Remove staging environment from CI config (rest of work for actual removal of the environment is on this issue: #8383 (comment 389280803))
  • (!57932 (merged)) Running bundle exec rake build:all locally should result in the same build as the CI pipeline.
    • This MR also includes some additional build/config/docs cleanup.
    • (!58419 (merged)) Revisit rake build:all after remaining CI changes, make sure it still does everything correctly.
  • (!57207 (merged))(!57723 (merged)) Eliminate custom "blog post only" pipeline build, and all associated symlinks.
    • (!58417 (merged)) Also remove as an entry in data/monorepo.yml, and consider any potential problems caused by this.
    • (!58418 (merged)) Remove remaining symlink by adding appropriate files.watch entries.
  • (#8383 (closed), !57878 (closed)) Decide what to do with the staging env and deploy_staging. Can we just allow redirects to be manually applied to the review env, and delete the staging env entirely?
  • Revisit/update pipeline performance metrics charts to account for new job names.: https://app.periscopedata.com/app/gitlab/561277/WIP:-www.gitlab.com-CI-CD-Pipeline-Metrics
    • Marked all old review/deploy charts as [OBSOLETE]
    • Added new bar charts showing average times of current *-build-and-(review|prod)-deploy jobs
    • Extended long-term total pipeline time graph to 1 year to show how much improvement we've made :)
  • (!58416 (merged)) Improve changes on check-edit-links job. See !57810 (comment 387117213)
  • (!58444 (merged)) For sub-site development mode, add Middleman redirects (https://middlemanapp.com/basics/redirects/) from root to /handbook (there's no root index file for sub-sites).
Edited by Chad Woolley