WIP: Middleman symlink-less file sharing (!49086) · Merge requests · GitLab.com / www-gitlab-com

Chad Woolley requested to merge cwoolley-middleman-symlinkless-file-sharing into master May 08, 2020

NOTE: This MR is complete and closed without merge. All incomplete tasks related to the handbook/engineering and handbook/marketing middleman build have been moved to the related monorepo issue, and all other incomplete or unrelated tasks have been moved to the top-level Handbook Improvement epic

What does this MR do?

This was a spike/exploration MR to iterate on what approaches and prerequisite tasks were needed. The actual work was done in #6689 (closed) and various other supporting MRs linked below.

There are issues with the symlink-based approach which was taken when doing this for the blog in !39183 (merged), because it has downsides:

it's confusing for people to see two copies of the same shared file in their editors
if you pick a symlink instead of the original file, some things don't work (e.g. git history in RubyMine)

So, if possible, we should avoid doing this four more times (which would result in people seeing six copies of all the shared files in their editors).

Instead, this MR will explore using the approach described here: https://github.com/middleman/middleman/issues/1902#issuecomment-286554690 (NOTE: I think the "duplicate partials path" problem mentioned there is not a real problem. I think that's just a side effect of the way Middleman looks for every file in every possible directory if it's not found - as long as I got destination_dir: "." correct as shown below, it worked for me)

Proof of concept for of including common content from outside the Middleman root dir with `files.watch`

If you make this change locally (note it deletes the layouts dir symlink), then run NO_CONTRACTS=true bundle exec middleman from the sites/blog directory, it will work (the layouts are found).

In theory, this allows us to have the monorepo grouped structure, where sites/handbook contains a custom middleman config and only group-specific content (as the blog already does) but without needing symlinks.

Index: sites/blog/config.rb
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- sites/blog/config.rb	(revision 71fc78eccdd1a7f545c730299532aa874ca1f2cf)
+++ sites/blog/config.rb	(date 1589263865116)
@@ -96,3 +96,11 @@
 ignore '/category.html'
 ignore '/.gitattributes'
 ignore '**/.gitkeep'
+
+# Monorepo setup to access common files
+
+files.watch(
+  :source,
+  path: File.expand_path('/Users/cwoolley/workspace/www-gitlab-com/source/layouts'),
+  destination_dir: "."
+)
\ No newline at end of file
diff --git sites/blog/source/layouts sites/blog/source/layouts
deleted file mode 100644
index e473c0618d580a4faf77c3ea3c38f70e061b5d96..0000000000000000000000000000000000000000
GIT binary patch
literal 0
Hc$@<O00001

Proof of concept of conditionally including composable Middleman Config files to implement sub-builds

(warning: brain dump ahead, needs organization and clarification)

Calling a sub-config from the top-level config

UPDATE: We probably won't end up using this approach of "composable config files", at least for now - the main reason being that we can't yet get rid of "PartialBuild" and the monolithic build approach it entails, so we will split things up in different ways. But, it's good to know that this is possible if we need it in the future*

The top-level config.rb can include other config files like this:

config_all_rb = File.join(Middleman::Application.root, 'config_all.rb')
instance_eval(File.read(config_all_rb), config_all_rb, 1)

This seems to work fine in dev. However, it is unknown whether more complex files can be successfully composed in this way.

What does this mean for us?

So, this could be used to conditionally include or exclude individual sub-builds based on some criteria. This opens up many possible options which we have been wanting.

For example to start with:

In dev and master, we could by default include all sub-builds.
But for MRs, we could conditionally only run the sub-builds based on which source files changed, which could be done by default on MRs (for speed) and via opt-in in development. The non-built files wouldn't be present, but we could try to work around that by "priming" the deployable artifacts with the latest artifacts from master.
There would also be some sub-builds which would always run, because they are always required. For example, certain common CSS or javascripts, and the top-level index page.
This approach should greatly speed up MR pipelines ⏩

...and it opens up for future optimizations:

Once we have this in place, we would finally be in a position (now that we have converted to rules) to be able to speed up the master build a lot by only building what has changed.
We would still have to deal with the "rsync deletion" problem, where master deploys the entire site to the prod bucket with the gcp rsync --delete option, so it will remove any deleted files, but also delete any files that weren't built in the current pipeline.
BUT, we could remove the deletion flag from normal master pipelines which only build what changed, then have a periodic scheduled job (every hour or day) which does a full build and deploys with the rsync --delete option.
This approach should greatly speed up master pipelines ⏩

Proof of concept spike tasks

Rough outline of tasks to finish proof of concept validation of the approaches described above. May not need to do all of these, or do them completely to be confident of the approach.

Part 1

Dedicated handbook partial build job. Note - this does not have the responsibility of building the asset bundles - that will remain at the top level for now. In the future, it can be delegated to individual sites to build their own bundles.

Part 2

This step and the following steps are currently tentative, we may not do them this way, not do them yet, or do them at all

Handbook-only builds for MRs

[-] ~~Make build_handbook job be the only one run if the only changes are to the files in the handbook sub-site (via rules)~~ Don't try to do this in the first pass - it's complicated and too much chance for error. Just keep building everything always, but with the sub-build.
"Prime" the rest of the full content to the review server with the artifacts from master. Doing this in a separate MR: !50598 (merged)
- NOTE: This may not currently be possible/reliable due to this bug
- Do this in a separate job that is not blocked by the build or test stages, to speed up the initial population, and make the deploy job dependent upon the prime job.
- Come up with a better name for the "prime" job. deploy_latest_master_to_review? Yes, review-prep.
- See https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#downloading-the-latest-artifacts
- ~~Will need to add an artifacts:paths entry to master deploy job to upload artifacts.~~ can't do this, it's over the 10M artifact upload size limit. Have to download the individual build jobs' artifacts instead
Update the review-prep job to match the actual build jobs being invoked.
Remove HANDBOOK_ENGINEERING_MARKETING case from PartialBuild (but leave the handbook_engineering_marketing? because other methods rely on it)
Remove all commented code in sites/handbook/config.rb

NOTE: For MR branches which are very out-of-date, review-prep job may cause inconsistencies or failures, when trying to run the older site code against the newer "primed" artifacts from master. However, the current "blog post" subset of the build doesn't even deploy non-blog files at all, so the experience shouldn't be any worse than that.

Part 3

Incorporate handbook partial build into master

Make a rake build_all job so we can still test building everything from local dev
Make master build work just like the MR build: building handbook engineering/marketing is the responsibility of the build-handbook job, and PartialBuild for this partial is a no-op.

Part 4

TESTING

Dev environment should still work:

Ensure that the top-level build still works for all content, with hot-reloading, asset rebuilds, etc.
- Test it with files actually moved, not just symlinked. May need to add files.watch to top-level build in order for it to work.

Make sure master works:

Fork it, and test behavior on master branch of the fork.

Part 5

Notify people to close out any pending MRs that touch files to be moved, or be prepared to resolve conflicts.

=========================================================================

END OF SPIKE as of 2020-06-02 - See the main issue for remaining/updated tasks and work.

=========================================================================

[-] Remove the symlink and actually move the files, then merge quickly.
[-] Merge to master.

Part 6

Convert blog to this approach

Note that the current partial build for the blog doesn't save much time on the overall build stage, at most a minute or so. The review step IS much slower, but this is because it is uploading everything. If we can use one or more of the approaches above of "priming" the review app bucket and/or skipping builds that aren't needed, these times should be equivalent.

full build: https://gitlab.com/gitlab-com/www-gitlab-com/pipelines/147325082/builds
blog-only build: https://gitlab.com/gitlab-com/www-gitlab-com/pipelines/147321342/builds

Other related/prerequisite tasks

Out-of-scope future tasks

Notes on tasks which are out of scope for this spike:

Asset builds should be per-site. Instead of the top level build being responsible for building all asset bundles, it can be delegated to individual sites to build their own bundles.
caching-optimized proxy resource build jobs should be set up per-site.
Deciding what to do about redirects. Do these have to be managed monolithically, or can they be delegated to the responsible sites?
Repeat the pattern/process taken for the handbook engineering/marketing build in this and related MRs for all other sub-sites, including switching the blog from symlinks to this approach.
Get the monkey patch for files.watch into Middleman and/or Frontman.

Radical Idea:

Get rid of the separate "deploy" job, and make each build job responsible for publishing its own artifacts. This may never be possible or desirable, because of the need to coordinate the deploy of dependent artifacts across separate jobs (e.g. js/css/image assets used by pages). But for sub-site builds which are a single build job, it would speed things up a lot, by eliminating the overhead of spinning up a separate job just to do the rsync deploy.

Other debugging notes

Add logging to middleman-core-4.3.3/lib/middleman-core/sources.rb Sources#find
Add logging to middleman-core-4.3.3/lib/middleman-core/sources/source_watcher.rb SourceWatcher#find

Tech Debt Cleanup Notes

Unrelated things I saw which should be cleaned up, making a note of them here for future MRs...

[-] Fix calls to Google cloud SDK rsync to not print verbose progress and fill up logs. UPDATE: Looks like this may not be possible without suppressing all output including filenames:
- https://groups.google.com/forum/#!topic/gce-discussion/J641jfYvOak
- https://cloud.google.com/storage/docs/gsutil/addlhelp/TopLevelCommandLineOptions
Move all non-top-level pages which are referenced as partials out from under source/handbook/engineering and ``source/handbook/marketing, so by default they are not built and output to public` and explicit `ignore` entries are not needed to prevent them being built and output.
- Write a linter task to enforce this. E.g. hook into middleman and identify anything used as a partial, and fail the build if these exist under the list of resources allowed by manipulate_resource_list
Rename the package.json "bundle" task and sub-scripts to something consistent and descriptive like build_assets, instead of combination of other overloaded terms like bundle, pipeline.
Fix this comment in config.rb: "# Populate the direction and releases pages only ~~on master~~ in the build_proxy_resource_* jobs or development machines."
Remove usages of autoload, e.g. in lib/homepage.rb, because autoload is going away and requires hacking around it by adding the monorepo root to the $LOAD_PATH in the sites' config.rbs
Combine rm public/sitemap.xml into extract_sitemap_urls
Eventually get rid of PartialBuild.
- It relies on assumptions that will no longer hold:
  1. Entire site will be built by PartialBuild
  2. There will be a consistent number of partial or sub-site builds.
- But, it is still needed because it defines some partial builds in terms of others, e.g. all_others?
includes/hiring-chart.html.erb should use a stable ID rather than a random number <% unique_id = "Chart#{rand(10000..99999)}" %>, to make it easier to do full-directory diffs of public when looking for regressions in sub-site monorepo builds.

Related issues

Relates #6689 (closed)

Edited Jun 04, 2020 by Chad Woolley

WIP: Middleman symlink-less file sharing

What does this MR do?

Proof of concept for of including common content from outside the Middleman root dir with files.watch

Proof of concept of conditionally including composable Middleman Config files to implement sub-builds

Calling a sub-config from the top-level config

What does this mean for us?

Proof of concept spike tasks

Part 1

Part 2

Part 3

Part 4

Part 5

=========================================================================

=========================================================================

Part 6

Other related/prerequisite tasks

Out-of-scope future tasks

Other debugging notes

Tech Debt Cleanup Notes

Related issues

Merge request reports

Proof of concept for of including common content from outside the Middleman root dir with `files.watch`