Skip to content

Create separate Middleman instance for the handbook/engineering and handbook/marketing

Chad Woolley requested to merge 6689-create-separate-middleman-instance-for-the-handbook into master

What does this MR do?

First step of pulling out handbook to a monorepo sub-site, without the need for permanent symlinks as currently used in the blog sub-site. This involves:

  • Moving the part of the site that is currently built by PartialBuild job "5/9" to a separate directory and build.
  • source/handbook/engineering and source/handbook/marketing will move to the new sites/handbook/source/handbook/marketing|engineering directories. source/handbook/index.html(.md) will also move.
  • On this MR for the first step, the directories and index file will not actually be moved, but only symlinked. This is a temporary approach, to reduce complexity if we need to revert, plus we need to give people warning about the move so they can close out MRs or be ready to rebase them. If it is stable with the symlinks for a few days, then we will do a follow-up MR to replace the symlinks with actually moving the files.
  • There will be a dedicated sites/handbook/config.rb Middleman config which builds only the moved files, and also contains necessary logic to allow the build to find all common/shared files which still live under the top level directory.

History

All of the work on this change up through 2020-06-01 was done on a separate exploratory/spike MR. See that issue for more details of the evolution and planning of the code in this MR, as well as notes taken on other topics which came up during the exploration. There were also many prerequisite or related MRs which have already been merged to master - those are all listed in that MR, as well as linked in the issue (#6689 (closed)) for this MR.

Testing

Testing approaches

Different testing approaches were needed for this MR. Here's a summary:

comparing generated files:

  • The output (i.e. public directory) of the build for the sub-site was compared to the output of the corresponding top-level master-branch version of the build, then the directories were diffed (via the Diff tool in RubyMine). It was verified that the generated directories were exactly identical. Here's the commands used for the two builds:
    • existing master build (top-level: git clean -dfx public && CI_NODE_INDEX=5 CI_NODE_TOTAL=9 bundle exec middleman build --bail
    • new sub-site build (from sites/handbook): git clean -dfx ../../public && time RAILS_ENV=test NO_CONTRACTS=true bundle exec middleman build --bail

ensuring no unexpected files got deployed

  • Since the review job now only deploys diffs compared to the latest version of master deployed by the review-prep-bucket job, it shouldn't be uploading anything which was not changed as part of the MR, as long as the branch is up-to-date with master. (NOTE: there may be couple of lingering things which still get unnecessarily deployed, but these should also be present on other MRs, and thus don't indicate a problem on this one).

manually reviewing generated site

  • Various functionality was explored in the review app. Important points:
    • partials which are used from the top-level
    • data-driven pages which use data from the top level (e.g. see usage helpers/team_helpers)
    • mermaid charts
    • javascript and styles used from top level
    • layouts used from top level

Testing Tasks

  • dev server (only works from top level, not sub-sites)
    • with files symlinked
    • with files moved
  • on a merge request
    • with files symlinked
    • with files moved
  • on master (on cwoolley-gitlab fork - failures are due to not having deploy credentials)
  • Make sure charts still work (https://6689-create-separate-middleman-instance-for-the-handbook.about.gitlab-review.app/handbook/engineering/development/dev/create-knowledge-be/index.html#hiring)
  • Manually run the integration tests (note that they've been disabled for months, and may have rotted).
    • Most of these ran locally; there were five failures. But this is expected, we disabled them because they were flaky when served via the Middleman dev server. Most of the failures looked familiar, like ones that were the usual flaky culprits.
  • Look at every file deployed by the review job, and understand what was different compared to the version deployed by review-prep-bucket, and why: 
    • Copying file://public/robots.txt [Content-Type=text/plain]...
Copying file://public/sitemap.xml [Content-Type=application/xml]...
Copying file://public/solutions/enterprise-class/enterprise-considerations.pdf [Content-Type=application/pdf]...
Copying file://public/solutions/reference-architectures/gitlab-reference-architectures.pdf [Content-Type=application/pdf]...
    • Notes on diffs:
      1. robots.txt - This is expected to differ on prod vs. review apps.
      2. sitemap.xml - This is only generated during the deploy job, so it's expected to always be uploaded.
      3. PDFs - These get generated, haven't looked into it, but there is probably a date or some other variable value inside the PDFs.

Related issues

Relates #6689 (closed)

/label ~"group::static site editor" ~backstage

Edited by Chad Woolley

Merge request reports