Add an environment variable to enable quicker template development

Why is this change being made?

Developing templates in Middleman with the preview server is exceedingly slow. There have been many epics, issues, MRs, and meetings about it. Right now the digital experience is exploring these challenges.

This MR provides an opt-in monkeypatch to two internal Middleman methods, which will allow a developer to make changes to Middleman template files without triggering a full sitemap rebuild.

Manual QA instructions

1. Check out the branch locally
2. Open the www-gitlab-com project
3. cd sites/marketing
4. DISABLE_ON_DISK_REBUILD=true NO_CONTRACTS=true bundle exec middleman serve (you can use --verbose and --instrument as well if you want to see what the call stack looks like/compare it without the new env variable).
5. The Middleman server startup time will be about the same as ever - this monkeypatch doesn't kick in until after the server has started at least once.
6. Once the server is fully running, visit http://localhost:4567/topics/application-security/
7. Make an edit to sites/marketing/source/includes/cms/topic/header.html.haml - I usually just delete an h1 or something that will be noticeable.
8. Reload the page (keep in mind you have to opt in to the livereload option - which may slow you down.
9. There's still a couple second delay before the change takes effect. Middleman still has to watch and render this file out of thousands. But it's more like 2-3 seconds, as opposed to the previous 1-2 minutes.
10. Revert your change, refresh the page
11. To see what I mean about data files, go make a change to data/topic/application-security.yml. This will instruct Middleman to get ready to change the sitemap.
12. Refresh the page - Middleman will know that a data file has changed, and it will regenerate the sitemap (since that step is required to render out the YAML content). This will take the amount of time we're used to (closer to a minute or so).
13. Shut down the server
14. To make sure the existing, expected behavior still works, run NO_CONTRACTS=true bundle exec middleman serve
15. Repeat all those changes - observe the difference in speed for the templating changes specifically.

In depth details for the interested

Initial intuition

After my day off on Friday, and some space away from Middleman, I woke up on Saturday with some inspiration about how we could pick up a lot of efficiency.

My first thought was to do something like Chad Wooley's OnlyDebuggedResources extension. Full MR here.

But even targeting specific destination paths or resources was still taking longer than I think we would like.

How the Middleman preview server works

I had some initial misunderstandings about what the Middleman preview server is doing behind the scenes. I thought it was running the equivalent of bundle exec middleman build and then acting as an HTTP server from the generated public/ dir.

Turns out, that's not exactly what it's doing. It is actually acting as a dynamic server and doing something similar to server-side rendering. In that Rack file, Middleman preview server is doing something like this (as far as I can estimate):

process_request runs on all call requests, and it builds a request_path from the Rack environment. Then it queries the @middleman.sitemap for the resource that matches, using by_destination_path.

At that point, it calls render on the Resource object that the sitemap gives back. That render call dynamically renders through the data and templates and returns a response - all in memory, rather than acting as just a static file server. Or rather - it does that unless the file is static, in which case it just calls send_file.

But regardless - the underlying Rack server in the Middleman preview server has the capability to dynamically render pages.

Which made me wonder: why is it so slow if it's not necessarily rebuilding the whole site?

The slowest method in the call stack: ensure_resource_list_updated!

Turns out that a variety of actions in the watched files trigger a method called `ensure_resource_list_updated!. This method basically rebuilds the entire sitemap, and calls through all of the sitemap manipulators that are registered. There's a comment in the source code that correctly identifies this operation as being very expensive. I think that's where most of our pain points are coming from. And it's a necessary step because, as the maintainer points out

This is necessary since in some templating languages, all files of the same type can be included in each other. A change to one could change the render of all the others. Haml does not provide us a dependency graph to know if its safe not to reload all the files.

We can totally disable this behavior in the config.rb file by simply setting @app.config[:disable_sitemap] to true. But if we do that in configuration - Middleman won't know anything at all about the website. It will build nothing.

So I wanted a middle-of-the-road option. In most of our day-to-day work, we only care about editing a handful of templates, some CSS, and some JavaScript. Since we're now serving CSS and JS from webpack without Middleman rebuilds - the question becomes: "can we find a way to make targeted changes to a small sampling of templates without triggering a full sitemap rebuild?".

This MR says yes, in the form an opt-in monkey patch, enabled via the DISABLE_ON_DISK_REBUILD environment variable.

The actual monkeypatch

When you set DISABLE_ON_DISK_REBUILD=true, the monkeypatch will set a custom config property in the ready callback.

def ready 
  @app.config[:disable_on_disk_rebuild] = true if ENV['DISABLE_ON_DISK_REBUILD']
end

Which then augments Middleman::Sitemap::Extensions::OnDisk::update_files to return early and skip calls to rebuild_resource_list! and ensure_resource_list_updated!.

I think this is fine, so long as you have only changed existing templates, and not made any changes to the source files that make up the sitemap. For example, changing the HAML in sites/marketing/source/includes/cms/topic/benefits.haml will still work, but changing the proxy rules in config.rb will not.

For that reason, I also left the Middleman::CoreExtensions::Data::Stores::LocalFileDataStore::update_files method alone. That's what queues up resource rebuilds when files in the data folder get changed.

Limitations: there are probably a lot?

I still don't have enough context on this project to know about the sharp edges here, so I'd really love some feedback from @laurenbarker and @brandon_lyon about what dragons lie ahead of me.

If this seems like a net benefit for us, I'd like to merge it in, try it out, and resolve bugs as they come up. If we find that it's really making improvements in productivity, I might build it out as a proper Middleman extension gem and send it over to the Middleman maintainers to see if we can get some hooks into the Middleman source to make this a little less hacky. Right now the whole thing could unravel if Middleman changed some of these methods out from underneath us.

@cwoolley-gitlab - CCing you for visibility, although I've only enabled this in the marketing site config since I know handbook is undergoing a migration, so I'm not sure if it's worth the trouble to you. But it was also largely inspired by some of the extensions you've written, so if you have a moment to give it a look, I'd be grateful. No worries if not.

Also CCing @jgarc257 @lduggan @sanmiayotunde @smcguinness1 and @mpreuss22 since we have an SSG workshop scheduled for tomorrow and I think this patch will be relevant to the conversation if we end up merging it in.

If we like this, I'll also add a note to doc/development.md about it, and maybe the README, but I wanted to get some collaboration on this before I wrote up any docs in the repo itself.

Author Checklist

• Provided a concise title for the MR
• Added a description to this MR explaining the reasons for the proposed change, per say-why-not-just-what
• Assign this change to the correct DRI