Feature Proposal: Replace frontend dispatcher with intelligent code-splitting

Background - Conditional Javascript

We currently have several ways of conditionally applying javascript to a specific page:

1. Page-specific javascript bundles

Our current, documented method for applying javascript to a given page is through page-specific javascript bundles. We add an "entry point" to our webpack.config.js which creates a new bundle with the code we point it to. We then conditionally include this bundle in the header by adding something like the following in our rails views (haml):

- content_for :page_specific_javascripts do
  = webpack_bundle_tag 'common_vue'
  = webpack_bundle_tag 'filtered_search'

Pros:

the code in a given bundle is isolated from the rest of the codebase, so we aren't including unnecessary code on pages where it isn't needed

Cons:

requires a lot of manual setup (update webpack config, add script tag to rails views)
our webpack config is getting very messy (there are 53 entry points in the CE webpack config alone!), and we have no documented naming convention or standard bundle directory structure that we consistently use across our codebase.
code shared between bundles ends up necessitating manually-generated "commons chunks" in the webpack config to prevent libraries from sneaking into the main bundle, and these common "parent" chunks need to be loaded in a particular order before the bundle that depends on them.

2. Dispatcher.js

We have a gigantic dispatcher.js module which is included in our main bundle and executed on each page. It conditionally instantiates objects and calls functions based on a data-page attribute of the <body> tag. This looks something like "projects:issues:new". The dispatcher is basically a large series of switch statements with cases for dozens of potential pages.

Pros:

no configuration changes, very simple to understand and implement
has potential to organize our code well by page if done right

Cons:

all code is included in the same bundle, whether or not it is ultimately executed on the page. this has network and code parsing performance implications.
this is a magnet for EE merge conflicts and has become so gigantic and messy that it is actively avoided by many developers

3. Code Splitting within dispatcher.js

This is currently only used in a few places, but we can create asynchronous webpack "chunks" automatically through dynamic import() statements. It looks like the following:

// taken from dispatcher.js
import(/* webpackChunkName: 'user_profile' */ './users')
  .then(user => user.default(action))
  .catch(() => {});

Pros:

no config necessary (the chunk name is actually configured in the special comment tag seen above)
only downloads and executes the given code on pages that need it

Cons:

still has same drawbacks as dispatcher.js, an unruly mess that is prone to merge conflicts
not currently documented, and prone to abuse if not understood correctly
async "chunks" are not fetched until the dispatcher code is parsed and executed, and pre-fetching would require manual <link rel="prefetch" /> tags.

4. Selector-based execution

We have dozens of legacy modules that do something akin to the following:

// from importer_status.js

$(function() {
  if ($('.js-importer-status').length) {
    var jobsImportPath = $('.js-importer-status').data('jobs-import-path');
    var importPath = $('.js-importer-status').data('import-path');

    new window.ImporterStatus(jobsImportPath, importPath);
  }
});

These execute on every page, and basically use the existence or non-existence of a selector (in this case, .js-importer-status) to determine whether some code gets executed, events get bound, etc.

Pros:

none, this is terrible

Cons:

this is literally the worst of both worlds

Proposal - a new page-based loader

I want to create a solution to this problem that takes the best bits of the options above and has none of the downsides. Some goals:

It should make our main.bundle.js entry point extremely lean by only including code that must be on every page. All other code should be loaded asynchronously in small page-specific chunks.
It should be simple to configure and understand, with no requirement to make changes to webpack config, or add code to rails views.
It should be far less prone to CE/EE merge conflicts.
It should not require manual pre-fetching to optimize performance
It should promote good programming practices, help us naturally organize our codebase and untangle our dependency tree.
It should simplify extracting common code from multiple chunks into async commons bundles that are loaded automatically when needed.

All of this is possible, and reasonably straight forward to achieve.

Design Steps:

We throw out dispatcher.js and replace it with a directory structure based on the <body data-page="my:current:page" /> attribute.

We'll create pages.js to replace dispatcher.js, and this will basically take the data-page attribute, split it by : and check for a directory that matches the first component, then asynchronously load a corresponding script from the /pages directory if it exists:
```
const [base, ...location] = document.body.dataset.page.split(':');

import(`./pages/${base}/index.js`)
  .then(page => page.default(location))
  .catch(() => { /* handle error, ignore if no script exists */ })
```
Example: on an issue page, with data-page="projects:issues:show", pages.js will load a webpack chunk for /pages/projects/index.js. This file should have a single default export function which gets passed the remainder of the data-page attribute (in this case ['issues', 'show']). It can then import any modules that are specific to projects:* pages and then load another sub-directory like /pages/projects/issues/index.js that does the same with all scripts pertaining to the projects:issues:* pages, and so on.
We take each one of our webpack entry points and convert them into index.js files within /pages/**/*, removing them from our webpack config, and removing the page_specific_javascripts directives from corresponding haml files. In the end our webpack.config.js will only have one entry point, one vendor bundle, and one runtime bundle. Everything else will be inferred chunk split points.
Move all other page-specific import and instantiation logic into these /pages/**/* style directory structures.

Example: If we have a module that is only needed when the user is in the project wiki page (like wikis.js), it should only get included and instantiated from within /pages/projects/wikis/index.js. (we could move wikis.js to /pages/projects/wikis/wikis.js as well)
We implement a very simple webpack plugin that will name chunks within the /pages/ directory automatically so we can avoid needing /* webpackChunkName: 'wiki_pages' */-style comment blocks. These names will all be exported into our webpack manifest where we can create a custom rails helper to automatically include <link rel="prefetch" /> tags for these scripts when they match the current page.

Example: On data-page="projects:issues:show", we will automatically include <link rel="prefetch" href="/assets/webpack/projects_issues_show.chunk.js" /> if such a script exists.
We update our frontend coding standards docs to enforce a strict demarcation between /page/**/index.js loader modules which import and instantiate versus pure-modules which define and export (virtually all other modules should fall in this category). We can eliminate side effects, make our codebase much easier to parse and understand, and make our modules much more testable.
We add async CommonsChunkPlugin definitions to our webpack config to auto-load libraries like d3 on-demand. Unlike non-async commons chunks, these work like magic without any extra script tags or manual import() calls. If we call import('./users') and users.chunk.js depends on d3, then a chunk containing D3 will be asynchronously loaded at the same time with no other intervention required!

Edited Sep 13, 2017 by Mike Greiling