Create Middleman extension to combine multiple data files into single data object
Problem to solve
We have some data files in the www-gitlab-com repo that is becoming very large and would prefer to break them up into smaller parts.
To limit the impact on existing code we want to be able to break up the data file into multiple parts but have it combined and accessible under a single key in the data
object that it currently resides under.
Details
Middleman has existing functionality that allows you to break up data files, see https://middlemanapp.com/advanced/data-files/, however this only works for nested objects and not collections. The data files we want to break up are primarily collections.
Example data file: https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/performance_indicators.yml
Requirements
Enable the Middleman project build process to read data from multiple data sources and combine it into a single object accessible in the data
object.
Example
Given the following data files:
|- data
|- performance_indicators
|- engineering.yml
|- product.yml
|- peopleops.yml
At build time the collection entries from the partial data files (engineering.yml, product.yml and peopleops.yml) should be accessible under a single key in the data object called: data.performance_indicators
as if they resided in a single data file called data/performance_indicators.yml
What to watch out for
Since there is an existing pattern that middleman uses to concatenate data files as well as sub-folders in the /data/ folder for the www-gitlab-com repo, we should not take a convention approach where this extension automatically applies this concatenation technique to all sub-folders in /data/ but should rather require it to be configured.