Commit f56e426e authored by Mike Street's avatar Mike Street
Browse files

Create post about 11ty and rss/xml

parent 0d0958c2
---
title: Build an 11ty calendar to list all your posts
intro: List all of your posts (or other collection items) in a calendar/diary format to give a timeline of your past
date: 2021-04-12
permalink: "blog/build-an-11ty-calendar-to-list-all-your-posts/"
tags:
- 11ty
- Javascript
---
{% raw %}
I wanted to create a calendar/diary page for all my posts so I could have an overview of my post history. After writing about my [writing schedule](/blog/my-2021-writing-schedule/) and making sweeping generalisations about my posting past, I thought it would be nice to create a single page that listed out all my posts by date.
I have previously thought about doing this, but the performance implications of doing it with a dynamic CMS have held me back. With my move to 11ty, the page can be build and rendered as static HTML. What a dream!
## Finished Result
If you would like to see what this looks like, you can head to [the diary](/diary). If you're interested in the code, you can [find it on Gitlab](https://gitlab.com/mikestreety/mikestreety/-/blob/master/app/content/diary.njk) with the [accompanying data file](https://gitlab.com/mikestreety/mikestreety/-/blob/master/app/content/diary.11tydata.js).
## Grouping by year
I originally looked to have my posts listed by year and [this Gitlab solution](https://github.com/11ty/eleventy/issues/1284#issuecomment-648749730) was a great start. However, I wanted to enhance this by month and day too.
I started manipulating the code to duplicate the `year` functionality with `month`. However, to get it
semantically nested with `ol` inside the `li` was turning into a bit of a headache
```html
{% set entryYear = entry.date.getFullYear() %}
{% set entryMonth = entry.date.getMonth() %}
{% if currentYear != entryYear %}
<h3>{{ entryYear }}</h3>
<ul>
<li>
{% endif %}
{% if currentMonth != entryMonth %}
{% set currentMonth = entryMonth %}
<h4>{{ entryMonth }}</h4>
<ul>
<li>
{% endif %}
```
## Page data & data files
While trying to bend Nunjucks to my will, I was constantly thinking about how much easier this would be if the array was already formatted by year and month for me to loop through.
I considered making a [global data file](https://www.11ty.dev/docs/data-global/), but as it was for one page, I thought I would keep the data close to the file.
Originally I was developing with the JavaScript code in the Front Matter:
```
---js
{
title: "Diary",
description: "All of my posts in one place, by year and month",
layout: "page.njk",
diary: function() {
// ...
}
}
---
```
However, after reviewing the size of the code, I opted to use a [directory specific data file](https://www.11ty.dev/docs/data-template-dir/) by creating a `diary.11tydata.js` alongside my `diary.njk`.
This makes the variables declared in here available in the corresponding file (or folder). I've used directory data files for other parts of my site, including applying a draft status to all [draft posts](https://gitlab.com/mikestreety/mikestreety/-/blob/master/app/content/drafts/drafts.json).
## The collection code
So this is what you are here for, the code.
You can see the final file [on Gitlab](https://gitlab.com/mikestreety/mikestreety/-/blob/master/app/content/diary.11tydata.js), but thought I would walk through each bit here.
First step is to declare some date related functions - getting nice month names along with calculating the date ordinal (e.g. 3**rd** or 19**th**)
```js
const month_names = Array.from({length: 12}, (e, i) => {
return new Date(null, i + 1, null).toLocaleDateString("en", {month: "long"});
})
const nth = function(d) {
if (d > 3 && d < 21) {
return 'th';
}
switch (d % 10) {
case 1:
return 'st';
case 2:
return 'nd';
case 3:
return 'rd';
default:
return 'th';
}
}
```
Next we export the module as a JavaScript object, with a function keyed as `diary`, which we will call in our template
```js
module.exports = {
diary: function() {
// Code goes here
}
}
```
Rather than walk through each of the bits of code, I've included it below commented:
```js
module.exports = {
diary: function() {
// Select the collection we want to loop
let entries = this.ctx.collections.blog,
// Create our placeholder array
output = [];
// Loop through each of the entries
for(let item of entries) {
// Check we have both a date and title
if(item.data.title && item.date) {
// Extract the year and month number (Jan = 0)
let year = item.date.getFullYear(),
month = item.date.getMonth();
// If the year hasn't been seen before, make a stub object
if(!output[year]) {
output[year] = {
title: year,
months: []
};
}
// If the month hasn't been seen before, make a stub object
// with a nice month name as the title
if(!output[year].months[month]) {
output[year].months[month] = {
title: month_names[month],
entries: []
};
}
// Add the entry to the keyed year/month array - only add the info we need
output[year].months[month].entries.push({
title: item.data.title,
url: item.url,
// This is just the date plus ordinal (e.g. 23rd)
date: item.date.getDate() + nth(item.date.getDate()),
});
}
}
// Return our array
return output
// Reverse the months (most recent first)
.map(y => {
y.months.reverse();
return y;
})
// Filter out any null years
.filter(a => a)
// Reverse the years (recent first)
.reverse();
}
}
```
## Displaying the posts
With our collection in a loop-able format, we can create several nested loops to output each of the entries. The resulting output is:
- Year
- Month
- Date - Entry 1
- Date - Entry 2
```html
<ol class="diary">
{% for year in diary() %}
<li>
<div><h2 id="{{ year.title }}">{{ year.title }}</h2></div>
<ol>
{% for month in year.months %}
<div><h3>{{ month.title }}</h3></div>
<ol class="diaryEntries">
{% for entry in month.entries %}
<li>{{ entry.date }} - <a href="{{ entry.url }}">{{ entry.title }}</a></li>
{% endfor %}
</ol>
{% endfor %}
</ol>
</li>
{% endfor %}
</ol>
```
The extra `<div>` elements are there purely for styling purposes (Flexbox FTW).
{% endraw %}
......@@ -11,7 +11,10 @@ tags:
{% raw %}
Following on from my previous post on [Creating an 11ty collection from a JSON API](/blog/creating-an-11ty-collection-from-json-api/), this post will cover creating a collection from an RSS feed instead.
<div class="info">
<p>This method parses the RSS feed as actual RSS which makes the feed slightly easier to process. If you are looking to extract more meta data or unconventional information stored in the feed, it might be worth reading the next post, <a href="/blog/making-an-11ty-collection-from-a-remote-xml-file/">Making an 11ty collection from a remote XML file</a>.</p>
<p>This post also builds on previous methodologies covered in a previous post - <a href="/blog/creating-an-11ty-collection-from-json-api/">Creating an 11ty collection from a JSON API</a></p>
</div>
A friend of mine is a [YouTuber](https://www.youtube.com/feeds/videos.xml?channel_id=UCEFZ7yABV_j9Ts3U18pZ1vw) and was enquiring about having a landing page which auto updated with his latest videos. My mind immediately jumped to 11ty and have a static site which rebuilds on a new feed item.
......
---
title: Making an 11ty collection from a remote XML file
intro: INTRO
date: 2021-07-29
permalink: "blog/making-an-11ty-collection-from-a-remote-xml-file/"
tags:
- RSS
- 11ty
- Javascript
---
{% raw %}
<div class="info">This method processes the RSS feed as raw XML which exposes more information but could be a bit more fragile. If you are looking for a way to process the RSS in a more robust, straight-forward way then read the previous blog post - <a href="/blog/create-11ty-collection-from-rss/">create an 11ty collection from any RSS feed</a>.</div>
After I wrote my previous post on [creating a collection from RSS](blog/create-11ty-collection-from-rss/), I noticed there was a disparity between the RSS feed I was given and the data I was getting out of the Javascript.
<strong class="info">TL:DR;</strong> I don't need the intro fluff, <a href="#final-code">take me to the finished code</a>!
The reason for this is the YouTube API contains several non-standard tags. If you look at [Tom Scott's](https://www.youtube.com/feeds/videos.xml?channel_id=UCBa659QWEk1AI4Tg--mrJ2A) RSS feed, you can see there are several tags which don't adhere to the [RSS specification](https://validator.w3.org/feed/docs/rss2.html).
For example **channel and video information**:
```xml
<yt:videoId>OOWcTV2nEkU</yt:videoId>
<yt:channelId>UCBa659QWEk1AI4Tg--mrJ2A</yt:channelId>
```
and **media meta data**:
```xml
<media:group>
<media:title>The Difference Between High Explosives and Low Explosives</media:title>
<media:content url="https://www.youtube.com/v/OOWcTV2nEkU?version=3" type="application/x-shockwave-flash" width="640" height="390"/>
<media:thumbnail url="https://i4.ytimg.com/vi/OOWcTV2nEkU/hqdefault.jpg" width="480" height="360"/>
<media:description>I didn't even realise that "low explosives" were a thing; let's talk about deflagration, detonation, and how high explosives can actually be safer. Thanks to Steve from Live Action FX: http://liveactionfx.com/ Filmed safely: https://www.tomscott.com/safe/ Camera: Simon Temple http://templefreelance.co.uk Edited by Michelle Martin: https://twitter.com/mrsmmartin I'm at https://tomscott.com on Twitter at https://twitter.com/tomscott on Facebook at https://facebook.com/tomscott and on Instagram as tomscottgo</media:description>
</media:group>
```
The plugin we were using was parsing the feed _as_ RSS and, in doing so, removed any non-standard tags and attributes.
RSS is just XML, which kind of means you can make your own tags up and, because it is just XML, we can parse is as such.
## Load the parser
Instead of the RSS parser loaded in the previous post, we are going to install a XML parser. Along with that, we need to install `node-fetch`, so we can `fetch()` the remote RSS file
```bash
npm i fast-xml-parser node-fetch --save
```
then include them in the top if your data file:
```js
const parser = require('fast-xml-parser');
const fetch = require('node-fetch');
```
## Fetch the feed and parse it
Once we `fetch` the feed URL, we need to process the response as text. As per the module instructions, we need to create a traversal object before converting to JSON.
```js
// Placeholder options object
let options = {};
// Fetch the feed and parse as text
let feed = await fetch(rss_feed)
.then(data => data.text());
// Create a JSON object
let json = parser.convertToJson(
// Create a tr
parser.getTraversalObj(feed, options),
options
);
```
We now have a `json` variable available which, if parsing the YouTube feed for example, has each feed item available ad `json.feed.entry`.
## Configure the parser
The XML parser has [several configuration options available](https://github.com/NaturalIntelligence/fast-xml-parser#xml-to-json) for converting to JSON. For working with the YouTube RSS, I found most of the defaults sufficient, however there were a couple I wanted to tweak centred around attributes.
Within the RSS, there are several pieces of information stored as attributes instead of entities. For example, the video thumbnail is in the RSS like
```xml
<media:thumbnail url="https://i1.ytimg.com/vi/LyfnoEa-P58/hqdefault.jpg" width="480" height="360"/>
```
By default, the XML parser omits this attributes, setting `ignoreAttributes` to true enabled these.
When enabled, any attributes were then prefixed with `@_` - this helps identify which items were originally attributes in the RSS. I didn't want this and so, set the `attributeNamePrefix` to `''`.
```js
let options = {
attributeNamePrefix: '',
ignoreAttributes: false,
};
```
## Tweaking the result
As per the original article, there were a few tweaks I wanted to make to the RSS data to make it more accessible within the template code. With the raw XML, this included making the thumbnail and video code more accessible:
```js
let data = json.feed.entry.map((video) => {
video.code = video['yt:videoId'];
video.image = video['media:group']['media:thumbnail'].url;
video.date = video.published;
return video;
});
```
With these changes, I didn't need to alter the template from the original code at all:
```html
{% for video in videos %}
<a href="https://www.youtube.com/watch?v={{ video.code }}">
<img src="{{ video.image }}" width="120">
<h3>{{ video.title }}</h3>
</a>
{% endfor %}
```
The advantage of using the raw XML means we could expose things like "view count" or "rating out of 5", which is also found in the RSS
<a name="final-code"></a>
## The complete code
With the changes above, our 11ty data file now looks like:
```js
const parser = require('fast-xml-parser');
const fetch = require('node-fetch');
const rss_feed = 'https://www.youtube.com/feeds/videos.xml?channel_id=[channel_id]';
module.exports = async function() {
let options = {
attributeNamePrefix: '',
ignoreAttributes: false,
};
let feed = await fetch(rss_feed)
.then(data => data.text());
let json = parser.convertToJson(
parser.getTraversalObj(feed, options),
options
);
let data = json.feed.entry.map((video) => {
video.code = video['yt:videoId'];
video.image = video['media:group']['media:thumbnail'].url;
video.date = video.published;
return video;
});
return data;
};
```
## Bonus: Make single pages
It may be that, along with a listing, you want to make a single/individual page too.
This can be done by generating a slug in the data file, and then making a single view with 11ty pagination configuration.
First, create a `slugify` filter and include it at the top of your 11ty data file. How you can do this can be found in a previous blog post: [Accessing 11ty filters within data files](/blog/accessing-11ty-filters-within-data-files/).
Next, when looping through your entries in the `map()` function, add an extra attribute of `slug`:
```js/1
let data = json.feed.entry.map((video) => {
video.slug = `/video/${slugify(video.title)}/`;
video.code = video['yt:videoId'];
video.image = video['media:group']['media:thumbnail'].url;
video.date = video.published;
return video;
});
```
Next, create a new page (I called mine `video.njk`) and, in the front-matter, set up pagination using the video data with a size of 1
```
---
pagination:
data: videos
size: 1
alias: video
permalink: "{{ video.slug }}"
---
```
11ty will build pages for each of the videos in the RSS feed. You will then have `video` available to you to build the rest of the page:
```html
<main class="single">
<h1>{{ video.title }}</h1>
<div class="video">
<iframe width="560" height="315" src="https://www.youtube.com/embed/{{ video.code }}" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
</div>
</main>
```
{% endraw %}
// Get month names
const month_names = Array.from({length: 12}, (e, i) => {
return new Date(null, i + 1, null).toLocaleDateString("en", {month: "long"});
})
// Work out date ordinal
const nth = function(d) {
if (d > 3 && d < 21) {
return 'th';
}
switch (d % 10) {
case 1:
return 'st';
case 2:
return 'nd';
case 3:
return 'rd';
default:
return 'th';
}
}
module.exports = {
diary: function() {
// Select the collection we want to loop
let entries = this.ctx.collections.blog,
// Create our placeholder array
output = [];
// Loop through each of the entries
for(let item of entries) {
// Check we have both a date and title
if(item.data.title && item.date) {
// Extract the year and month number (Jan = 0)
let year = item.date.getFullYear(),
month = item.date.getMonth();
// If the year hasn't been seen before, make a stub object
if(!output[year]) {
output[year] = {
title: year,
months: []
};
}
// If the month hasn't been seen before, make a stub object
// with a nice month name as the title
if(!output[year].months[month]) {
output[year].months[month] = {
title: month_names[month],
entries: []
};
}
// Add the entry to the keyed year/month array - only add the info we need
output[year].months[month].entries.push({
title: item.data.title,
url: item.url,
// This is just the date plus ordinal (e.g. 23rd)
date: item.date.getDate() + nth(item.date.getDate()),
});
}
}
// Return our array
return output
// Reverse the months (most recent first)
.map(y => {
y.months.reverse();
return y;
})
// Filter out any null years
.filter(a => a)
// Reverse the years (recent first)
.reverse();
}
}
---js
{
title: "Diary",
description: "All of my posts in one place, by year and month",
layout: "page.njk",
diary: function() {
let entries = this.ctx.collections.blog;
let output = [];
const month_names = Array.from({length: 12}, (e, i) => {
return new Date(null, i + 1, null).toLocaleDateString("en", {month: "long"});
})
const nth = function(d) {
if (d > 3 && d < 21) return 'th';
switch (d % 10) {
case 1: return "st";
case 2: return "nd";
case 3: return "rd";
default: return "th";
}
}
for(let item of entries) {
if(item.data.title && item.date) {
let year = item.date.getFullYear(),
month = item.date.getMonth(),
month_name = month_names[month];
if(!output[year]) {
output[year] = {
title: year,
months: []
};
}
if(!output[year].months[month]) {
output[year].months[month] = {
title: month_names[month],
entries: []
};
}
output[year].months[month].entries.push({
title: item.data.title,
url: item.url,
date: item.date.getDate() + nth(item.date.getDate()),
});
}
}
return output
.map(y => {
y.months.reverse();
return y;
})
.filter(a => a).reverse();
}
}
---
title: Diary
description: All of my posts in one place, by year and month
layout: page.njk
---
<h1>Diary</h1>
<p>All my blog posts, chronologically ordered from now back in time.
<div class="post-listing">
<ul class="diary">
<ol class="diary">
{% for year in diary() %}
<li>
<div><h2 id="{{ year.title }}">{{ year.title }}</h2></div>
<ul>
<ol>
{% for month in year.months %}
<div><h3>{{ month.title }}</h3></div>
<ul class="diaryEntries">
<ol class="diaryEntries">
{% for entry in month.entries %}
<li>{{ entry.date }} - <a href="{{ entry.url }}">{{ entry.title }}</a></li>
{% endfor %}
</ul>
</ol>
{% endfor %}
</ul>
</ol>
</li>
{% endfor %}
</ul>