Commit ef1eb197 authored by Sebastiaan Deckers's avatar Sebastiaan Deckers 🐑

feat: custom response headers

deprecate: cache digest (experimental cookie fallback)
deprecate: hsts/immutable/serviceworker header overrides

The custom headers mechanism is more flexible and supercedes the specific HTTP header support.

Cache digest support was limited to a Service Worker implementation. Better to just implement the server-side "diary" approach in future patch. Hopefully clients eventually implement it natively.
parent 6f32c747
# @commonshost/server
Static HTTP/2 webserver for single page apps and progressive web apps.
Static HTTP/2 web server for single page apps and progressive web apps.
- HTTP/2, HTTP/1.1, and HTTP/1.0
- Content encoding for Brotli and Gzip compression
- Server Push Manifests & Cache Digest
- Scalable, multi-processor clustering
- Auto-generate HTTPS certificate for localhost development
## Key Features
- HTTP/2 with fallback to HTTP/1.x
- HTTP/2 Server Push Manifests
- Brotli and Gzip compression negotiation
- Multi-core clustering
- Multiple hosts with different domains
- Fallback for client side routing
- HTTP to HTTPS redirect
- CORS
- Immutable caching of revved files
- Custom response headers
- `http:` to `https:` URL redirect
- Auto-generate TLS certificate for localhost development
# Host Options
Example:
```js
module.exports = {
hosts: [{
domain: 'example.net',
root: './build/site',
fallback: { 200: './app.html' },
directories: { trailingSlash: 'never' },
accessControl: { allowOrigin: '*' },
headers: [{ fields: { 'X-Frame-Options': 'deny' } }],
manifest: [{ get: '/app.html', push: '/app.css' }]
}]
}
```
## `hosts[].domain`
Default: `''` or `'localhost'` (only for the default site, see [serveDefaultSite](#constructor))
The DNS hostname of the site. May be specified as an Internationalized Domain Name (IDN) containing non-ASCII characters.
The DNS hostname of the site. May be specified as an [Internationalized Domain Name (IDN)](https://en.wikipedia.org/wiki/Internationalized_domain_name) containing non-ASCII characters.
Multiple sites on the same server can each have their own configuration.
Examples:
```js
{
module.exports = {
hosts: [
{ domain: 'example.net' },
{ domain: 'xn--yfro4i67o.example.net' },
......@@ -22,6 +40,15 @@ Examples:
The path to the base directory containing static files to serve.
```js
module.exports = {
hosts: [{
domain: 'example.net',
root: './website'
}]
}
```
If no root is specified, the server tries to auto-detect static site generator or packaging tool output directories. For example: `./dist`, `./public`, `./_site`, and many more.
If no directory is auto-detected, the current working directory is used. A warning message is logged to indicate this fallback behaviour.
......@@ -32,103 +59,112 @@ Default: `{}` (no fallback)
An object mapping HTTP status code to file paths.
Typically used for client side routing.
Fallbacks are supported for:
- `200` to serve a missing file as a success response. Typically used for client side routing.
- `404` to serve a missing file as an error response. Typically used for client side routing.
- `200` to serve a missing file as a success response.
- `404` to serve a missing file as an error response.
Example:
```js
{
fallback: {
200: './index.html'
}
}
```
## `hosts[].cacheControl`
Sets the HTTP caching headers.
### `hosts[].cacheControl.immutable`
Default: `[]`
An array of path [globs](https://www.npmjs.com/package/micromatch) for immutable files.
Special named patterns can be used as shorthand. These are:
- `hex` — Matches hexadecimal hash revved files. Example: `layout-d41d8cd98f.css`
- `emoji` — Matches emoji revved files. Example: `app.⚽️.js`
By default, all non-immutable responses have the header:
```
cache-control: public, max-age=1, must-revalidate
```
File paths that match the patterns set by the `cacheControl.immutable` option are considered to *never, ever* change their contents. To tell browsers never to revalidate these resources, they are served with the header:
```
cache-control: public, max-age=31536000, immutable
```
Examples:
```js
{
cacheControl: {
immutable: [
'/library/v1.2.3/**/*.js'
]
}
module.exports = {
hosts: [{
domain: 'example.net',
fallback: {
200: './index.html'
}
}]
}
```
## `hosts[].directories`
Behaviour of directory paths.
Behaviour of directory paths in *clean URLs*.
### `hosts[].directories.trailingSlash`
Default: `'always'`
Enforces a consistent *clean URL* for directory paths.
Enforces a consistent ending character for directory paths.
The value is a string that can be:
- `'always'` to use HTTP redirects to append a `/` at the end of the directory name.
- `'never'` to use HTTP redirects to strip any `/` at the end of the directory name.
- `'always'` to use an HTTP redirect to append a `/` at the end of the directory name.
- `'never'` to use an HTTP redirect to strip any `/` at the end of the directory name.
## `hosts[].accessControl`
Settings related to CORS.
Settings related to [Cross-Origin Resource Sharing (CORS)](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS). The server responds to `OPTIONS` method requests with the appropriate response headers to allow (or deny) third-party access.
### `hosts[].accessControl.allowOrigin`
Default: `'*'`
If specified, sets this value as the `access-control-allow-origin` header on every response.
If specified, sets this value as the `Access-Control-Allow-Origin` header on every response.
## `hosts[].serviceWorker`
## `hosts[].headers[]`
Sets custom HTTP response headers. The `headers` array contains header objects. Each header object has the `fields` and `uri` properties.
```js
module.exports = {
hosts: [{
domain: 'example.net',
headers: [
{
fields: {
'Content-Security-Policy': "default-src 'self'",
'Feature-Policy': "payment 'none'",
'Strict-Transport-Security': 'max-age=31536000',
'Referrer-Policy': 'strict-origin-when-cross-origin',
'X-Frame-Options': 'SAMEORIGIN',
'X-XSS-Protection': '1; mode=block',
'X-Content-Type-Options': 'nosniff'
}
},
{
uri: '/service-worker.js',
fields: {
'Service-Worker-Allowed': '/'
}
},
{
uri: '/scripts/{filename}.{hash}.js',
fields: {
'Cache-Control': 'public, max-age=31536000, immutable'
}
},
{
fields: {
'Set-Cookie': ['foo=bar', 'bar=baz']
}
}
]
}]
}
```
Settings related to service workers.
## `hosts[].headers[].fields`
### `hosts[].serviceWorker.allowed`
`fields` is a required object whose property names and values are the HTTP headers to set on the response.
Default: `'/'`
Property names must be valid HTTP header field names. Only case-insensitive alphanumeric characters (`A` through `Z`, and `0` through `9`) and hyphens (`-` dash) are allowed.
If specified, sets this value as the `service-worker-allowed` header on every response.
| Example | Validity |
|-|-|
| `Referrer-Policy` | Allowed |
| `:status` | Not Allowed |
## `hosts[].strictTransportSecurity`
Values may contain named template segments in braces. The values are substituted using matches from the `uri` template.
Settings related to HTTP Strict Transport Security.
Values may also be arrays of strings to set multiple headers with the same name.
### `hosts[].strictTransportSecurity.maxAge`
## `hosts[].headers[].uri`
Default: `5184000` (60 days)
`uri` is an optional string as a [URI Template](). The header fields are applied only if the request pathname matches the route.
Sets this value as the `strict-transport-security` header on every response.
If no `uri` property is set, the `fields` headers are applied to all responses.
## `hosts[].manifest`
......@@ -142,10 +178,9 @@ The `manifest` property type can be:
Example: Inline manifest
```js
{
hosts: [
domain: 'localhost',
root: './dist',
module.exports = {
hosts: [{
domain: 'example.net',
manifest: [
// Example:
{
......@@ -155,7 +190,7 @@ Example: Inline manifest
push: ['**/*.css', '**/*.js']
}
]
]
}]
}
```
......@@ -170,7 +205,7 @@ $ npx @commonshost/manifest generate ./dist ./manifest.json
```
```js
{
module.exports = {
hosts: [{
manifest: './manifest.json'
}]
......
......@@ -4,7 +4,6 @@ const { logger } = require('./middleware/logger')
const { playdoh } = require('playdoh')
const { hostOptions } = require('./middleware/hostOptions')
const { allowedMethods } = require('./middleware/allowedMethods')
const { serviceWorkerScope } = require('./middleware/serviceWorkerScope')
const { allowCors } = require('./middleware/allowCors')
const { cdnLoopPrevention } = require('./middleware/cdnLoopPrevention')
const { resolveRequest } = require('./middleware/resolveRequest')
......@@ -22,7 +21,6 @@ module.exports.app = (options, files) => {
}
app.use(hostOptions(options, files))
app.use(allowedMethods(['GET', 'HEAD', 'OPTIONS']))
app.use(serviceWorkerScope())
app.use(allowCors())
app.use(cdnLoopPrevention(options.via))
app.use(resolveRequest())
......
const mime = require('mime')
const compressible = require('compressible')
const { checkImmutable } = require('./checkImmutable')
const ONE_SECOND = 1
const ONE_YEAR = 31536e3
const UriTemplate = require('uri-templates')
const compressors = [
{ extension: '.br', encoding: 'br' },
......@@ -11,23 +8,36 @@ const compressors = [
{ extension: '.deflate', encoding: 'deflate' }
]
const variableRegex = /\{([a-zA-Z]+)\}/g
const headerCharRegex = /[^\t\x20-\x7e\x80-\xff]/
/**
* True if val contains an invalid field-vchar
* field-value = *( field-content / obs-fold )
* field-content = field-vchar [ 1*( SP / HTAB ) field-vchar ]
* field-vchar = VCHAR / obs-text
*/
function checkInvalidHeaderChar (val) {
return headerCharRegex.test(val)
}
module.exports.buildHeaders = function buildHeaders (
sourceFile,
immutablePatterns,
supportedEncodings,
fileIndex
fileIndex,
options
) {
let resolved = sourceFile
const headers = {}
const type = mime.getType(sourceFile.absolute)
headers['content-type'] = type.startsWith('text/')
? `${type}; charset=utf-8`
: type
if (type !== null) {
headers['content-type'] = type.startsWith('text/')
? `${type}; charset=utf-8`
: type
}
const isImmutable = checkImmutable(sourceFile.absolute, immutablePatterns)
headers['cache-control'] = isImmutable
? `public, max-age=${ONE_YEAR}, immutable`
: `public, max-age=${ONE_SECOND}, must-revalidate`
headers['cache-control'] = 'public, max-age=1, must-revalidate'
if (compressible(type)) {
for (const { extension, encoding } of compressors) {
......@@ -41,5 +51,32 @@ module.exports.buildHeaders = function buildHeaders (
}
}
}
if (options !== undefined) {
for (const { uri, fields } of options.headers) {
if (uri) {
const predicate = new UriTemplate(uri)
const parts = predicate.fromUri(sourceFile.pathname)
const replacer = (match, key) => parts[key]
if (parts !== undefined) {
for (const [name, value] of Object.entries(fields)) {
const substitution = Array.isArray(value)
? value.map((part) => value.replace(variableRegex, replacer))
: value.replace(variableRegex, replacer)
if (!checkInvalidHeaderChar(substitution)) {
headers[name] = substitution
}
}
}
} else {
for (const [name, value] of Object.entries(fields)) {
if (!checkInvalidHeaderChar(value)) {
headers[name] = value
}
}
}
}
}
return { headers, resolved }
}
const { queryDigestValue } = require('cache-digest')
const { getCacheDigest } = require('./getCacheDigest')
module.exports.cacheDigestFilter = (request, cookies, baseUrl) => {
const cacheDigest = getCacheDigest(request, cookies)
if (!cacheDigest) return () => true
const digestValue = Buffer.from(cacheDigest, 'base64')
return (pathname) => {
const url = baseUrl + pathname
const etag = undefined
const validators = false
try {
queryDigestValue(
digestValue,
url,
etag,
validators
)
return false
} catch (error) {
return true
}
}
}
const emojiRegex = require('emoji-regex')
const isEmojiRevved = new RegExp(
'.+' + // Base filename
'[-_.]' + // Separator
`(?:${emojiRegex().source})+` + // Emoji ranges
'(?:\\.\\w+)+$' // File extension(s)
)
const isHexRevved = new RegExp(
'.+' + // Base filename
'[-_.]' + // Separator
'[0-9a-fA-F]+' + // Hexadecimal hash
'(?:\\.\\w+)+$'// File extension(s)
)
const regexes = Object.create(null)
module.exports.checkImmutable = (filepath, patterns) => {
for (const pattern of patterns) {
if (!(pattern in regexes)) {
regexes[pattern] =
pattern === 'emoji' ? isEmojiRevved
: pattern === 'hex' ? isHexRevved
: new RegExp(pattern)
}
if (regexes[pattern].test(filepath)) {
return true
}
}
return false
}
function parseHeader (payload) {
const digests = payload.split(', ')
for (const digest of digests) {
const [value, ...flags] = digest.split('; ')
if (flags.includes('complete')) {
return value
}
}
}
module.exports.getCacheDigest = (request, cookies) => {
const cookie = cookies.get('cache-digest')
if (cookie) {
return cookie
}
const header = request.headers['cache-digest']
if (header) {
return parseHeader(header)
}
}
const { untilBefore } = require('until-before')
const { getHost } = require('../helpers/getHost')
const loopbackRequestCache = new WeakMap()
module.exports.isLoopback = (request) => {
if (loopbackRequestCache.has(request)) {
return loopbackRequestCache.get(request)
}
const host = untilBefore.call(getHost(request), ':')
const matchLoopback = host === 'localhost' ||
host === '127.0.0.1' ||
host === '::1'
loopbackRequestCache.set(request, matchLoopback)
return matchLoopback
}
const Cookies = require('cookies')
const { getDependencies } = require('../helpers/getDependencies')
const { cacheDigestFilter } = require('../helpers/cacheDigestFilter')
const { checkImmutable } = require('../helpers/checkImmutable')
const { getOrigin } = require('../helpers/getOrigin')
const { requestDestination } = require('request-destination')
const { computeDigestValue } = require('cache-digest')
const HTTP2_PRIORITY_DEFAULT = 16
......@@ -16,8 +12,6 @@ async function resolveDependencies (request, response, next) {
}
const baseUrl = getOrigin(request)
const cookies = new Cookies(request, response)
const pushedImmutables = []
request.pushResponses = []
const dependencies = getDependencies(
......@@ -26,8 +20,6 @@ async function resolveDependencies (request, response, next) {
request.resolved.relative
)
const cacheDigestContains = cacheDigestFilter(request, cookies, baseUrl)
if (request.httpVersionMajor === 2 && response.stream.pushAllowed) {
const headers = {
':method': request.method,
......@@ -47,49 +39,23 @@ async function resolveDependencies (request, response, next) {
const pushPromises = []
for (const [relative, priority] of dependencies) {
const dependency = request.fileIndex.relative.get(relative)
if (!cacheDigestContains(dependency.pathname)) continue
headers[':path'] = dependency.pathname
response.log.info(`PUSH ${baseUrl}${dependency.pathname}`)
const pushPromise = pushStream(headers, priority)
.then((pushResponse) => ({ pushResponse, dependency }))
pushPromises.push(pushPromise)
if (checkImmutable(
relative,
request.options.cacheControl.immutable
)) {
dependency.isImmutable = true
if (request.method === 'GET') {
pushedImmutables.push(dependency.pathname)
}
}
}
request.pushResponses = await Promise.all(pushPromises)
} else if (request.httpVersionMajor === 1) {
const links = []
for (const [relative] of dependencies) {
const { pathname } = request.fileIndex.relative.get(relative)
if (!cacheDigestContains(pathname)) continue
const destination = requestDestination(relative)
links.push(`<${pathname}>; rel=preload; as=${destination}`)
}
response.setHeader('link', links)
}
if (request.method === 'GET') {
if (request.headers['cache-digest']) {
if (cookies.get('cache-digest')) {
cookies.set('cache-digest')
}
}
}
if (pushedImmutables.length) {
const urls = pushedImmutables.map((pathname) => [baseUrl + pathname, null])
const digestValue = computeDigestValue(false, urls, 2 ** 7)
const cookie = Buffer.from(digestValue).toString('base64').replace(/=+$/, '')
cookies.set('cache-digest', cookie)
}
next()
}
......
......@@ -26,7 +26,6 @@ async function serveDependencies (request, response) {
response.stream.session.localSettings.maxConcurrentStreams
) - 1)
const immutablePatterns = request.options.cacheControl.immutable
const supportedEncodings = new Negotiator(request).encodings()
const fileIndex = request.fileIndex
const isHeadResponse = request.method === 'HEAD'
......@@ -35,9 +34,9 @@ async function serveDependencies (request, response) {
pushQueue.defer((callback) => {
const { headers, resolved } = buildHeaders(
dependency,
immutablePatterns,
supportedEncodings,
fileIndex
fileIndex,
request.options
)
pushResponse.respondWithFile(
resolved.absolute,
......
......@@ -2,7 +2,6 @@ const { serveResponse } = require('./serveResponse')
const accepts = require('accepts')
const parseUrl = require('parseurl')
const { extname } = require('path')
const { createReadStream } = require('fs')
module.exports.serveFallback = (options) => {
const passthrough = serveResponse(options)
......@@ -16,12 +15,12 @@ module.exports.serveFallback = (options) => {
(extension === '' || extension === '.html')
) {
response.statusCode = 404
response.setHeader('content-type', 'text/html; charset=utf-8')
const file = createReadStream(options.placeholder.hostNotFound)
file.once('error', (error) => {
next(error)
})
file.pipe(response)
request.resolved = {
absolute: options.placeholder.hostNotFound,
pathname: encodeURI(url.pathname)
}
request.fileIndex = { absolute: new Map() }
passthrough(request, response, next)
return
}
}
......
......@@ -2,7 +2,6 @@ const Negotiator = require('negotiator')
const { createReadStream, stat } = require('fs')
const { promisify } = require('util')
const { buildHeaders } = require('../helpers/buildHeaders')
const { isLoopback } = require('../helpers/isLoopback')
const { serveDependencies } = require('./serveDependencies')
const { NotFound, InternalServerError } = require('http-errors')
......@@ -10,16 +9,11 @@ module.exports.serveResponse = () => {
return async function serveResponse (request, response, next) {
const { resolved, headers } = buildHeaders(
request.resolved,
request.options.cacheControl.immutable,
new Negotiator(request).encodings(),
request.fileIndex
request.fileIndex,
request.options
)
if (!isLoopback(request)) {
headers['strict-transport-security'] =
`max-age=${request.options.strictTransportSecurity.maxAge}`
}
const statusCode = response.statusCode || 200
if (request.httpVersionMajor === 2) {
......@@ -87,13 +81,9 @@ module.exports.serveResponse = () => {
file.pipe(response)
})
file.once('error', (error) => {
file.removeAllListeners()
if (!response.headersSent) next(error)
else if (!response.finished) response.end()
})
file.once('end', () => {
file.removeAllListeners()
})
}
}
}
......
module.exports.serviceWorkerScope = () => {
return function serviceWorkerScope (request, response, next) {
const scope = request.options.serviceWorker.allowed
if (scope && request.headers['service-worker'] === 'script') {
response.setHeader('service-worker-allowed', scope)
}
next()
}
}
const test = require('blue-tape')
const { cacheDigestFilter } = require('../src/helpers/cacheDigestFilter')
const fixtures = [
{
scenario: 'No digest present',
request: { headers: {} },
cookies: { get: () => undefined },
baseUrl: 'https://example.com',
given: [
'/style.css'
],
expected: [
'/style.css'
]
},
{
scenario: 'Digest present in header',
request: { headers: { 'cache-digest': 'AfdA; complete' } },
cookies: { get: () => undefined },
baseUrl: 'https://example.com',