Support content negotiation for semantic web and media types

Manish Kumarrequested to merge
gitlab-community/gitlab-org/gitlab-pages:1135-content-negotiation into master

What does this MR do and why?

Closes #1135

This MR introduces HTTP content negotiation (Accept header parsing and MIME-based file resolution) into GitLab Pages, enabling semantic web and linked data publishing (such as FOAF, RDF, Turtle, JSON-LD, N-Triples, N3, OWL) alongside regular web pages.

Background & Problem

Previously, when a client requested an extensionless URL (e.g. GET /alice or GET /) without a file extension, GitLab Pages only resolved to .html (or index.html). Clients requesting semantic representations using headers like Accept: text/turtle or Accept: application/rdf+xml would either receive HTML or a 404, preventing static sites hosted on GitLab Pages from serving linked data and semantic web content using standard URI patterns.

Solution

  1. RFC 9110 Compliant Accept Header Parser:

    • Parses Accept header quality values (q=0.0 - 1.0) and specificity weights (exact type/subtype > type/* > */*).
    • Discards zero-quality entries (q=0).

  2. MIME and Extension Resolution:

    • Maps requested media types to corresponding extensions:
      • application/rdf+xml -> .rdf
      • text/turtle -> .ttl
      • text/n3 -> .n3
      • application/n-triples -> .nt
      • application/ld+json -> .jsonld
      • application/owl+xml -> .owl
      • text/html, application/xhtml+xml -> .html
    • Dynamically resolves through candidate file extensions based on client preference.
    • Preserves backward compatibility: falls back to .html if no preferred semantic format exists on disk.

  3. HTTP Caching:

    • Sets Vary: Accept header on responses that underwent content negotiation so intermediate caches and CDNs cache variants accurately.

  4. Testing:

    • Unit tests for RFC 9110 Accept header parser (TestParseAcceptHeader).
    • Integration tests in TestReader_ContentNegotiation covering:
      • Directory index content negotiation (/ with Accept: application/rdf+xml -> index.rdf)
      • File content negotiation (/page with Accept: application/rdf+xml -> page.rdf, text/turtle -> page.ttl, application/ld+json -> page.jsonld)
      • Quality weighted selection (text/turtle;q=0.9, application/rdf+xml;q=0.8)
      • Fallback to .html when requested format does not exist
      • Default behavior without Accept header
      • Direct extension requests bypass negotiation and do not set Vary: Accept

Merge request reports