Inline $ref's prior to processing the OpenAPI documents

Problem

The OpenAPI library used by API Security Testing doesn't transparently handle $ref inlining causing customer support requests as we find gaps in our support.

Proposal

One way to handle $ref's is to inline them prior to processing the OpenAPI document. This Spike will explore using a nodejs local (on the container) service communicating over websockets with API Security Testing.

This would be a quick short term fix for the ongoing $ref problems if it works out. That would give us some breathing room to explore better long term fixes.

One potential limitation is recursive schemas. Recursive references cannot be inlined without changing the intent of the schema (limiting the depth to an arbitrary number).

Implementation plan

• Create a new IFileLoader implementation
  • Ask inlining service to inline a json document by providing the contents of the file to the service
  • The inlining service can ask IFileLoader to get a URL. This will allow the use of overrides in resolving references.
  • Returns the inlined document which is returned from IFileLoader
  • Websocket client
• Create a new nodejs inline service
  • Uses the @apidevtools/json-schema-ref-parser npm module to perform inlining
  • Supports custom resolvers that can request files from IFileLoader via websockets
  • WebSocket server

sequenceDiagram
    participant W as Worker Entry
    participant S as Scan Engine
    participant I as Inline Service
    participant T as Target App

    W->>S: Validate
    S->>+I: Inline $refs
    I->>+S: Get file/url
    S->>T: Get resource w/auth
    T->>S: Contents
    S->>-I: Contents
    I->>-S: Inlined Contents
    S->>S: Validate
    S->>W: Validation Result

Websocket messages

inline_refs

Request:

{
  "event": "inline_refs",
  "data": {
    "id": "request identifier, uuid, filename, etc.",
    "content": "file contents as UTF8 string"
  }
}

Response:

{
  "event": "inline_refs_response",
  "data": {
    "id": "request identifier",
    "content": "inlined contents as UTF8 string"
  }
}

get_url

Request:

{
  "event": "get_url",
  "data": {
    "url": "file://...|http://...|https://..."
  }
}

Response:

{
  "event": "get_url_response",
  "data": {
    "url": "url from request",
    "content": "file contents as UTF8 string"
  }
}