Skip to content

Provide new API to abstract the subscription update business logic

Background

Adblock Plus and similar browser extensions support filter list subscriptions. These curated filter lists mean that a user can have the benefit of ad blocking without having to manually write and maintain their own filters. Different filter lists target different things, e.g. adverts on German websites, or privacy-invading trackers. Filter lists are also automatically refreshed by Adblock Plus, so that the subscribing user will automatically receive updated filters periodically.

As well as being convenient for the user, automatic filter list subscription updates serve an important business function for eyeo - usage metrics. When a subscription is updated, a web request is made to fetch the latest version of the filter list. From those web requests usage of both the individual filter lists and the ad blocking extensions themselves can be derived. It's therefore very important that the extensions update filter list subscriptions in a consistent way, at the correct times and with the correct query parameters. Otherwise these important metrics could be compromised.

The business logic for exactly when and how to request filter list subscription updates is quite complex. It seems to be defined in lib/downloader.js, lib/synchronizer.js and lib/subscriptionClasses.js. The code seems to be somewhat intertwined and to have been designed with some assumptions which are no longer true with manifest v3.

With manifest v3 some of the browser APIs work differently from before, for example we cannot rely on window.setTimout in the same way. We also must take care to persist any state of the filter subscriptions in storage whenever it is changed. We also don't use all the same custom datatypes that adblockpluscore provides. We therefore need a new API in adblockpluscore which is sufficiently abstracted away from such things.

What to change

  1. Provide a new lib/subscriptionStateUpdate (or similar) API which can be used from both the existing lib/synchronizer.js code and from the manifest v3 extension.

The API should at it's heart provide a function or functions which take the state of a subscription and return its updated state, along with any required actions. It should be left to the consumer of the API to call the API regularly, to persist any updated state and to perform any actions (e.g. perform a download). The subscription's state should be passed each way using standard, JSON serializable JavaScript data structures, e.g. Objects and Arrays.

For example, I imagine the API to work something like this:

import {updateState} from "adblockpluscore/lib/subscriptionStateUpdate";

...

let performDownload;
let subscriptionState = {
  id: "anti-circumvention",
  url: "https://easylist-downloads.adblockplus.org/abp-filters-anti-cv.txt",
  lastCheck: ...,
  lastVersion: ...,
  downloadCount: ...
  ...
}

([performDownload, subscriptionState] = updateState(Date.now(), subscriptionState));

if (performDownload)
{
  // Attempt to download the subscription using the Fetch API.
  // On success parse the subscription's headers and return them as an
  // Object.
  // On failure return the exception.
  let fetchResult = fetch(...).then(...);

  subscriptionState = updateState(Date.now(), subscriptionState, fetchResult)[1];
}

// At this point persist the subscription's updated state to chrome.storage.local...
  • I think that any unmodified fields in the state Object should be left alone, and simply returned back to the caller unmodified.
  • Perhaps the API should provide one function which can be optionally passed the results of an update. Alternatively the API could be split into two functions.
  • For the update result, the API should accept a Promise which resolves to the subscription's parsed header (as an Object). To indicate failure I don't mind if the API simply expects the Fetch exception to be thrown, or some abstracted format for the error.
  • The API should do as much work as possible to encapsulate the business logic of subscription updates. So for example, it should handle redirections and take care to provide (where possible) sensible defaults for fields like downloadCount.
  • The API should not perform any side-effects, or rely on any external state stored in memory.
  • The API should - as far as possible - avoid using other adblockpluscore code. The API should also avoid expecting custom adblockpluscore data types are being used. For example, the manifest v3 extension might not be using the Subscription Object at all.
  • I don't mind if the API updates the URL to include the required GET parameters directly, or if it instead provides those as an Object for the consumer to append. Either way, the API must take care of figuring out exactly what GET parameters should be sent.
  • It's important the state for a subscription is always JSON serializable as with the manifest v3 extension we will need to write it to storage every time it is changed.
  1. Update the lib/synchronizer.js code to use this new API.
  2. Update the existing tests to also test this new API directly, or perhaps write some new tests for the new API.
Edited by Dave Vandyke