P

planning

Wiki-style planning project

(This is an idea that needs one or more new app developers that are able to combine the existing parts into a consolidating project. Follow "Groups->FOSS-push->Request Membership" above, or try IRC #foss-push on freenode.)

FOSS push notifications (independent from google servers)

Instead of requiring every FOSS app to implement its own push notification solution, the idea is to work on a shared push implementation that provides an alternative library that app developers can use to subscribe for and send push messages.

Having only one watchdog/proxy process running on a system minimizes battery usage and allows to firewall direct app access to the network. And it provides a central place to work out and solve the problems posed by some mobile OSes.

Vocabulary

FOSS

Free and open source software.

App Client

The app client as used by the end-user, usually an Android or iOS app. Runs on the end-user's device. For example the Nextcloud Talk, Telegram, or Deltachat mobile applications.

App Server (optional for peer to peer)

A server that provides the app client with data. Runs on a server. For example a Nextcloud or Telegram server.

Push Server

The push server is a middleman between an app server and a push client. It allows the app server to push messages to push clients that are listening to the push server. Runs on a server. With the legacy protocol (see below) this part of the architecture can not be implemented in app servers that are based on web servers, and it was therefore often implemented externally and centralized by a phone/os company. This part of the architecture is missing, but can easily be substituted with current email servers, as most implementations support the push-IMAP extension by now.

Push Client

The push client listens for messages from the push server (network communication) and forwards them to the app client (inter-process communication). Runs on the end-user's device. The inter-process communication part of the architecture is missing.

Legacy Push Data Flow

The legacy data flow is as follows, assuming all clients and servers are available:

  1. The app client registers with the push client. The push client forwards the registration to the push server.
  2. The push client returns a token that the app client registers with the app server.
  3. To send a message to the app client, the app server sends the message to the push server (including the token), the push server delivers it to the push client, and the push client delivers it to the app client.
  4. Optionally: The app client de-registers from the push client and the app server (and subsequently the push client from the push server).

Limitations of the legacy push protocol

Legacy push messaging depends on third-party servers, traffic and IP registrations.

The legacy push messaging protocol is a third-party implementation that does not support own push servers. Thus, even if an app server implementation (non-web-app) would be capable to provide push server protocol capabilities themselves, one would still have to use the central third party serves.

An own (free) implementation of a legacy push client and server would still require to host a separate own legacy push protocol server in many cases. For example, the person that hosts a web server with Nextcloud, would also have to host a separate push server. And the end-user must enter the details of the push server(s) into the push client.

FOSS push protocol alternatives

A free push client implentation allows to support different protocols:

The push-IMAP protocol extension allows to use existing email infrastructure as push servers (own or provider based), making the administration of additional push servers optional. Sending (stripped down) email messages also removes the need to implement yet another message sending function in the app server. And ther exists a push-IMAP protocol client library that separates push chat messages from the regular email communication and supports the autocrypt standard (see below). The missing parts to build a shared push client are mostly inter-process communication and further separation of the new Email-notification messages from the Email-chat messages.

HTML 5 Server Side Events (SSE) extension A standard subscription protocol of the W3C. Allows to use or extend existing webserver software as push server. https://en.wikipedia.org/wiki/Server-sent_events https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events

For a multi-account capable implementations of SSE see

Other HTTP push protocol variants (https://en.wikipedia.org/wiki/Push_technology) may also allow app servers to directly implement push server capabilities, remove the need for third-party servers, and allow ad-hoc operation, but would require maintaining additional server capabilities.

gotify.net A websocket server programmed with google's Go for sending and receiving messages. Would require separate server installations.

dhtproxy servers in the peer-to-peer (P2P) OpenDHT network backbone The OpenDHT distributed hash table implementation (facilitating the GNU Jami (former Ring) text/file/audio/video messenger) supports dhtproxy nodes. Mobile clients can connect to these, subscribe for, and receive messages addressed for the mobile device. https://github.com/savoirfairelinux/opendht

https://prototypefund.de/project/open-source-push-service-fuer-android-apps/

Missing parts to implement

The most universal and straight forward way seems to have common push client code that can be build for multiple plattforms, and to start implementing it by using existing code in the push client.

For example by splitting the existing deltachat codebase into three packages, a platform independent lib, a platform specific push notification client, and the user (chat) frontend, as outlined here.

On android the FOSS push protocol lib may also be introduced into the MicroG (free GCM implementation) push client code. Other operating systems have different notification frameworks.

push client service (daemon app)

On android, this part should create a permanent notification to stay in foreground ("shown" only while screen is off, see deltachat), and it provides

  • inter-process communication with client apps (android content-provider)
  • network communication protocol support for:
    • autocrypt enabled, not-email-disturbing IMAP-push messaging (i.e. https://github.com/deltachat/deltachat-core lib, to be extended with a a new "Email-notify spec" to filter compliant messages)
    • SSE
    • XMPP ?
    • dhtproxy
    • ...other push server protocols?

app client support:

  • client app library providing methods to watch (subscribe, DB observe) and access the content-providers of the push client service (inter-process communication with the push client service deamon)

app server / message sending support

push server

Protocol selection

The push client should be able support multiple protocols. Yet, having some common protocol options could make sense to allow interoperability. Some important features for a FOSS solution would be federation, availability, and wide-spread usage (existing users, and bullet proof implementations). Mature contenders with push extensions are IMAP, XMPP, ...?

  1. IMAP is an open and wide-spread "on-the-wire" protocol for message store-and-forward delivery.

The most prominent advantages of supporting IMAP push is that the email server infrastructure is already there, and that it does not need to host and administer any further permanently running servers. Many users already have at least one IMAP email server that they trust, and they are free to change servers/providers anytime. As a federated system, it does not enforce any particular organizational structure, while already being very accessible from virtually every programming language and platform. App developers would not have to maintain own servers. Projects may still setup own central IMAP push servers, but they would not have to.

The other advantage is that there is already an implementation available with most of what is needed for IMAP push notifications: The https://github.com/deltachat/deltachat-core lib handles general, autocrypted, and spam exempt, Email-chat spec compliant messages by listening on a connection to an arbitrary, regular email account (with IMAP push support), while not disturbing the regular email communication of the account with the higher frequency chat messaging. It would seem desirable to not just use the authenticated email chat messages for the notifications, but to have a slightly adapted "Email-notify spec" that further separates the notifications from the general chat messages, and reduces the data overhead after the initial hand-shake. As the deltachat experience showed, there were quite a lot of things to solve to get the messaging to work right, so the idea is to refactor deltachat-core into a library and a service, upstreaming the necessary extensions, and making the the service application support multiple client apps and push protocols https://github.com/deltachat/deltachat-core/issues/113#issuecomment-368319972 (i.e. support the deltachat client apps as well as other apps subscribing to their specific notifications).

  1. SSE This is the most straight forward HTML extension, but requires to set up a server to run permanently, and make it use a SSE message queueing library. (For a comparison of SSE, websockets, and webpush see: https://www.fastly.com/blog/server-sent-events-fastly) Even though it is fairly common for projects to run or rent their own webserver, they are not as widely available as email accounts.

  2. Other protocols listed at https://stackoverflow.com/questions/20329403/android-push-notification-without-using-gcm would require mobile and web app writers to support a new protocol, maintain their own separate servers or use central servers. Instead of just sending a (https://autocrypt.org compliant) email message to a user's account.

Guidelines

To ease transition or parallel publishing, the android client library could be kept similar to the gcm lib.

Prior work