Simple multi-cast, multi-channel communication foundation layer (primary) (!87317) · Merge requests · GitLab.org / GitLab

Thomas Randolph requested to merge tor/feature/observability-working-group/multi-channel-comms-harness/low-control-robust into master May 11, 2022

What does this MR do and why?

For #358899 (closed)

What

This is an implementation for a multi-cast, multi-channel communication harness.
This feature uses RxJS Subjects as the multi-cast implementation.

This feature is the initial work for creating a generic communication harness that many disparate apps (in the GitLab codebase) can use to communicate information that is not necessarily specific to their app. For example, performance, error, or telemetry data.

The overall architecture is intended to eventually encompass this demo data flow.
You can see more about this proposal by watching this 12-minute overview (and discussion!) about the proposal.
This MR implements this initial section of the data flow, which provides the base mechanism for channel-based communication:

Utility

While the contents of this MR aren't necessarily intended to be used "as-is" (see the roadmap below for further abstractions), it is effective (and most powerful) in this most raw state.

The best use case for a tool like this is for when applications want to broadcast that they are doing something (or requesting something, or updating something, etc.), but do not - or should not - have an explicit link to other sections of the application. Likewise, when parts of the application want to respond to non-explicitly occurring events without a direct link to what caused that thing to happen, they should use a utility like this.

This utility underpins asynchronous, non-deterministic, decoupled reactive application architectures.
While that may not be the future of the entirety of the GitLab product, it is certainly the intended behavior of the observability reporting harness.

The following API is exposed by these additions:

API	Arguments	Return	Use
`openChannel`	`options` - Object - Required `options.name` - String - Required	RxJS Subject	Create a new channel, or return the existing channel for a given name.
`flushChannel`	`options` - Object - Required `options.name` - String - Required	`undefined` (`void`)	Remove references to an existing channel by name

Examples

An application that would like to report its performance data could implement something like this in its Vue app:

// application bootstrap
const performanceChannel = openChannel({ name: "performance::our-group::our-team::our-feature" });
const commsChannel = openChannel({ name: "our-group::our-team::our-feature::stateless-communication" });
const channels = {
    performance: performanceChannel,
    comms: commsChannel,
};

comms.subscribe( ( event ) => {
    if( event.name === 'our-component::hello!' ){
        alert( `This page is ready to go. <br /> Actual message: ${event.data.message}` );
    }
} );

ourApp.provide( 'channels', channels );

// Deeper, in a Vue component
{
    ...
    inject: ['channels'],
    created(){
        this.channels.performance.next({ name: "our-component::started" });
    },
    async mounted(){
        await $nextTick();

        this.channels.performance.next({ name: 'our-component::ready' });
        this.channels.comms.next({ name: 'our-component::hello!', data: { message: 'Hello there! This component is started and ready to take your requests!' } });
    }
}

This is perfectly valid, but as of this MR, nothing will happen beyond the manually triggered alert. In later MRs, we'll add default behavior (and expected payload structures) for common channels like 'performance' or 'telemetry' or 'errors'. We'll also add abstractions, since some folks are opposed to the .next and .subscribe terminology of Observable streams.

However, given only this implementation, apps (and components, and discrete pieces of logic, etc.) are enabled to have non-directional, multi-cast messaging which effectively hands each part (of any size) of the app a free API for other teams - or other apps - to consume.

Roadmap

Step	Issue	MR(s)	Description
1	Multi-channel communication harness (#358899 - closed)	We're here!	Implement the basic message flow for the communication layer to rely on.
2	Multi-destination adapter system for communicat... (#358900 - closed)		Implement the adapter system so that arbitrary data can be sent to arbitrary endpoints. The Performance Aggregator example is the proposed model, where adapters are used to segment data into more and more specific handlers, and finally to send the same uniform source data to the appropriate services
3	Pipeline Authoring generic messaging harness im... (#358903 - closed)		Integration with the grouppipeline authoring team to confirm viability and smooth out edges
4	[spike] Create Generic Communication Harness (#344772 - closed)		RFC / Beta release for public consumption

How

While talking deeply about the specific implementation here isn't really necessary, I do think some folks might have questions about "Why RxJS?"

I'll try to note what I think are the biggest benefits of using a reactive library like this.

This targets proposed language specifications.
Ultimately the goal should be to remove the library in favor of the platform-provided tools, where possible.
Reactive programming separates application events from the code that responds to them, decoupling unassociated operations.
While the line can be blurry here, there are some clear places where each and every application likely shouldn't need to concern themselves with what happens with performance or telemetry information once the user has "done" that behavior. The communication harness takes care of those things.
Existing tooling relies on our UI framework and/or syntax unlikely to be standardized and lacks the extremely high complexity ceiling.
We should be able to swap out tooling for specific use-cases - like inter-app communication - without needing to also swap out the framework we use to render the UI!
This includes swapping the implementation out for platform-native options, which is why targeting syntax more likely to be standardized is important.
While we can get started simply with both existing tooling and RxJS, only RxJS offers tooling to allow for dramatically more complex solutions later as we grow to need more significant implementations of data streams.
RxJS has an enormous amount of extensibility and tooling.
It has its own testing utilities to make asynchronous testing more straight-forward.
RxJS also has a huge amount of optional operators that will let us scale up the complexity of our use-case without introducing painful migrations or insidious technical debt.

Screenshots or screen recordings

N/A; this is not currently used by the UI.

How to set up and validate locally

Run the test suite 😉

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

I have evaluated the MR acceptance checklist for this MR.

Extras

Fallback option - use in case of "emergency"; it is sub-par in basically every way.

Edited May 17, 2022 by Thomas Randolph

Simple multi-cast, multi-channel communication foundation layer (primary)