Observability X – Sources

Observing Observability

Observability has been reduced to the straightforward processing of data collected and its forwarding to a remote centralized endpoint. While frameworks such as OpenTelemetry have standardized this, they overlook a fundamental aspect of true observability: the dynamic nature of observation itself.

A more effective and efficient observability approach establishes live, adaptable connections between information sources and observers, both locally and remotely, as well as online and offline.

A shift from passive data collection to active observation streams fosters a more responsive and insightful system. It allows observers to dynamically adjust their perspective based on the observed data and enables sources to evolve in their information disclosure strategies.

The cornerstone of this sophisticated approach lies in comprehending four fundamental concepts: sources, which distribute data about subjects; subjects, which represent entities or aspects of the system being observed; subscribers, who have the autonomy to select observation methods; and the dynamic nature of subscription itself, which facilitates the formation and adaptation of observation streams.

Sourcing Subscriptions

As discussed in the previous post, a Source is obtained through a Context. In the toolkit, one component that implements Context is a Conduit. In the code below, a Conduit is created with a Composer that returns the Pipe of a Channel. Typically, a Composer constructs an interface around the Pipe; however, below, the Pipe of the named Channel is returned to the instrumentation client as the instrument interface.

var cortex = cortex ();

var input = cortex.name ( "input" );

var output = cortex.name ( "output" );

var circuit = cortex.circuit ();

var conduit = circuit.conduit ( Channel::pipe );

var source = conduit.source ();

var pipe = conduit.compose ( input );

The Source interface allows an observer to subscribe to one or more named Channels. When a new Channel is created for a Subject within the context of the Source each Subscriber will be called to allow registration of one or more pipes with a subject-specific Registrar. Here is the Subscriber interface

interface Subscriber < E >

extends Resource {

void accept (

Subject subject,

Registrar < E > registrar

);

}

Below we demonstrate the conditional nature of the registration of a Pipe with a Subject and in turn Channel. If the Subject in the callback has the name input then we forward its emissions to an output named Pipe, otherwise, we return a Pipe that prints each emitted value. Finally, we emit a “hello” value.

var subscription =

source.subscribe (

( subject, registrar ) ->

registrar.register (

( subject.name () == input )

? conduit.compose ( output )

: System.out::println

)

);

pipe.emit ( "hello" );

The above prints “hello” after sending the value to the input, output, and console pipes. We can observe the pipe value transfer by registering another pipe that captures all subjects and their emitted values.

source.consume (

capture ->

System.out.printf (

"%s -> %s%n",

capture

.subject ()

.name (),

capture

.emission ()

)

);

The following is printed, with the last line from our initial subscription. Observability that is observable!

in -> hello

out -> hello

hello

We still have much more to cover in the coming posts in the series. Still, hopefully, you are starting to see the simplicity and sophistication of the Substrates API and its possibility as a new toolkit for synthesizing novel instruments, intelligent observers, and eventually digital twins. Next up will be a deep dive into Pipes.