Playing with Pipelines

This post walks through one of the Substrates showcase examples hosted on GitHub, demonstrating two of the most critical aspects of observability pipelines – the production and consumption of captured measurement data and published events.

Modeling the Game

The code introduced in multiple parts below is based on the Game of Ping Pong, where a Ball is passed back and forth between two Players. In the Game, a Player is both a producer and consumer of an Event, the Event being the serving of the Ball from one Player who previously received the Ball from the other Player.

In the code, a Player is modeled as an Instrument that emits Events where the emission property of an Event is a Ball. The Player interface acts as a protocol decorator around an Inlet provided by the Substrates framework. Typically, such interface methods would capture, transform, and validate data before passing it onto the Inlet and the pipeline.

A Game is modeled as a Subscriber. Whenever a Player is created, the accept method of the Game interface is called, and an Outlet is registered. The Outlet created within the bind method, shown later, will call serve on the Player for the Ball received. So whenever the Subject is pong, the Outlet forwards the Ball onto ping. When the Subject is ping, the Outlet forwards onto pong. A Game here acts as an observer and a controller, wiring up pipelined events with actions.

Within the bind method of the Game class, the Game uses the registrar of a Player to register (link) an Outlet that extracts the emission from an Event and forwards it to another Player via the serve method. The Outlet is decorated with a filter that forwards the Ball as long as the Game is not over, which is determined by the serves property.

The count of serves could have been tracked in the Ball or Game types, but instead, the Event sequence property is used.

One last method, start, needs to be added to the Game type that starts the Ball rolling by serving to ping first.

That is it for the model in which Ball was mapped to Event emission, Player to Instrument, and Game to Subscriber.

Wiring the Circuitry

The Cortex interface is the surface of the pipeline fabric from where all circuitry and referent types are assembled. In the code below, the lifetime of the Circuit (executor) created by the Cortex is scoped to a try-finally block.

With a Circuit in hand, a Conduit (channel) can now be created for the Player Instruments to transfer Event emissions between Inlets (producers) and Outlets (consumers). Creating a Conduit requires a Cartridge, an interface that is an abstraction for interfacing with Instrument factories (suppliers). A Cartridge is created from the Player constructor.

Calling the instrument method on the Conduit, the two Player instances required for the Game are automatically created.

Before starting the Game, it must subscribe to the Conduit‘s Source component to receive and react to Events.

Running the Game

Finally, the Game can be started. But to prevent the main Thread from exiting the try block and closing the Circuit Resource immediately, the Current component is used to await the delivery of all Ball serve Events emitted by Players.

Wrapping it up

The showcase example above covers several key concepts within the Substrates API. However, unless one is developing a new Instrument, as in the Player above, knowledge of the Circuit, Conduit, and Cartridge interfaces is sufficient for most purposes.

In a future post, the ability and benefits of pipelining multiple instances of Circuit and Conduit will be explored.