The Components of a Circuit
The Humainary initiative is progressing rapidly along a path towards delivering the ultimate observability toolkit in the form of a network circuit consisting of a diverse set of instruments, contexts, subscribers, and sinks. We distinguish between an observability circuit and data pipeline based on the degree of intelligence and locality of processing emittances (signal or value).
For the most part, the data pipelines touted by the vendor contingent within the observability community are mindless movers of measurements from a source point to a back-end endpoint in the cloud – other than the batching, compressing, and dropping of data payloads, there is not much to talk about here as vendors do most work after the costly and wasteful transmission.
Whereas all the Humainary instrument libraries, built on top of the Substrates foundational library, are designed to publish, consume, and transform emittances effortlessly across different collocated integrated points of contextualization, communication, coordination, and consumption. The diagram below presents the components of the circuitry, including communication flows.
The Flow of Signals
An Instrument, such as a Counter, will publish emittances to the Context it was created by when methods within its interface are called. For a Counter, this would be inc. The Context, acting as a Conduit and Source of emittances, will first coordinate with one or more registered Subscriber instances for interest in the emitting Instrument, if it has not been done so previously for the particular Named Instrument, after which it will forward the emittance onto one or more registered Sink instances.
Typically a Sink is registered via a Subscriber, but it is also possible to enroll a Sink directly with a Context when there is no need to decide based on the Name of the Instrument, which is what the Subscriber interface allows.
Each Humainary instrument library follows the same pattern of flow and control, which provides for circuit composability.
Composing Circuits
The interfaces defined in the Substrates library, the foundation for all instrument libraries, make it straightforward to connect and communicate across a diverse set of Instrument interfaces. In extending the Source interface, the Context interface allows registering one or more Sink instances that can take an emittance publication (output) from an instrument and translates this onto a method call (input) onto another type of Instrument interface. There can be many Context instances even of the same Instrument interface type but with different configuration parameters allowing a circuit to have multiple perspectives on the same observations but with different settings – the beginnings of a (collocated) collective intelligence.
The ultimate goal for our network circuit design is to make this transparently distributed across multiple mirrored worlds via event streaming and simulation to allow operations to plug in agent-based observers that can steer systems at different scopes and scales. This architectural approach will finally deliver on AIOps and make chaos engineering experiments effective.
Circuit Code Construction
To make the above less abstract, here is a snippet of code taken from our showcase repository that demonstrates the integration of two existing Humainary libraries – Stacks and Observers. There are two Context instances created here. The first manages Site instruments within the Stacks API. A Site , is an extension of the Instrument interface, used to capture and collect a call stack when an emit method call is made. The Observers.Context created, sources inputs from the Stacks.Context, taking the depth of the collecting Stack as input into an accumulating function that updates a Tally.
The objective of phase one is to make this code clean and concise. Phase two offers an externalized configuration option. Phase three will bring visual tooling and embedded simulation. Finally, phase four delivers a distributed collective intelligence platform.