The Components of a Circuit
The Humainary initiative is progressing rapidly along a path towards delivering the ultimate observability toolkit in the form of a network circuit consisting of a diverse set of instruments, contexts, subscribers, and sinks. We distinguish between an observability circuit and data pipeline based on the degree of intelligence and locality of processing emittances (signal or value).
For the most part, the data pipelines touted by the vendor contingent within the observability community are mindless movers of measurements from a source point to a back-end endpoint in the cloud – other than the batching, compressing, and dropping of data payloads, there is not much to talk about here as vendors do most work after the costly and wasteful transmission.
Whereas all the Humainary instrument libraries, built on top of the Substrates foundational library, are designed to publish, consume, and transform emittances effortlessly across different collocated integrated points of contextualization, communication, coordination, and consumption. The diagram below presents the components of the circuitry, including communication flows.
The Flow of Signals
Instrument, such as a
Counter, will publish emittances to the
Context it was created by when methods within its interface are called. For a
Counter, this would be
Context, acting as a
Source of emittances, will first coordinate with one or more registered
Subscriber instances for interest in the emitting
Instrument, if it has not been done so previously for the particular
Instrument, after which it will forward the emittance onto one or more registered
Sink is registered via a
Subscriber, but it is also possible to enroll a
Sink directly with a
Context when there is no need to decide based on the
Name of the
Instrument, which is what the
Subscriber interface allows.
Each Humainary instrument library follows the same pattern of flow and control, which provides for circuit composability.
The interfaces defined in the Substrates library, the foundation for all instrument libraries, make it straightforward to connect and communicate across a diverse set of
Instrument interfaces. In extending the
Source interface, the
Context interface allows registering one or more
Sink instances that can take an emittance publication (output) from an instrument and translates this onto a method call (input) onto another type of
Instrument interface. There can be many
Context instances even of the same
Instrument interface type but with different configuration parameters allowing a circuit to have multiple perspectives on the same observations but with different settings – the beginnings of a (collocated) collective intelligence.
The ultimate goal for our network circuit design is to make this transparently distributed across multiple mirrored worlds via event streaming and simulation to allow operations to plug in agent-based observers that can steer systems at different scopes and scales. This architectural approach will finally deliver on AIOps and make chaos engineering experiments effective.
Circuit Code Construction
To make the above less abstract, here is a snippet of code taken from our showcase repository that demonstrates the integration of two existing Humainary libraries –
Observers. There are two
Context instances created here. The first manages
Site instruments within the
Stacks API. A
Site , is an extension of the
Instrument interface, used to capture and collect a call stack when an
emit method call is made. The
Observers.Context created, sources inputs from the
Context, taking the depth of the collecting
Stack as input into an accumulating function that updates a
The objective of phase one is to make this code clean and concise. Phase two offers an externalized configuration option. Phase three will bring visual tooling and embedded simulation. Finally, phase four delivers a distributed collective intelligence platform.