Simplifying Instrument Synthesis


Three overriding principles are applied in nearly every design decision for the Substrates API, which is the foundation for all other instruments (types/kits) under the Humainary initiative – consistency, conciseness, and correctness.


The first principle pertains to the users (callers) of the instruments managed by the Substrates runtime. No matter the type of Observability instrument employed, all other aspects connected to configuration, creation, contextualization (naming, nesting), resource management, routing, pipeline integration, interaction contract, and extensibility should be conceptually and code-wise the same. Across instrument kits built on Substrates API, the only difference a caller should experience is specific to the Instrument extension interface. This seems obvious, yet no other open-source solution has done so to date, OpenTelemetry included.


The second principle is that a developer of a new instrument type, an extension of the Instrument interface, should only ever have to focus on the data measurement capture facet; publication, pipelining, contextualization, resource management, integration, extensibility, diagnostics, scalability, concurrency, etc., should be removed entirely from the design concerns and coding required of a developer. This principle will be detailed in the rest of this post, where the Inlet interface is pivotal.


This might seem a strange principle when the scope of such assessment pertains solely to the measurement data capture operation. How difficult can it be to increment a Counter or measure the timing interval between two Timer operations? Only when we need to publish along a pipeline does what seemed relatively straightforward become far more challenging and unnecessarily complicated. Correctness, in particular ordering, comes into play when an Observer consumes a series of event-triggered emitted measurements.


An example will best highlight the issues with pipelines beyond measurement by an Instrument. Below is an interface for an Instrument that accumulates values provided by clients. It is important to note that the Accumulator interface does not expose the cumulative total to clients. It only provides callers with the means to add to the internal state. This differs from other metrics libraries, where the instrument’s internal state is made public in some form to support data collection by a sampling thread.

The following code will detect whether an Outlet, a pipeline consumer, observes a correct output sequence from an Accumulator, where a value emitted is not less than a previously consumed value made available via the Event::emission method.

Here is the first attempt at implementing the Accumulator interface, using an AtomicLong to manage the internal state, as multiple Threads can call the Accumulator implementation simultaneously. See the problem with pipelining the AtomicLong cumulative value?

The problem with the above code is that the Inlet::emit and AtomicLong::addAndGet are not within the same transactional scope. The value is calculated and then emitted. There is a small window for corruption in the pipeline sequencing between these two methods.


A simple crude solution to this problem is to enforce single-threaded execution at the Accumulator::add method level, severely limiting scalability, especially when the pipeline has multiple Subscribers, some of which are gateways to external endpoints.

The Inlet interface, which extends the Pipe interface, offers various overloaded emit methods that can be used to offload the execution of the state change to a Circuit’s Current (event flow). As shown below, addressing the correctness issue with the Atomic implementation requires only a small code change. Instead of the client calling Thread executing the state change, the pipelining Executor Thread will now perform the AtomicLong::addAndGet invocation providing the delta value passed to the emit method.

If the pipeline only accesses the internal state field, the field type can change from an AtomicLong to a long primitive.

Another way to address the correctness in pipelining an Instrument’s internal state and its changes is not to keep the state. Rather than sending the cumulative value, all that is pipelined is the delta value. This can be useful when you want to record each add event. But in doing so, the burden of aggregation is placed on each consumer connected to the pipeline. Such consumers will then have to synchronize access to their internal state. And what do consumers do with their internal state? Well, more than likely, transmit it further along the pipeline. We’re back to square one, with the problem now duplicated.