Principles
Three overriding principles are applied in nearly every design decision for the Substrates API, which is the foundation for all other instruments (types/kits) under the Humainary initiative – consistency, conciseness, and correctness.
Consistency
The first principle pertains to the users (callers) of the instruments managed by the Substrates runtime. No matter the type of Observability instrument employed, all other aspects connected to configuration, creation, contextualization (naming, nesting), resource management, routing, pipeline integration, interaction contract, and extensibility should be conceptually and code-wise the same. Across instrument kits built on Substrates API, the only difference a caller should experience is specific to the Instrument extension interface. This seems obvious, yet no other open-source solution has done so to date, OpenTelemetry included.
Conciseness
The second principle is that a developer of a new instrument type, an extension of the Instrument interface, should only ever have to focus on the data measurement capture facet; publication, pipelining, contextualization, resource management, integration, extensibility, diagnostics, scalability, concurrency, etc., should be removed entirely from the design concerns and coding required of a developer. This principle will be detailed in the rest of this post, where the Inlet interface is pivotal.
Correctness
This might seem a strange principle when the scope of such assessment pertains solely to the measurement data capture operation. How difficult can it be to increment a Counter or measure the timing interval between two Timer operations? Only when we need to publish along a pipeline does what seemed relatively straightforward become far more challenging and unnecessarily complicated. Correctness, in particular ordering, comes into play when an Observer consumes a series of event-triggered emitted measurements.
Cumulation
An example will best highlight the issues with pipelines beyond measurement by an Instrument. Below is an interface for an Instrument that accumulates values provided by clients. It is important to note that the Accumulator interface does not expose the cumulative total to clients. It only provides callers with the means to add to the internal state. This differs from other metrics libraries, where the instrument’s internal state is made public in some form to support data collection by a sampling thread.
|
1 2 3 4 5 6 7 8 9 10 |
interface Accumulator extends Instrument< Long > { void add ( long delta ); } |
The following code will detect whether an Outlet, a pipeline consumer, observes a correct output sequence from an Accumulator, where a value emitted is not less than a previously consumed value made available via the Event::emission method.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
conduit .source () .consume ( new Outlet<> () { private long last; @Override public void accept ( final Event< Long > event ) { final var value = event.emission (); assert value > last; last = value; } } ); |
Here is the first attempt at implementing the Accumulator interface, using an AtomicLong to manage the internal state, as multiple Threads can call the Accumulator implementation simultaneously. See the problem with pipelining the AtomicLong cumulative value?
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
final class Atomic implements Accumulator { private final Inlet< Accumulator, Long > inlet; private final AtomicLong state = new AtomicLong (); Atomic ( final Inlet< Accumulator, Long > inlet ) { this.inlet = inlet; } public void add ( final long delta ) { inlet.emit ( state.addAndGet ( delta ) ); } } |
The problem with the above code is that the Inlet::emit and AtomicLong::addAndGet are not within the same transactional scope. The value is calculated and then emitted. There is a small window for corruption in the pipeline sequencing between these two methods.
| # | Thread | Call | Value |
|---|---|---|---|
| 1 | A | addAndGet | 1 |
| 2 | B | addAndGet | 2 |
| 3 | B | emit | 2 |
| 4 | A | emit | 1 |
A simple crude solution to this problem is to enforce single-threaded execution at the Accumulator::add method level, severely limiting scalability, especially when the pipeline has multiple Subscribers, some of which are gateways to external endpoints.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
final class Synchronized implements Accumulator { private final Inlet< Accumulator, Long > inlet; private long state; // constructor omitted public synchronized void add ( final long delta ) { inlet.emit ( state += delta ); } } |
The Inlet interface, which extends the Pipe interface, offers various overloaded emit methods that can be used to offload the execution of the state change to a Circuit’s Current (event flow). As shown below, addressing the correctness issue with the Atomic implementation requires only a small code change. Instead of the client calling Thread executing the state change, the pipelining Executor Thread will now perform the AtomicLong::addAndGet invocation providing the delta value passed to the emit method.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
public void add ( final long delta ) { inlet.emit ( delta, state::addAndGet ); } |
If the pipeline only accesses the internal state field, the field type can change from an AtomicLong to a long primitive.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
final class Simple implements Accumulator { private final Inlet< Accumulator, Long > inlet; private long state; // constructor omitted public void add ( final long delta ) { inlet.emit ( delta, ( long value ) -> state += value ); } } |
Another way to address the correctness in pipelining an Instrument’s internal state and its changes is not to keep the state. Rather than sending the cumulative value, all that is pipelined is the delta value. This can be useful when you want to record each add event. But in doing so, the burden of aggregation is placed on each consumer connected to the pipeline. Such consumers will then have to synchronize access to their internal state. And what do consumers do with their internal state? Well, more than likely, transmit it further along the pipeline. We’re back to square one, with the problem now duplicated.