Coordinating Change
When crafting custom instruments for measuring software execution, managing the instrument’s state can be surprisingly intricate. The primary challenge lies in ensuring precise and consistent data collection and measurement across potentially large volumes of concurrent requests. Each request may modify internal state values that subsequently feed into a pipeline. Without proper concurrency control, race conditions can occur, resulting in inaccurate data or, in severe cases, a corrupted system state. Ensuring consistency conflicts with maintaining high performance, particularly when state updates are frequent or rapid. Synchronous state management can guarantee data integrity but will in general introduce latency. Striking the right balance is crucial, especially when the instrument must operate within tight resource budgets.
Simplifying State
An observability toolkit, particularly a highly extensible one, is pivotal in simplifying state management for custom instrumentation. It provides built-in mechanisms to safeguard and simplify state handling across concurrent operations, complex pipelines, and distributed systems. By using a well-designed toolkit, engineers can concentrate on inspecting, transforming, and transmitting data rather than grappling with concurrency and state consistency difficulties.
Safeguarding State
Let us construct a rudimentary illustration to explain the complexities associated with state management, which hamper the development of custom instruments known as percepts. This concept will become more comprehensible as the series of posts develops. Below is an initial implementation of a Valve percept.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
import io.humainary.substrates.api.Substrates.Channel; import io.humainary.substrates.api.Substrates.Pipe; final class Valve { enum Status { OPENED, CLOSED } private final Pipe < Status > pipe; Valve ( final Channel < Status > channel ) { pipe = channel.pipe (); } void open () { pipe.emit ( OPENED ); } void close () { pipe.emit ( CLOSED ); } } |
Leaving aside concurrency concerns, the above code could be considered flawed because it permits the methods, open or close, to be executed consecutively without altering the valve’s status. Nothing prevents a consumer or observer at the other end of the pipe from receiving OPENED followed by OPENED or CLOSED followed by CLOSED. To address this, the percept needs to maintain a state and ensure that the emit method is executed only when a specific condition is met, as in the revised code below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
final class Valve { enum Status { OPENED, CLOSED } private final Pipe < Status > pipe; private Status state = CLOSED; Valve ( final Channel < Status > channel ) { pipe = channel.pipe (); } void open () { emit ( OPENED ); } void close () { emit ( CLOSED ); } private void emit ( final Status status ) { if ( state != status ) { state = status; pipe.emit ( status ); } } } |
Unfortunately, the above code lacks thread safety. Multiple executing threads can concurrently invoke the open() and close() methods, resulting in an incorrect sequence of emitted states. The valve percept developer must consider concurrency, the performance of state controls introduced into the code, and the ordering of emitted states to downstream consumers.
Staging States
The Substrates API simplifies the development of custom instruments to an unprecedented degree. One of its key features is eliminating state management complexities by offloading state changes to pipeline processing. This ensures the safety and reliability of the instrument and the state emitted.
To achieve this, a minor modification is required to the pipe’s initialization code within the constructor. Specifically, a Path object is created by Channel and then used to create a Pipe. With this change, the Path code now performs the state change. Because it is executed by only one thread at a time, the current (thread) within a circuit, emissions are guaranteed to be ordered up to the emit call point.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
import io.humainary.substrates.api.Substrates.Channel; import io.humainary.substrates.api.Substrates.Pipe; import io.humainary.substrates.api.Substrates.Stage; final class Valve { enum Status { OPENED, CLOSED } private final Pipe < Status > pipe; Valve ( final Channel < Status > channel ) { pipe = channel .path () .guard ( CLOSED, ( prev, next ) -> prev != next ).pipe (); } void open () { pipe.emit ( OPENED ); } void close () { pipe.emit ( CLOSED ); } } |
In a future post in this series, we will return to the Path interface when building analytical percepts.
Summing Up
A toolkit, like Humainary’s Substrates, with its built-in state management and pipeline execution capabilities significantly simplifies instrument development by effectively managing concurrency and consistency challenges. This allows developers to focus on measuring software execution and extracting valuable insights rather than getting bogged down by troubleshooting low-level state issues.
Today, it is impossible to develop such percepts with existing toolkits, such as OpenTelemetry, without being forced to map them to an existing instrument such as a metric, trace, or log.