Naming is Hard
In the realm of observability, naming holds paramount importance. It directly influences measurement tracking, correlation, and interpretation, significantly impacting system visibility and problem-solving capabilities. Names are essential for distinguishing the diverse range of system measurements collected by observability tools. A consistent and unified naming and reference system across instruments is crucial for achieving a comprehensive understanding. In large-scale systems, hierarchical naming conventions provide context at various levels, from high-level services to specific functions. This hierarchical approach facilitates filtering and aggregation across services, enabling observability tools to group and display related data meaningfully. Names serve as multifaceted organizational and functional tools in observability, empowering developers to structure and navigate vast quantities of data.
An Absence of a Name
Unfortunately, today’s tooling lacks a unified Name interface, which is a significant issue, especially considering that observability heavily relies on consistent identifiers to link and aggregate across different percepts and levels of granularity. For instance, names might appear as simple strings for certain metrics but as structured fields with additional descriptive attributes in logs or traces. This inconsistency complicates how we refer to or query observability data, necessitating additional logic to handle different data structures based on the instrument type. Currently, each instrument in OpenTelemetry has a different approach to representing and handling names, leading to fragmentation and complications with data.
A Name for All Names
The Substrates API offers a Name interface that extends Extent (a hierarchical concept) to address this concern. Below is some code showing various common operations and capabilities of the Name interface.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 |
// initialize a circuit fabric var cortex = cortex (); // construct a Name instance using a class var name = cortex.name ( java.lang.String.class ); // prints the FQN of the name out.println ( name.path () ); // extend the name to include a new name part name = name.name ( "toString" ); // prints parts in reverse order starting with “toString" name .streamFrom () .map ( Name::part ) .forEach ( out::println ); // using functional methods to traverse and collect out.println ( name.foldTo ( first -> new StringBuilder ( first.part () ), ( sb, next ) -> sb .append ( "\\" ) .append ( next.part () ) ).toString () ); |
Contextual Coupling
A more significant challenge arises when considering names, references, identifiers, and subjects. When an instrument needs to know and maintain such information it is akin to creating an instance of an object in programming code and forcing it to be aware of its name in the source code. This analogy underscores the redundancy and confusion that emerge when an instrument, whose primary function is to capture and transmit data, is burdened with contextual awareness regarding its reference or identification within a broader system. OpenTelemetry establishes a strong connection between what should be distinct concerns, and it does this in different and convoluted ways across instruments and programming languages. Some of this can be attributed to the evolution of instrument types over different periods, with tracing being the initial step, followed by metrics, and then logging, not to mention multiple vendors taking turns to steer. The most common cause for the decline in the overall design and development quality is the urgency to deliver something quickly in a very siloed manner. No single individual or vendor had a vision of a solid foundation.
Separating Concerns
A percept (instrument) should only be responsible for its core function: sensing, synthesizing, and sending signals. Everything else, including identification and naming, belongs to the telemetry channel/pipeline infrastructure. Percepts should be able to evolve independently of identity schemes or routing patterns.
In observability, systems often require the aggregation of data from multiple instruments and sources. When instruments are decoupled from their naming, they become simpler and can be easily combined to create more complex pipelines. At the pipeline level, data collected by each instrument can be enriched, combined, and classified, allowing observability systems to apply higher-order logic without requiring modifications to individual instruments. This level of decoupling also enables more advanced composition strategies, such as dynamically adjusting identifiers based on runtime context or system states.
Substrating Subjects
In the Substrates API, two concepts are used to construct and organize percepts: conduit and circuit. The circuit component is responsible for managing computational resources, such as queues and threads, within the underlying runtime. To facilitate scalability, developers can partition and map the domain space into multiple circuits. Each circuit is responsible for creating one or more instances of the next important component, a conduit, which manages the underlying channels and pipes utilized by percepts.
The code below uses the Valve custom percept developed in the last post in this series.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
var cortex = cortex (); var circuit = cortex.circuit ( cortex.name ( "network.5g" ) ); var conduit = circuit.conduit ( cortex.name ( "region.eu-nl" ), Valve::new ); var valve = conduit.compose ( cortex.name ( "pop.ams" ) ); |
In a future post in this series, we’ll revisit conduits, sources, and sinks. For now, let’s focus on the following code. It sets up the capturing of emittances from valves connected to a conduit created by a circuit. Once the valve is opened, the sink is drained, and the subject of each capture is printed. Here it’s only one.
|
20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 |
var sink = conduit .source () .sink (); valve.open (); sink .drain () .map ( Capture::subject ) .forEach ( subject -> out.println ( subject.foldTo ( first -> new StringBuilder ( first .name () .path ( '/' ) ), ( sb, next ) -> sb.append ( "/" ) .append ( next .name () .path ( '/' ) ) ).toString () ) ); |
When executed, the code will output the following sequence: the name of the circuit, followed by the name of the conduit, and finally the name of the valve. No matter the instrument employed the interaction and structuring are the same. This is a significant simplification of the observability instrumentation space.
|
1 2 3 |
network/5g/region/eu-nl/pop/ams |