Breaking Free

In modern software systems, observability has traditionally been guided by the “three pillars” approach: traces, logs, and metrics. While these conventional instruments have been effective in the past, they increasingly appear as limitations rather than tools in an era characterized by rapidly evolving software architectures and complex systems. Today’s toolkits represent a one-size-fits-all approach that fails to capture the nuanced needs of modern systems. This post advocates for extensible observability toolkits that empower teams to construct custom instruments (percepts) tailored to their requirements.

Understanding Instruments

At their most fundamental level, instruments are perceptual entities that perceive and emit observations about a subject of interest. This definition elucidates the true nature of instrumentation as a sensing and communication mechanism, transcending its role as a mere data collection tool.

A percept, an instrument, reads (senses) and records (emits) some aspect of a system, which we refer to as a subject, through an interface designed for instrumentation. The percept may have an internal state as well as configuration. A percept when invoked can filter and transform raw arguments or read states before sending a value to a channel. A channel abstracts the transport, routing, and flow control logic via a provided pipe component. A channel also provides access to the subject, although this is rarely necessary for custom instruments; in most cases, the pipe component is sufficient.

Implementing Instruments

Below is an example of a counter implemented using the Substrates API. It is important to note that this counter is not concerned with naming, concurrency, or state management. Instead, it focuses solely on translating an increment action into a value emitted along a pipeline to possibly many other percepts.

A gauge percept is easy to imagine from the above example of a counter. Percepts are the inlets.

Clear. Concise. Consistent. 🤍

Future Ready

The future of observability lies in developing novel instruments that capture system behavior more naturally, rather than refining existing log, metric, and trace-based instruments. Organizations that invest in extensible observability platforms gain a competitive advantage by enhancing their understanding and optimizing systems, leading to improved performance, reliability, and user experience. We must adopt a more flexible and extensible approach to observability, transcending the traditional three-pillar paradigm.

By standardizing observability at the substrate level through the Substrates API, we foster a remarkable level of simplicity that supports creativity and complexity to emerge in higher layers. Unlike other toolkits, which compel observations into metrics, traces, or logs, this design eliminates rigid constraints, enabling more flexible and contextual insights that adapt to the system’s unique requirements in sensing situations.

The Substrates API offers a simple but incredibly powerful and extensible foundation that lets observability in all its forms truly help us gain deep insights without forcing us to put things into rigid categories.

Next Steps

In forthcoming posts in the Observability X series, we will illustrate how percepts can be integrated into a composite circuit comprising conduits, channels, names, references, subjects, pipes, stages, and queues.