Our brain houses billions of neurons (nerve cells) that communicate with each other through intricate networks of neural circuits. These circuits play a fundamental role in various cognitive functions, sensory processing, motor control, and generating thoughts and emotions. Why should it be different for Observability?
Author: William David Louth
Observability – The Significant Parts
Most current observability technologies don’t fair well as a source of behavioral signals or inferred states. They are not designed to reconstruct behavior that would allow the level of inspection we would need to translate from measurement to signal and, in turn, the state effectively. They are designed with data collection and reporting in mind of the event, not the signal or state.
Observability – Flat and Stateless
We should not differentiate whether an agent is deployed, especially with companies electing to manually instrument some parts of an application’s codebase using open-source observability libraries. Instead, we should consider whether the observer, an agent or library, is stateless concerning what and how it observes, measures, composes, collects, and transmits observations.
Bounded Observability
Reducing and compressing measurements is critical, which is much helped by representations extracted from the environment via hierarchical boundary determination. When this is not done automatically, what happens then is that the custom dashboard capabilities of the Observability solution need to be used to reconstruct some form of structure that mirrors the boundaries all but lost in the data fog. Naturally, this is extremely costly and inefficient for an organization.
The Data Fog of Observability
The overemphasis on data instead of signals and states has created a great fog. This data fog leads to many organizations losing their way and overindulging in data exploration instead of exploiting acquired knowledge and understanding. This has come about with the community still somewhat unconcerned with a steering process such as monitoring or cybernetics.
Observability in Perspective
There are many perspectives one could take in considering the observability and monitoring of software services and systems of services, but here below are a few perspectives, stacked in layers, that would be included.
Change in Observability
Observability is effectively a process of tracking change. At the level of a measurement device, software or hardware-based, change is the difference in the value of two observations taken at distinct points in time. This change detection via differencing is sometimes called static or happened change. Observability is all about happenings.
A Story of Observability
Once upon a time, there was a period in the world where humans watched over applications and services by proxy via dashboards housed on multiple screens hoisted in front of them – a typical mission control center. The interaction between humans and machines was relatively static and straightforward, like the environment and systems enclosed.
Bi-directional Observability Pipelines
Substrates changed everything by introducing the concept of a Circuit consisting of multiple Conduits fed by Instruments that allowed Observers to subscribe to Events and, in processing such Events, generate further Events by way of calling into another Instrument. But with the introduction of Percept and Adjunct, it is now possible for Observers attached to Circuit and its locally registered Sources to process Events that have come from a far-off Circuit within another process.
The Evolution of Substrates
With the latest update to the Substrates API, the metamorphosis to a general-purpose event-driven data flow library interface supporting the capture, collection, communication, conversion, and compression of perceptual data through a network of circuits and conduits has begun.