The Nervous System
Since the very beginning of the hype of Observability, we have contended that the link with Controllability must be maintained for there every to be a return of investment (ROI) that matches the extravagant claims from vendors pushing a message of greater visibility, when in fact what it is offered is the ability to put in motion the movement of massively large mindlessly and meaningless data from one cloud to another.
The following statement from an alternative engineering domain dealing with instruments in the physical world re-inforces this viewpoint that seems visionary when it should be common sense for a systems engineer tasked with monitoring and management systems of services.
“Measurement and control is the brain and nervous system of any modern plant. Measurement and control systems monitor and regulate processes that otherwise would be difficult to operate efficiently and safely while meeting the requirements for high quality and low cost.” – The Condensed Handbook of Measurement and Control, 3rd Edition.
There should exist one or more feedback loops between Observability and Controllability processes. An Observability component supplies a Controllability component with signals (of significance) generated from events occurring during the execution of a process (or plant). While regulating flow and interfacing with analyzers and actuators, the Controllability component must also emit signals back to the Observability component to influence the measurement process, including adjusting the functioning of sampling, sequencing, and sensitivities.
There are both bottom-up and top-down signaling streams within the human brain. Looking at the current approaches to Observability, we see neither provisioning nor placement for such fundamental mechanisms, means, and modes of co-operation between components.
Raw data, not of a signal kind, is grabbed, gobbled, garbled, and gorged on before being shoved down a pipeline via a collector out towards a black hole that some refer to as a “management” dashboard. In doing so, we maintain an ongoing engineering illusion of control when it is near impossible, certainly impractical, for humans to reason and respond to the rates of changes (events) occurring at space and time scales computing operates at in the digital world.
Even if we were to replace a human with some form of superintelligence, there would still be the issue of how it reaches out into an environment so far removed from it – spatially and temporally. The solution is that Observability and Controllability of some degree are preferably collocated, much like how they are at highly automated process plants.
We are not that good at doing so for all the talk of distributed computing, not even when it is of utmost importance for critical services systems. We move data around in more significant volumes, but the Observability and Controllability functionality distribution is centralized or non-existent.
Observability is still in its infancy. The toddler phase of screaming about unknown unknowns, deep systems, and debugging data must and will pass if we stop giving too much attention to noise and instead attune to signals. The community of practitioners, including site reliability engineering (SRE) and DevOps, needs to wrestle away control and take back parental responsibility from legacy logging, cloud, and application performance monitoring marketers and vendors who at present still have far too much say in the upbringing of this newborn within the software world.
We need to reach out to others in other engineering domains, such as process measurement, control, and automation, to not have to learn from far too many of our unnecessary failures and missteps along this development path.
For us, that starts with getting the basics, the fundamental, not pillars, in place, starting with offering greater diversity at the instrument level and having an instrument infrastructure, a toolkit, that allows for multiple instruments to be composed and combined coherently concisely. Well, that is the plan for 2022. Happy New Year.