To acquire the knowledge of suitable software performance heuristics, developers must experience software execution in a new, more modern manner - a simulated environment of episodic machine memory replay.
The mirroring of software execution behavior, as performed by Simz (online) and Stenos (offline), has the potential to be one of the most significant advances in software systems engineering. Its impact could be as significant as that of distributed computing.
This post introduces the reasoning, thinking, and concepts behind a technology we call Signals, which we believe has the potential to have a profound impact on the design and development of software, the performance engineering of systems, and the management of distributed interconnected applications and services.
There is always tension between adaptability and structural stability in engineering and possibly life. We want our designs to be highly adaptable. With adaptation, our designs attempt to respond to change, sensed within the environment, intelligently with more change, though far more confined and possibly transient, at least initially. But there are limits to how far we can accelerate adaptation without putting incredible stress on the environment and the very system contained within.
Today, the stimulus used to develop machine intelligence is sensory data, which is transferred between devices and the cloud – the same data that concerns many consumers. But what if instead of sending data related to such things as a thermostat’s temperature set point, what was transmitted mostly concerned the action taken by the embedded software machine – an episodic memory of the algorithm itself?
Our brain houses billions of neurons (nerve cells) that communicate with each other through intricate networks of neural circuits. These circuits play a fundamental role in various cognitive functions, sensory processing, motor control, and generating thoughts and emotions. Why should it be different for Observability?
Most current observability technologies don’t fair well as a source of behavioral signals or inferred states. They are not designed to reconstruct behavior that would allow the level of inspection we would need to translate from measurement to signal and, in turn, the state effectively. They are designed with data collection and reporting in mind of the event, not the signal or state.
We should not differentiate whether an agent is deployed, especially with companies electing to manually instrument some parts of an application’s codebase using open-source observability libraries. Instead, we should consider whether the observer, an agent or library, is stateless concerning what and how it observes, measures, composes, collects, and transmits observations.
Reducing and compressing measurements is critical, which is much helped by representations extracted from the environment via hierarchical boundary determination. When this is not done automatically, what happens then is that the custom dashboard capabilities of the Observability solution need to be used to reconstruct some form of structure that mirrors the boundaries all but lost in the data fog. Naturally, this is extremely costly and inefficient for an organization.
The overemphasis on data instead of signals and states has created a great fog. This data fog leads to many organizations losing their way and overindulging in data exploration instead of exploiting acquired knowledge and understanding. This has come about with the community still somewhat unconcerned with a steering process such as monitoring or cybernetics.