The observability crisis isn’t about missing data; rather, it’s about missing meaning. While we’re inundated with metrics, traces, and logs, we’re simultaneously deprived of understanding. Our systems are narrating stories, yet we’re merely collecting the words. Confinement to a pixel-level fixation on metrics, traces, and logs renders today’s observability practices fundamentally retrospective. They measure the consequences, not the progression. They concentrate on fragments, not the underlying forces. To truly comprehend systems—to anticipate, guide, and sustain—we must transcend data collection and adopt the perception of dynamic narratives.
Drawing inspiration from the insights of cybernetics and systems thinking, we propose a cohesive framework centered around three essential, interacting functional components: Observability, Controllability, and Operability. This universal framework, underpinned by foundational capabilities (measurement, memory, and models), offers a potent lens for conceiving, comprehending, and controlling virtually any purposeful system, effectively bridging the disparity between abstract theoretical concepts and tangible practical applications.
Modern observability suffers from an existential void, mistaking metric abundance for genuine understanding. Current practices emphasize granular measurement but neglect synthesizing comprehensive system insights. A paradigm shift toward continuous, holistic assessments of system stability and confidence—visualized as intuitive, dynamic representations—leverages innate human cognitive strengths. This transition addresses the root architectural flaw in current methodologies, replacing numerical theater with true insight into complex digital environments.
Dazzling dashboards, collapsing systems: Has observability become a beautiful lie? We’ve meticulously instrumented our complex digital world, yet meaningful insights remain elusive amidst a deluge of data. This post argues that the limitations of traditional observability in the face of emergent behaviors and dynamic topologies demand a radical shift towards a more comprehensive “situational intelligence” – one that moves beyond mere monitoring to true comprehension and proactive action. It’s time to admit: observability alone can’t save us.
What if simplicity isn’t about wielding a machete to prune the excess, but about constructing a lighthouse whose beam remains clear and steadfast, piercing through any fog? We frequently envision simplicity as minimalism: a stark, uncluttered space, a reduction to the fundamental elements. It’s akin to stripping a machine down to its core components.
This post introduces the Serventis framework, illuminating its semiotic architecture and demonstrating its power to transmute raw data into rich, resonant sense-making. We’ll explore how Serventis cultivates multi-agent intelligence, fosters situational awareness, and lays the bedrock for adaptive, self-aware systems.
Stop chasing the elusive dream of becoming a hotbed of internal innovation. It’s a costly, frustrating, and often futile pursuit. Instead, embrace your true strength as a large, established enterprise: your ability to adopt, adapt, and scale the innovations of others. Be a smart, fast follower. Be a ruthless acquirer of promising technologies. Be a master of integration and optimization.
Like a spacecraft with perfect sensors but no navigation intelligence, our tools measure everything while comprehending little. The administrator mindset treats observability as a data problem, capturing snapshots of system state through logs, metrics, and traces. In contrast, the engineer mindset sees it as a comprehension challenge, focusing on how events form sequences, how components create feedback loops, and how patterns emerge over time.
What if the events we perceive aren’t objective truths but rather interpretations shaped by our cognitive frameworks? This question lies at the heart of a paradigm shift in computer science, moving us from a rigid, event-centric view to a more nuanced, sign-centric understanding of systems. In this new paradigm, understanding isn’t about what objectively occurs, but how meaning emerges through interpretation and context.
Current observability standards misappropriate terms like “semantics” while delivering mere data collection. This post critiques OpenTelemetry’s “Semantic Conventions” as standardized naming taxonomies that lack genuine interpretive power. Rather than capturing meaning, these approaches create cognitive burden through maximum data capture—contradicting how intelligence functions as a selective abstraction engine.
Our observability tools are working against the very way our brains make sense of the world. While we think in high-level situations and narratives, our dashboards bombard us with disconnected metrics and fragmented data points. This fundamental mismatch forces engineers to waste precious cognitive resources translating numbers into meaning—a burden that becomes dangerous during critical incidents when clarity matters most.
The world of modern software is a symphony of interconnected services, microservices, and cloud infrastructure, constantly evolving and adapting to changing demands. Yet our tools for understanding these complex systems often remain rooted in a paradigm of static dashboards and isolated metrics. While these tools might suffice for simpler systems, they fall short when applied to the dynamic complexity of cloud-native architectures, digital twins, and large-scale service ecosystems.
The following post challenges the epistemological assumption that quantitative data provides the most reliable and objective form of knowledge. It argues that our obsession with counting and measuring has led to a distorted understanding of reality, where numbers replace the richness and complexity of human experience. This critique aligns with postmodern and phenomenological perspectives, which emphasize the importance of subjective experience and the limitations of objective measurement. We’ve become prisoners of our invention.
For decades, we’ve built computational systems under a grand illusion: that by mastering data collection, storage, and movement, intelligence and insight will naturally emerge. The reality we face today across observability, digital transformations, data engineering, and artificial intelligence proves otherwise. Our systems are efficient but not effective. They process but don’t understand. They inform but don’t enlighten. This isn’t just a technical failure—it’s philosophical. We’ve engineered complexity but not comprehension. We’ve built means for data but not for meaning.
We trace the evolution of system management from manual configuration to dynamic situational awareness, highlighting the shift from maintaining static states to enabling systems to self-regulate and adapt in real-time. As we consider the future of system observability, it has become evident that data collection alone is insufficient. We require systems capable of comprehending the significance of the data they generate. This is where our Semiotic Twin concept comes into play. Just as human experts interpret numbers beyond mere numerical values, our monitoring systems must transcend mere metric recording to attain genuine situational awareness.
In our quest to model human experience through computation, we often forget that a situation is more than the sum of its measurable parts. While we can map events, log data, and predict patterns, we cannot fully capture the ineffable quality of human awareness—the way a mother’s intuition sparks before conscious thought, or how a shared glance between strangers can rewrite social rules in an instant.
In the drive to “observe all the things,” we’ve built comprehensive pipelines that churn out terabytes of data. Yet data alone isn’t understanding. Without context, without synthesis, even the most powerful observability suite is just a loudspeaker blaring noise.
In our exploration of system observability and situational intelligence, we established a universal pattern for understanding system behavior through sources, subjects, signs, and signals (including states). This recursive pattern is foundational to what we call Semiosphere, a living semiotics system, which builds upon Substrates and Signetics—a computational execution model enabling the processing, routing, and transformation of signs and signals. Together, these elements create an architecture that supports situational awareness, digital twins, and business process intelligence.
In this post, we introduce the concept of Container in the Substrates API. A Container is a collection of Conduits of the same emittance data type. With the concept of Container, we introduce a new element in the Subject hierarchy. So instead of a Conduit being parented by a Circuit, it can now also be parented by a Container which is then parented by a Circuit.
In an industry that prides itself on innovation and progress, the observability sector stands as an anomaly – a field seemingly trapped in amber, where genuine evolution has been replaced by an endless cycle of marketing-driven rebranding. Despite decades of supposed advancement, the fundamental approaches to understanding system behavior remain stubbornly unchanged, masked only by increasingly elaborate terminology and ever-growing data collection.
Software and systems performance engineering needs to undergo a fundamental shift. While the traditional focus on hardware optimization—known as mechanical sympathy—remains valuable, the increasing complexity of modern distributed systems demands a more comprehensive approach. This document outlines system sympathy, a new mindset for understanding and optimizing system-wide performance in enterprise environments.
The future of software development doesn’t lie in relinquishing our understanding to AI, but rather in establishing a genuine partnership that augments our cognitive faculties while preserving the profound comprehension that enables the creation of exceptional software. As we progress with the integration of artificial intelligence, maintaining this equilibrium between automation and awareness will be paramount for the ongoing evolution of our discipline.
Simplicity isn’t the antithesis of complexity; rather, it’s the outcome of acquiring knowledge from complexity. True learning doesn’t entail accumulating an excessive amount of information or processes. Instead, it entails comprehending patterns sufficiently deeply to simplify them, thereby integrating them into our fundamental understanding.
We’re almost done with this series, and we’ve covered some rather profound concepts. Let’s take a quick look at what we’ve learned so far. We’ll break it down into two parts: form and function.
The Substrates API enables the direct utilization of Pipes and Channels, while simultaneously offering a mechanism to construct percepts that adorn these fundamental elements of any sensing and synthesis pipeline. To construct a Conduit that facilitates the on-demand creation of percepts upon receiving a name, we must provide a Composer to the conduit method within the Circuit interface.
At its heart, observability is about change. Without change, there’s nothing to observe. A static system, frozen in time, offers no insights. Time itself is meaningless without change—it’s merely a measure of movement, of transformation. A clock ticks because its hands move; atoms vibrate. Change is the pulse of existence. So, when we talk about observability, we’re exploring how we perceive and interpret change within the systems we design.
We need a unified theory of observability to address the limitations of current practices while still being practical and straightforward to implement. It should go beyond different areas of the system, like service monitoring, user experience, and system changes. It should combine quantitative measurements with qualitative understanding while connecting with context.
Organizations adopting AI for cloud operations face a critical evaluation of AIOps approaches. While recent advancements in AIOps frameworks standardize AI agent evaluation and enhancement, they risk a dead-end relying on conventional observability models lacking semantic depth. Service Cognition principles should form the AIOps foundation for truly autonomous cloud operations.
Organizations are drowning in an ocean of metrics. Every click, interaction, and transaction generates quantitative data points, creating vast repositories of numbers that promise insights but often deliver confusion. The challenge isn’t just the volume of data – it’s the fundamental question of how to transform these raw metrics into meaningful qualitative understanding.
The Substrates API separates the concerns of task definition, scheduling, and execution, while simultaneously enabling robust patterns for composing asynchronous workflows. The capability to post scripts, coupled with control over execution order facilitates adaptable and maintainable event-driven systems.
Resource management is crucial for scaling digital twin representations in service architectures. Proper resource cleanup ensures system stability during scaling up and down. Rigorous resource management prevents resource exhaustion and orphaned resources consuming capacity. The Substrates API’s approach to resource management ensures resources are acquired and released during scaling operations
The State and Slot interfaces provide a type-safe approach to managing and transmitting state. The State interface offers an immutable way to collect and chain Slots, each with a name, type, and value. It enforces type safety through method overloading, supporting primitives, strings, names, and states. These interfaces create a foundation for constructing states and communicating changes type-safely and performantly for small sets.
How we model observables in our systems shapes how we understand, monitor, and reason about their behavior. In the proposed subject-based model, all changes are simply emissions from a subject, eliminating the artificial distinction between states and signals that plague many current observability systems.
We had Claude.ai compose a review in the style of Papers We Love for an article from 2014. The goal was to see how well it predicted the current state of Observability 10 years on.
A significant challenge with siloed-oriented telemetry toolkits, such as OpenTelemetry, is that each instrument employs a distinct data collection and transmission process. Each pillar, including tracing, metrics, and logging, possesses numerous configuration parameters to manage underlying resource usage, such as buffers and execution threads. There is no inherent mechanism to synchronize or share context among these systems, resulting in fragmented and potentially conflicting telemetry data.
In observability, channels underpin streaming architectures for real-time monitoring and alerting, enabling dynamic system reactions and scalability. Understanding and leveraging channels creates resilient, scalable, and maintainable systems, which is essential in today’s interconnected world. Instruments provide a high-level, domain-specific interface on top of channels. Beneath the surface, they are essentially specialized wrappers around channel operations.
Traditionally, observability data pipelining operates like a single assembly line, where workers halt, examine items, and consult manuals before proceeding. This approach is functional but slow due to inspections. Instead of thinking of observability as one long pipeline, think of it as a graph with many possible routes. Once we know where something needs to go, we can create a specific route for it, like how a GPS creates a particular path for your destination instead of having to check a map at every intersection.
Observability has been reduced to the straightforward processing of data collected and its forwarding to a remote centralized endpoint. While frameworks such as OpenTelemetry have standardized this, they overlook a fundamental aspect of true observability: the dynamic nature of observation itself. A more effective and efficient observability approach establishes live, adaptable connections between information sources and observers, both locally and remotely, as well as online and offline.
In our pursuit of comprehending complex systems through observability, we’ve developed increasingly convoluted tools that focus on capturing the structural context—the static backdrop of components, configurations, and infrastructure against which our systems function. However, these tools overlook the rich tapestry of behavioral and situational contexts that lend meaning to system events.
In the realm of observability, naming holds paramount importance. It directly influences measurement tracking, correlation, and interpretation, significantly impacting system visibility and problem-solving capabilities. Names are essential for distinguishing the diverse range of system measurements collected by observability tools.
Managing state in observability instrumentation presents unique challenges, particularly when dealing with concurrent operations. The Humainary Substrates API offers an elegant solution by offloading state management to pipeline processing. Through its Stage interface and built-in concurrency controls, developers can create custom instruments (percepts) without getting entangled in thread safety issues.
Today’s observability toolkits represent a one-size-fits-all approach that fails to capture the nuanced needs of modern systems. This post advocates for extensible observability toolkits that empower teams to construct custom instruments (percepts) tailored to their requirements.
Our approach to observability has been constrained by the notion that instruments and observers are fundamentally distinct entities. This post offers an alternative perspective: observers are themselves instruments, constructing more comprehensive and insightful observations by integrating data from lower-level instruments.
Simplification initiatives in organizations often paradoxically increase complexity due to misinterpretation and uncoordinated implementation across different levels. Achieving meaningful simplification requires a holistic approach, clear communication, and an understanding of complex systems dynamics to avoid the pitfalls of oversimplification or mere tactical efficiency improvements.
Organizational silos form in complex systems when collaboration becomes costly or uncertain, leading to inefficiencies and communication barriers. Effective integration requires balancing standardization with simplification, fostering collaboration across units, and managing the tension between short-term metrics and long-term transformative work.
This article presents an A-Z glossary of key concepts related to observability in complex systems and software engineering. It covers topics ranging from Attention and Boundaries to Topologies, emphasizing the importance of intelligent data analysis, contextual understanding, and adaptive learning in monitoring and managing modern distributed systems.
In an era where AI is rapidly transforming our digital landscape, how can we ensure that human-AI collaboration reaches its full potential? The answer lies in a paradigm shift towards task-centricity.
The observability community should move away from traditional metaphors like pillars and pipelines and adopt new ones like substrates and circuits. By doing this, we can gain a new and innovative outlook on tools and techniques, leaving behind outdated thinking that prioritizes data over decisions and content over control.
The prevailing metaphors of pillars and pipelines in observability have limited our understanding and hindered progress. These metaphors promote siloed thinking and a focus on data collection over actionable insights.
Abstraction and simplification are two fundamental principles that often work together in the design of systems. With abstraction, we reduce system complexity by focusing on the essential aspects in the area of structure, elements, and behavior.
Here we explore why the industry needs to move beyond the legacy tools and embrace a more dynamic and adaptable approach to gleaning genuine value from the ever-growing ocean of data collected.
As engineering systems grow ever more complex, the engineering community’s focus on simplistic measurement and reporting hinders achieving operational scalability by way of sensemaking and steering of such systems of systems.
To acquire the knowledge of suitable software performance heuristics, developers must experience software execution in a new, more modern manner – a simulated environment of episodic machine memory replay.
The mirroring of software execution behavior, as performed by Simz (online) and Stenos (offline), has the potential to be one of the most significant advances in software systems engineering. Its impact could be as significant as that of distributed computing.
This post introduces the reasoning, thinking, and concepts behind a technology we call Signals, which we believe has the potential to have a profound impact on the design and development of software, the performance engineering of systems, and the management of distributed interconnected applications and services.
There is always tension between adaptability and structural stability in engineering and possibly life. We want our designs to be highly adaptable. With adaptation, our designs attempt to respond to change, sensed within the environment, intelligently with more change, though far more confined and possibly transient, at least initially. But there are limits to how far we can accelerate adaptation without putting incredible stress on the environment and the very system contained within.
Today, the stimulus used to develop machine intelligence is sensory data, which is transferred between devices and the cloud – the same data that concerns many consumers. But what if instead of sending data related to such things as a thermostat’s temperature set point, what was transmitted mostly concerned the action taken by the embedded software machine – an episodic memory of the algorithm itself?
Our brain houses billions of neurons (nerve cells) that communicate with each other through intricate networks of neural circuits. These circuits play a fundamental role in various cognitive functions, sensory processing, motor control, and generating thoughts and emotions. Why should it be different for Observability?
Most current observability technologies don’t fair well as a source of behavioral signals or inferred states. They are not designed to reconstruct behavior that would allow the level of inspection we would need to translate from measurement to signal and, in turn, the state effectively. They are designed with data collection and reporting in mind of the event, not the signal or state.
We should not differentiate whether an agent is deployed, especially with companies electing to manually instrument some parts of an application’s codebase using open-source observability libraries. Instead, we should consider whether the observer, an agent or library, is stateless concerning what and how it observes, measures, composes, collects, and transmits observations.
Reducing and compressing measurements is critical, which is much helped by representations extracted from the environment via hierarchical boundary determination. When this is not done automatically, what happens then is that the custom dashboard capabilities of the Observability solution need to be used to reconstruct some form of structure that mirrors the boundaries all but lost in the data fog. Naturally, this is extremely costly and inefficient for an organization.
The overemphasis on data instead of signals and states has created a great fog. This data fog leads to many organizations losing their way and overindulging in data exploration instead of exploiting acquired knowledge and understanding. This has come about with the community still somewhat unconcerned with a steering process such as monitoring or cybernetics.
There are many perspectives one could take in considering the observability and monitoring of software services and systems of services, but here below are a few perspectives, stacked in layers, that would be included.
Observability is effectively a process of tracking change. At the level of a measurement device, software or hardware-based, change is the difference in the value of two observations taken at distinct points in time. This change detection via differencing is sometimes called static or happened change. Observability is all about happenings.
Once upon a time, there was a period in the world where humans watched over applications and services by proxy via dashboards housed on multiple screens hoisted in front of them – a typical mission control center. The interaction between humans and machines was relatively static and straightforward, like the environment and systems enclosed.
Substrates changed everything by introducing the concept of a Circuit consisting of multiple Conduits fed by Instruments that allowed Observers to subscribe to Events and, in processing such Events, generate further Events by way of calling into another Instrument. But with the introduction of Percept and Adjunct, it is now possible for Observers attached to Circuit and its locally registered Sources to process Events that have come from a far-off Circuit within another process.
With the latest update to the Substrates API, the metamorphosis to a general-purpose event-driven data flow library interface supporting the capture, collection, communication, conversion, and compression of perceptual data through a network of circuits and conduits has begun.
It is time for new direction closer aligned to goals, focused more on the dynamics of systems that humans are already highly adapted to with their social intelligence, within which situation is a crucial conceptual element of the cognitive model. Understanding and appropriately responding to different social situations is fundamental to social cognition and effective interpersonal interactions.
Disruptions are one factor affecting the maintenance of service quality levels. A disruption is an interruption in the flow of (work) items through a network that can, for a while, make it inoperable or where the network flow performance is subpar. Depending on the severity of the disruption, a network may need to replan and restructure itself for a period afterward. There are two main categories of disruptions: disturbance and deviation.
The low-level data captured in volume by observability instruments has closed our eyes to salient change. We’ve built a giant wall of white noise. The human mind’s perception and prediction capabilities evolved to detect significant changes to our survival. Observability has no steering mechanism to guide effective and efficient measurement, modeling, and memory processes. Companies are gorging on ever-growing mounds of observability data collected that should be of secondary concern.
The Recognition-Primed Decision (RPD) model asserts that individuals assess the situation, generate a plausible course of action (CoA), and then evaluate it using mental simulation. The authors claim that decision-making is primed by recognizing the situation and not entirely determined by recognition. The model contradicts the common thinking that individuals employ an analytical model in complex time-critical operational contexts.
The OODA loop emphasizes two critical environmental factors – time constraints and information uncertainty. The time factor is addressed by executing through the loop as fast as possible. Information uncertainty is tackled by acting accurately. The model’s typical presentation is popular because it closes the loop between sensing (observe and orient) and acting (decide and act).
Data and information are not surrogates for a model. Likewise, a model is not a dashboard built lazily and naively on top of a lake of data and information. A dashboard and many metrics, traces, and logs that come with it are not what constitutes a situation. A situation is formed and shaped by changing signals and states of structures and processes within an environment of nested contexts (observation points of assessment) – past, present, and predicted.
VPA is a technique used by researchers across many domains, including psychology, engineering, and architecture. The basic idea is that during a task, such as solving a problem, a subject will concurrently verbalize, think aloud, what is resident in their working memory – what they are thinking during the doing. Using Protocol Analysis, researchers can elicit the cognitive processes from start to completion of a task. After further processing, the information captured is analyzed to provide insights that can improve performance.
The next generation of Observability technologies and tooling will most likely take two distinctly different trajectories from the ever-faltering middle ground that distributed tracing and event logging currently represent. The first trajectory, the high-value road, will introduce new techniques and models to address complex and coordinated system dynamics in a collective social context rebuilding a proper foundation geared to aiding both humans and artificial agents.
When designing observability and controllability interfaces for systems of services, or any system, it is essential to consider how it connects the operator to the operational domain regarding the information content, structure, and visual forms. What representation is most effective in the immediate grounding of an operator within a situation?
Because of limited processing resource capacities, brains focus more on some signals than others – signals compete for the brain’s attention. This internal competition is partially under the bottom-up influence of a sensory stimuli model and somewhat under the top-down control of other mental states, including goals – this is very similar to how situational awareness is theorized to operate optimally.
Unfortunately, many of the solutions promoted in the Observability space, such as distributed tracing, metrics, and logging, have not offered a suitable mental model in any form whatsoever. The level of situation awareness is still sorely lacking in most teams, who appear to be permanently stalled at ground zero and overtly preoccupied with data and details.
Looking back over 20 years of building application performance monitoring and management tooling, little has changed, though today’s tooling does collect more data from far more data sources. But effectiveness and efficiency have not improved; it could be argued that both have regressed.
Science and technology have made it possible to observe the motion of atoms, but humans don’t actively watch such atomical movements in their navigation of physical spaces. Our perception, attention, and cognition have evolutionary scaled to an effective model for us in most situations. Distributed tracing spans, and the data items attached, are the atoms of observability.
Two distinct hemispheres seem to form within the application monitoring and observability space – one dominated by measurement, data collection, and decomposition, the other by meaning, system dynamics, and (re)construction of the whole.
The underlying observability model is the primary reason for distributed tracing, metrics, and event logging failing to deliver much-needed capabilities and benefits to systems engineering teams. There is no natural or inherent way to transform and scale such observability data collection analysis to generate signals and inferring states.
Humanism is a philosophical stance at the heart of what Humainary aims to bring to service management operations. It runs counter to the misguided trend of wanton and wasteful extensive data collection so heavily touted by those focused on selling a service rather than solving a problem, now and in the future.
As computing and complexity scaled up, the models and methods should have reduced and simplified the communication and control surface area between man and machine. Instead, monitoring (passive) and management (reactive) solutions have lazily reflected the complexity’s nature at a level devoid of simplicity and significance but instead polluted with noise.
There are at least two distinct paths to the future of observability. One path that would continue increasing the volume of collected data in its attempt to reconstruct reality in high-definition on a single plane with little consideration for effectiveness or efficiencies. Another would focus on seeing the big picture in near-real-time from the perspective of human or artificial agents.
We propose a model which can better serve site engineering reliability and service operations by being foundational to developing situational awareness capabilities and system resilience capacities, particularly adaptability and experimentation, as in dynamic configuration and chaos engineering.
The Double Cone Model is a valuable conceptualization in thinking about more efficient and effective methods to handle data overload and generate far more actionable insight from a model much closer to how the human mind reasons about physical and social spaces.
All points of experience within a topology offer some visibility, but the language (codes, syntax) and model (concepts) employed can differ greatly. This is problematic when the goal is to determine the intent and outcome of an interaction’s operation(s).
Today’s data, such as logs, traces, and metrics, are too far removed to be the basis for a language and model that illuminates the dynamic nature of service interaction and system stability inference and state prediction formed across distributed agents.
Observability is purposefully seeing a system in terms of operations and outcomes. In control theory, this is sometimes simplified to monitoring inputs and outputs, with the comparative prediction of the output from input, possibly factoring in history.
It could be argued that no one fully understands what AIOps pertains to now in its aspirational rise within the IT management industry and community. AIOps is a moving target and a term hijacked by Observability vendor marketing. It’s hard to pin down.
In interpreting a script or a scene within a movie, humans must identify the setting and actors and understand the dialog from the multi-sensory feed flowing into the brain. Observability is somewhat similar, except that solutions today have not had a billion years to evolve efficient and effective ways of detecting the salient features.
Context is crucial when it comes to the Observability of systems. But Context is an abstract term that is hard to pin down. Does it represent structure as in the configuration of software components? Does it represent behavior as in tracing a service request? Does it represent some attributes associated with a metric? Does it encompass purpose?
Many software systems have self-regulation routines that must be scheduled regularly. Observability libraries and toolkits are no different in this regard, with the sampling of metric values or resource states being notable examples; another less common one would be the status of inflight workflow constructs.
With an event-driven architectural approach as Substrates, whenever a value needs to be calculated from a series of events, a stateful consumer invariably must use another circuit component to continuously publish the result after each event processing. But there is an alternative option in the Substrates API.
The Substrates API has two basic categories of Instrument interfaces. The first category includes Instrument interfaces that offer a direct means of interaction by a caller. The second type of Instrument is one with no direct means of triggering Event publishing in the interface.
Today the approach to observability at various stages within the data pipeline, from Application to Console, has been to create different models in terms of concepts, structure, features, and values. But what if the model employed were the same across all stages in a data pipeline?
The typical flow of execution for an observability Instrument is for instrumentation within the application code to make a single call to a method defined in the Instrument interface. But there are cases where a single call into an Instrument interface causes the dispatching of multiple events.
A significant challenge in building observability agents or heavily instrumented applications or frameworks is in scaling, both up and down, the resources consumed. There is a trade-off here in terms of time and cost.
