Organizations are drowning in an ocean of data. Every click, interaction, and transaction generates quantitative data points, creating vast repositories of numbers that promise insights but often deliver confusion. The challenge isn’t just the volume of data – it’s the fundamental question of how to transform these raw metrics into meaningful qualitative understanding. While modern systems excel at collecting and storing quantitative measurements, they struggle with the more nuanced task of deriving genuine meaning and status from such.
This post introduces a framework designed to close that gap. By employing a universal vocabulary of signs and signals, it turns data into actionable service insights. This approach guides organizations through the progression from data collection to meaningful interpretation, enabling more informed decision-making in a complex world.
By rethinking how data is defined and analyzed, the framework offers a scalable, cross-disciplinary way to achieve shared understanding across domains. It mixes principles of semiotics and systems thinking with practical applications for service monitoring and operations, offering a clear, actionable path from observation to insight.
Our journey in this series of posts begins with managing the flood of incoming data – organizing, structuring, and preparing it for analysis. From there, we must navigate through several critical transformations: from simple cataloging to nuanced classification, from discrete signals to collective consensus, and ultimately to the emergence of widely recognized status. The progression isn’t merely technical; it represents a fundamental shift in understanding and interpreting information. Each stage builds upon the previous one, adding layers of meaning and context that gradually transform raw numbers into rich, qualitative insights. This transformation involves not just computational processes, but also social and cognitive elements that help shape our understanding of what the data truly means in a broader context.
From Metrics to Signs
Within the vast ocean of metrics, a remarkable pattern emerges – the complex interactions between services can be distilled into sixteen fundamental signs. These signs form a universal language for describing service behavior, like how DNA’s four bases combine to encode all biological life. This standardization represents a crucial step from quantitative metrics to qualitative understanding. Think of these signs as the missing service management lexicon. These signs naturally cluster into several groups of actions and happenings.
Operational Signs: START and STOP represent the basic temporal boundaries of any service interaction. These signs bookend every service execution, providing critical context for all other signs between them. CALL extends this lifecycle pattern into inter-service communication, creating spans as services interact with one another. SUCCEED and FAIL represent the binary outcomes that ultimately matter for service health and reliability. However, understanding them requires context from other signs – a FAIL followed by a RETRY then SUCCEED tells a different story than an immediate SUCCEED.
Conditional Signs: RETRY and REDIRECT represent tactical responses to environmental conditions. These signs indicate a service’s ability to adapt its behavior. RETRY shows persistence in the face of failure, while REDIRECT demonstrates flexibility in routing and processing paths. RECOURSE also fits here, showing how services can adapt by returning to alternative processing modes. REJECT, DROP, and DELAY form a cluster related to resource constraint handling. REJECT indicates an active decision to refuse work, DROP shows forced request shedding under pressure, and DELAY represents a compromise between immediate rejection and immediate processing.
Provisional Signs: SCHEDULE, SUSPEND, and RESUME form a sophisticated set of temporal control mechanisms. While START and STOP deal with the basic lifecycle, these signs represent intentional manipulation of execution timing. A service might SCHEDULE work for optimal resource utilization, SUSPEND execution when conditions aren’t favorable, and RESUME when circumstances improve. This sign set reveals how services manage work over time.
Exceptional Signs: ELAPSE and DISCONNECT signify breaches of essential operational constraints. ELAPSE highlights the failure to meet time-based boundaries, such as deadlines or latency requirements, while DISCONNECT points to the loss of network connectivity, a foundational element in distributed systems. These signs are pivotal for understanding system robustness and reliability, as they indicate breakdowns in the foundational assumptions of time and connectivity that underpin service interactions.
These groupings reveal not just what signs mean individually, but how they relate to each other and combine to describe complex service behaviors. For instance, a service might START, encounter a DISCONNECT, perform multiple RETRY operations, and ultimately SUCCEED – telling a story of resilient recovery from network issues.
Signs as Tokens
A sign effectively serves as a token—a fundamental unit of meaning. Through the sequential arrangement and interaction of these tokens, complexity and cognition emerge. Similarly, words combine to form sentences, paragraphs, and narratives, conveying nuanced ideas. Signs, in turn, construct sequences that enable systems to express their intentions, respond to alterations, and adapt intelligently to novel environments. By treating signs as tokens in a system language, we transcend mere measurement and gain insights into the underlying system intent and behavior. This approach enables us to discern patterns indicative of impending issues, recognize successful adaptation strategies, and enhance our comprehension of how our systems process and respond to evolving circumstances. This linguistic approach to observability provides us with a more extensive vocabulary for describing and comprehending system behavior, which in turn enables more sophisticated monitoring.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 |
/// A [Sign] classifies operations, transitions, and outcomes that occur during service request /// execution and inter-service calling, such classifications enable analysis of service behavior. enum Sign { /// Signals the initiation of a service execution. /// This is the first event in a service's lifecycle. START, /// Signals the completion of a service execution, regardless of its outcome. /// This is typically the final signal in a service's interaction. STOP, /// Signals a service operation representing either: /// - Outbound: A caller initiating a request to another service /// - Inbound: A service receiving a request from a caller CALL, /// Signals that a service execution or inter-service call completed /// successfully, meeting all expected criteria. SUCCEED, /// Signals that a service execution or inter-service call encountered /// an error condition and did not complete as expected. FAIL, /// Signals activation of a degraded service mode after a failure, such as /// - Serving cached data instead of live data /// - Routing to a backup service /// - Using a simplified alternative implementation RECOURSE, /// Signals that a service request was forwarded to a different endpoint /// or handler than originally targeted. REDIRECT, /// Signals that a service execution or call exceeded its allocated /// time budget and was terminated. ELAPSE, /// Signals an automatic reattempt of a failed service call, typically /// as part of a retry policy with backoff. RETRY, /// Signals that a service actively declined to process a request, /// typically due to policy violations or resource constraints. REJECT, /// Signals that a service request was discarded without processing, /// typically due to overload conditions or circuit breaking. DROP, /// Signals that processing of a request was intentionally /// postponed for later execution. DELAY, /// Signals that a service request has been queued for /// execution at a future time. SCHEDULE, /// Signals that an in-progress service execution was temporarily /// halted but may resume later. SUSPEND, /// Signals that a previously suspended service execution /// has restarted processing. RESUME, /// Signals a connection failure where the client could not /// establish or maintain network connectivity with the service. DISCONNECT, } |
From Signs to Signals
In distributed systems, service interactions create a fascinating pattern of observations and actions that we capture through signals. While signs give us the vocabulary to describe what happened – a call was made, a task started, an operation failed – signals help us understand these events from distinct perspectives.
Consider what happens when one service calls another. This single interaction creates two different but related observations. The caller actively initiates the call, while the receiving service observes this call being made to it. Both services record the same underlying event, but their perspective on it differs fundamentally. This dual nature of service interactions is what our signal system captures through its combination of signs and orientations.
Every signal emittance combines four essential elements: the source (the subject talking), the sign (what is or has happened), the subject (who is it happening to), and the orientation (from which perspective). When a service acts, it emits a signal with a RELEASE orientation – “I (source/subject) am doing this (sign).” When a service observes an action being done to it, it records a signal with a RECEIPT orientation – “I (source) see this (sign) being done by (subject) X.” The source tells us who made the observation and the subject the service the observation is about.
This dual-perspective recording creates natural pairs of signals for each sign. When Service A starts a task, it emits a START signal with RELEASE orientation. Service B, observing this same event, records a STARTED signal with RECEIPT orientation. Both signals refer to the same event and carry Service A as the source, but they capture different perspectives on that event. This pairing helps us understand not just what happened in our system, but how each participant experienced and recorded that event and which participant the sign pertains to.
The power of this approach emerges in how it helps us understand complex interactions. For every sign in our vocabulary – whether it’s CALL, SUCCEED, FAIL, or any other – we can track both the active perspective of the service acting and the passive perspective of the service observing it. This dual recording creates a rich tapestry of information that helps us better understand, monitor, and debug our distributed systems.
By capturing both perspectives on every interaction, we create a more complete and accurate picture of system behavior. We can cross-reference the RELEASE signals from one service with the corresponding RECEIPT signals from another, helping us validate that events were both sent and received as expected. This dual recording also helps us identify issues where one service’s view of interaction differs from another’s, providing crucial information for debugging and system understanding.
Given the sixteen signs and two orientations, there are thirty-two signals. When we name signals, we follow a simple yet powerful convention that reflects their orientation. Every sign (like START, CALL, or FAIL) generates two signal names based on whether it’s being released or received. For the RELEASE orientation, we use the sign name directly. For the RECEIPT orientation, we add “ed” to create the passive form of the verb.
This pattern creates natural pairs of signal names that tell the complete story of each interaction. When Service A starts an operation, it emits a CALL signal (RELEASE orientation). Service B, observing this same event, emits it as a CALLED signal (RECEIPT orientation). The “ed” suffix captures the passive nature of received signals.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
/// The `Orientation` enum classifies the method of signal recording against a service. /// /// There are two ways a `Sign` can be recorded against a `Service`: /// /// - `RELEASE`: Indicates the projection (release) of a signal from the self-perspective. /// The service is informing other observers (services) of an operation or outcome. /// - `RECEIPT`: Indicates the acknowledgment (receipt) of a signal observed within some message response or event notification. /// Receipt of a signal should be taken as being generated in the past, whereas release is in the present. /// /// This classification helps to understand the context and timing of signals in service interactions. /// Receipt of a signal should be taken as being generated in the past, whereas release is in the present. enum Orientation { /// The projection (release) of a signal. RELEASE, /// The perception (receipt) of a signal. RECEIPT, } |
Next Steps
The journey from raw metrics to qualitative understanding doesn’t conclude with signals. While signals provide a comprehensive vocabulary and a dual-perspective recording of service interactions, they represent isolated moments in time. The next challenge lies in integrating these individual observations into meaningful service health and behavior judgments.
Think of signals as words in a conversation. Just as individual words combine to form sentences and conversations that convey meaning, sequences of signals tell us stories about service behavior. A service that emits a pattern of CALL → FAIL → RETRY → SUCCEED signals conveys a different story than one showing CALL → FAIL → DROP. The first pattern suggests resilience and eventual success, while the second indicates a more serious problem.
This elevation from metrics to meaning transforms observability from a practice of measurement to one of understanding. By treating system behavior as a language to be understood rather than just quantities to be measured, we gain unprecedented insight into how our systems think, adapt, and evolve. In essence, signs and their sequences provide a grammar of system behavior – a way to understand not just what systems do, but how they think. This linguistic model of observability opens new frontiers in system design, monitoring, and optimization, enabling us to build more resilient, adaptive, and intelligent distributed systems.
In our next installments, we’ll explore how we move from these signal sequences to status judgments through several key transformations:
Signal Streaming and Sequencing: We’ll examine how signals flow through our system over time, creating temporal patterns that reveal service behavior trends. Just as a film creates motion from still frames, our system will derive meaning from sequences of signals.
Individual Status Formation: We’ll explore how individual services form judgments about their health and capability based on the signals they emit and receive. This process mirrors how individuals in social networks form self-assessments based on their interactions and experiences.
Collective Status Emergence: Perhaps most fascinating, we’ll see how networks of services develop collective understanding through shared signal patterns. When multiple services observe similar patterns of interaction with a particular service, a consensus emerges about that service’s status – much like how reputation forms in social networks.
Status Propagation and Adaptation: Finally, we’ll look at how these status judgments flow through the service network, influencing future interactions and helping services adapt their behavior based on collective intelligence.
This passage from signals to status represents a crucial step from quantitative metrics to qualitative understanding. While signals tell us what occurred, status tells us what it means. This meaning-making process transforms our service network from a collection of independent actors into a cognitive system capable of collective understanding and adaptation. The implementation of this transformation involves sophisticated patterns for signal processing, state management, and distributed consensus. We’ll explore concrete examples of how to build these systems while maintaining the clarity and elegance of our signal-based foundation.