An organization can adopt OpenTelemetry (OTel): every service emits spans, every host reports metrics, and data flows perfectly through the collector into a store. The pipelines are healthy, and the dashboards are full. Yet, if you ask whether the system itself is healthy, the data alone cannot answer. Data records events. Interpreting their meaning (like latency shifts, degradation, or needed action) resides entirely in someone’s mind, or nowhere at all.
This gap is what the word “observability” conceals.
Observability is a property of a system: it means a system’s internal state can be inferred from its outputs. OTel transmits data records, but stops there. The vital work of inference, situational intelligence, is left unaddressed.
OTel is Just a Delivery Service
The layer at which OpenTelemetry standardizes is simply the layer at which telemetry travels. It offers a transport contract: components agree on data batches and basic lifecycles (start/consume/shutdown).
|
1 2 3 4 5 6 |
type Traces interface { component.Component consumer.Traces } |
The contract is honored at the seams. A batch arrives at a processor, and a batch departs to the next consumer. But the actual work between arrival and departure is opaque. Ownership rules dictate memory safety during handoffs, but they are completely silent on what computation the processor actually performs.
The Baggage of the OTel Collector
OTel made a distinct architectural decision: instrumentation runs inside the application, but processing runs in a separate Collector process. This decision wasn’t driven by architecture; it traces back to two operational origins:
The Serverless Origin: Early serverless functions held no state across invocations. A separate, longer-living process was needed to hold the state the function could not.
The Vendor Agent Lineage: Observability vendors historically used separate agents so they could ship updates independently of their customers’ deployment cycles. The OTel Collector inherited this vendor-driven shape.
By standardizing the wire format but leaving the pipeline undefined, OTel commoditized the transport layer while letting vendors retain control over the inference layer, allowing them to feign sensemaking and charge for collection.
The Missing Layer: Temporal Processing
Because OTel lacks a vocabulary for temporal, compositional processing, building a transformer to detect latency shifts or maintain rolling windows requires specialized skills. Teams are forced to use basic processor callbacks to manually write algorithms. The result? We have an industry where telemetry is universal, but interpretation is rare.
We can close this gap by moving the standardization boundary down a level, creating a framework where teams can assemble inference from shared, composable operators. Instead of OTel handing you an empty place to run code:
|
1 2 3 4 5 6 7 |
Processor { // here is a telemetry batch; // do something; // call next; } |
A framework for the inferential layer hands a language to compose computation:
|
1 2 3 4 5 6 7 8 |
Flow<Input, Output> { // compose a typed sequence of defined operations; // materialize its state per attachment; // attach it to any compatible pipe; // emit the derived result into the topology; } |
In this framework, Fiber<E> is a reusable, type-preserving processing recipe, and Flow<I,O> is a type-changing composition (map, stateful scan, rolling window). The unit of agreement is the flow (of emissions), not the batch.
Advantages of AN Inference Framework
Transitioning from a black-box processor to a framework of named operators changes three critical things:
The type boundary: An OTel processor consumes traces and emits traces. A Flow admits arbitrary input and output, allowing the value to physically change in flight. The circulation process is: stimulus → sign → status → situation.
|
1 2 3 4 5 6 |
Flow<Span, Latency> Flow<Latency, Window<Latency>> Flow<Window<Latency>, Stability> Flow<Stability, Condition> |
Reuse: An OTel algorithm is invisible to the processor next to it. A Fiber is a reusable recipe. Flows compose end-to-end, meaning the transformer one team builds can be imported by another. This turns specialist work into shared work.
Reasoning: OTel can only reason about components. It cannot reason about the computation itself. A framework whose compositions and state are part of the specification finally allows us to reason about the system’s logic.
A system is observable when its state can be inferred from what it emits. OTel delivers what the system emits. A framework that delivers the inference is the one that actually earns the word.