Current
Today the approach to observability at various stages within the data pipeline, from Application to Console, has been to create different models in terms of concepts, structure, features, and values. There is a model within the Application that is focused on Instruments and Events. There is another model for the Collector (Agent) that manages Telemetry and Messages as it acts as a buffer and intermediary between one or more Endpoints. At the Endpoint, there is yet another for Analytics involving Records and Relationships. Finally, the Console, usually within a Browser, offers another model consisting of Dashboards and Charts.
This would not be failing if each of the different models offered useful higher-order abstractions, but that is not the case, for the most part; in nearly all projects and products used today, the same concepts, such as log, trace, metric, are ever present at each stage, but yet always accessible, or not, in a different manner and method, with different tools, techniques, and sometimes terminology. A Trace Event becomes a Trace Message, a Trace Message becomes a Trace Record, and a Trace Record becomes a Trace.
The same concept can be seen in many different representations, resulting in little code reuse across the stages and components involved, even if the same programming language and runtime were involved. It is impossible to take some code currently running in the Analytics Endpoint and move it closer to the Application to reduce wasteful data transmission and shorten the time for situation classification and problem identification. There is no justification for this unfortunate predicament that systems and data engineer teams face other than a lack of a strategic vision beyond yesteryear technologies long past their sell-by-date.
Future
What if the model employed were the same across all stages in a data pipeline? What if such a unifying model was based on the movement of Events over Circuits, Conduits, and Instruments? What if a Cartridge, a plugin into a Circuit, could be deployed into whichever stage in the pipeline a Systems Engineer felt necessary to meet performance requirements and address capacity constraints? What if an Instrumentation Engineer’s programming model was the same for a Data Engineer and a UX Developer? What if a User could run simulations of one or more Circuits locally and remotely fed with past episodic memories or probable future projections? This is the grand vision behind Substrates. Only then can the focus shift to higher-order concepts such as Situations, where a Situation can pertain to a single Application or multiple Applications connected to the same Collector or Analytics Endpoint. An exciting development that would change so much thinking and practice while opening up tremendous possibilities.