The Illusion of Pillars
You can be pretty sure that when someone mentions pillars in discussing what Observability is, you are more than likely conversing with a vendor sales engineer or an engineer who has only ever listened to vendor noise and failed to think critically about the nature of monitoring and management. Fortunately, pillars have fallen flat on their face of late. Sadly, pipelines are the new pillars. All a misdirection from the fact that practically all Observability and Application Performance Monitoring (APM) vendors are not delivering value (more than a 75% failure rate), just more noisy data, computation overhead, network traffic consumption, storage space, dashboard widgets, and operator cognitive overload.
Do not try to bend the spoon; that is impossible.
Instead, only try to realize the truth. There are no pillars.
What you consider a pillar is merely a checkbox in some vendors’ product factsheet.
A Misdirection of Goals
Talk of pillars and pipelines is not a simplification. There is no significance to such conceptual wish-wash. This is an attempt to stupefy the market and community. The goals of improved situation awareness, effective decision making (intelligence), and efficient systems steering (controllability) are being put aside or entirely ignored in keeping every site reliability engineer chattering on about and being buried under large volumes of data and details (logs).
Busy with Data Doings
Observability and Application Performance Monitoring (APM) vendors keep engineering teams busy doing with data instead of making past, present, and projected sense of the state (status) of systems under concern. Today’s vanity dashboards, offered up by Grafana and Prometheus, only serve to have engineers attend to styled data detail instead of engaging them with meaningful models and visual representations consisting of settings, scenes, situations, states, and spaces of projection (scenarios). The deluge of data and detail is degrading engineering efforts to perceive, understand, and reason about behaviors and states.
Nearly every site reliability engineer (SRE) today considers their systems mission-critical. You would typically associate this classification with medical solutions, military operations, and now self-driving vehicles. Yet within these domains, the awareness of the situation and states, both present and projected, is ubiquitous with models of reality that require an immediate intelligent response. Instead, the Observability, SRE, and DevOps communities keep talking up pipelines (streams), pillars (data types), packets (filters), platforms (dashboards), and pixels (sparklines).
Automobile manufacturers will need to understand both how to design driver-vehicle interfaces that do not overload the driver at lower levels of automation, bring the driver back into the loop as quickly as possible at the higher levels of automation…
Situation awareness (SA), as a driver’s integrated understanding of what is happening in the driving environment, is critical for successful performance, with poor SA being implicated as a significant cause of vehicle crashes.
Handbook of Human Factors for Automated, Connected, and Intelligent Vehicles