I doubt any observability engineer performs work close to what observability should represent. Most of those holding this title do not actually increase observability; instead, they build the machinery they expect will eventually produce it.
They instrument services. They deploy agents. They configure collectors. They operate clusters. They manage storage. They tune retention. They build dashboards. They define alerts. But none of that is observability.
It’s a little like confusing the construction of a telescope with astronomy.
Building a better telescope doesn’t automatically mean you understand the stars.
The industry quietly made a decision years ago that observability would emerge if only enough data were collected. So we started from the bottom of the stack: logs, metrics, traces, events, collectors, exporters, databases, dashboards.
The assumption was simple. Collect everything first. Understanding will come. The trouble is that “later” never arrives.
Teams become consumed by the mechanics of collection. Every new service requires instrumentation. Every deployment changes telemetry. Every dashboard needs maintenance. Every alert needs tuning. Storage costs grow. Cardinality explodes. New pipelines appear. More telemetry is generated to manage the telemetry already generated.
Work expands to fill available time: an engineer tasked with observability ends up maintaining telemetry infrastructure, becoming an expert in evidence collection rather than system understanding.
Many observability engineers function merely as advocates for the field. They articulate its underlying philosophy and describe the distinction between monitoring and observability. They understand collectors, exporters, and backends.
None ever arrive at the destination their observability tools were meant to enable. It’s a bit like talking endlessly about swimming while never getting into the water. The conversation becomes the activity. The thing itself never happens.
I don’t think this is the fault of individual engineers. They’re following the path the industry laid out for them. The entire ecosystem teaches observability as a bottom-up exercise: first collect, then store, then search, then visualize, then maybe derive understanding. But understanding doesn’t emerge simply because enough records exist.
Observability isn’t a warehouse full of telemetry. It is the ability to infer the current state of a system from the evidence available. That definition immediately changes where you begin.
- Instead of asking, “What data can we collect?” you ask, “What states must we be able to distinguish?”
- Instead of starting with exporters, you start with system behavior.
- Instead of asking what telemetry exists, you ask what questions the system should always be able to answer.
That is a top-down problem. Collection becomes an implementation detail rather than the objective.
Genuine observability work is rare. I have collaborated with numerous vendors in this field, yet I have not encountered an engineer doing observability within those groups. They know about telemetry and dashboarding but not much else.
When you do encounter it, it’s often outside the commercial observability industry altogether—in cyber-physical systems, aerospace, industrial control, robotics, or other domains where understanding system state isn’t a nice-to-have but the entire purpose of the engineering. Those disciplines begin with the model of the system.
The software industry largely began with the logs. And that single decision may explain why, after all these years, we are still remarkably good at collecting data, yet surprisingly poor at knowing what the situations arising within our systems.
The field sold instrumentation as observability. Engineers followed the incentives. The result is sophisticated monitoring infrastructure that still leaves humans staring at graphs trying to reverse-engineer system behavior.
Reversing it means treating observability as a modeling and inference discipline first, with data collection as the servant, not the master. Until then, most “Observability Engineers” will keep building better telescopes while the stars remain mysterious. The good news: the critique is spreading. The bad news: institutional inertia is strong.
