A Story of Observability

This article was first published in 2019.

The Descriptive Period

Once upon a time, there was a period in the world where humans watched over applications and services by proxy via dashboards housed on multiple screens hoisted in front of them – a typical mission control center. The interaction between humans and machines was relatively static and straightforward, like the environment and systems enclosed. The machines churned away on the data, and the humans turned their gaze to the peaks and the troughs thrown up glaringly and garishly into their eyesight from the streams of sparklines and status widgets.

The Explorative Period

But then the world changed. The environment became much more complex and altered quicker than could be supervised by those responsible for maintaining the dashboards as well as those who attended the metrics projected onto such flat panes. It then became necessary to peer below the surface of the shiny widgets to dig further into the vast depths of the data dungeons. Many humans lost their way along this journey that took them through a boundaryless “unknown unknowns” forest region covered by a dense data fog; even with their tagging and labeling of traced paths, it was a treacherous endeavor. Now relegated to a mere librarian role, the machines could not assist humans, wholly detached from an expedition’s purpose and intent. Machines instead watched over the humans and tried their best to anticipate their future data requests by indexing.

The Attentive Period

With costs mounting and resources near depleted, the humans turned away from their overindulgence with ad hoc data exploration. Humans had lost sight of the big picture and, in doing so, neglected the operational aspects of their work. With the environment continuing to grow in complexity unabated, it was clear that a new way of thinking and acting was required that reflected the fundamental nature of the beast that had developed and shown itself while they were off exploring. Turning to the machines again, the humans programmed them to reason and communicate more human-like with cognitive capacities reflected in responsibilities and dialogs shared between both. Now, the machines only signaled significance, guiding humans along a signposted path of simplified state markers, always oriented to the current situation and offering contextual assistance.

The Cooperative Period

For a period, there was saneness and system stability, with humans being able to assess the present situation more accurately and in near real-time. But they knew that the beast of complexity would eventually return with the steady passing of time and progression of change. They had won a battle but not the war. They needed to train their armies, but instructing the machines on how best to classify the significance of events proved increasingly cumbersome and was always in retrospect of what was previously noted. The communication barrier between machines and humans must be eliminated to allow a much richer sharing of knowledge, experience, wisdom, and perspectives. The machines needed to see how a human observes and classifies the system’s past and present memories within the machines themselves. Finally, a mirrored simulated universe was designed, unifying machines and humans, and for a long time, progress continued.