At the start of the design process, metaphors are crucial in shaping a project’s direction. They provide a conceptual framework for understanding the design challenge and can help team members with diverse backgrounds align their understanding. Metaphors also bring new perspectives and innovative solutions. However, if the metaphors no longer fit a project or new insights arise, they may need to be discarded.

Could the selection of metaphors be the reason for the numerous unsuccessful endeavors in fulfilling the ambitious commitments made by the observability community, by both vendors and customers?

Pillars and Pipelines

The concept of pillars and pipelines is commonly used in marketing materials and technical books related to observability. Pillars represent a foundation of fragmented data, including logs, metrics, and traces. Pipelines emphasize the idea of data collection flowing in a single direction for each type of siloed data.

Such metaphors have limited our understanding of monitored and managed computational systems. Pillars promote narrow thinking, while pipelines focus on data movement instead of situational awareness. These oversimplifications fail to capture the complexity of these systems and their interactions with other domains. Extracting meaningful insights from the abundance of collected data is challenging. Teams are overwhelmed and struggle to identify critical signals. Integration of sensory information is pushed downstream with insufficient context and time delays, hindering its usefulness for control purposes.

Holistic and Humane

There is a pressing need for new holistic paradigms to address systems’ complexity better. The emphasis on gathering and examining data retrospectively has resulted in a reactive approach. Today’s tools are more proficient at identifying problems after they occur instead of proactively preventing them.

We need to use metaphors that encourage integrated systems thinking to improve observability. This means creating new frameworks, models, and tools that combine instrumentation, measurement, and control. Instead of only collecting data, we should prioritize meaningful insights and remediating actions.

Thinking in terms of pipelines and pillars has influenced the observability industry but has limited the tools to being data collectors. While data collection is important, its value lies in generating useful courses of action. By recognizing this and shifting towards solutions that are more integrated, context-aware, and proactive, the industry can provide more value. This evolution will enable organizations to better understand and manage their systems, ultimately enhancing overall reliability and performance.