The current attention that is given to DORA metrics (deployment frequency, lead time for changes, change failure rate, and time to restore) seems to be a repeat of much of the simplistic thinking that came out of the DevOps movement in particular golden signals and service level agreements in the context of site reliability engineering.
This need to boil all engineering management problems down to a few metrics seems never to diminish. It could be argued that we have replaced the KISS principle with KISSES – keep it stupidly simplistic engineering systems.
As engineering systems grow ever more complex, the engineering community’s focus on simplistic measurement and reporting hinders achieving operational scalability by way of sensemaking and adaptive steering of such systems.
A thorough grasp of the interconnectedness of the situation is crucial. This understanding lays the foundation for implementing adaptive governance – policies that adjust and autonomous processes that self-manage. Intelligence is not to be found in metrics. It can only be obtained when action is appropriate to the overarching context.
We confuse simplistic thinking with abstraction, wherein complex phenomena and processes are reduced to rudimentary binary answers of true or false, up or down, above or below, and before or after.
While the allure of simplistic solutions is understandable, especially when dealing with complex socio-technical systems, prioritizing them can often come at the expense of long-term effectiveness. Nuance in these systems requires a thoughtful approach that balances ease of implementation with robust and sustainable outcomes. This avoids the pitfalls of short-termism and fosters a focus on long-term gains, particularly organizational learning.
Reviewing the experience of software development over the past three decades reveals a deeper issue. The aim of climbing abstraction layers keeps getting disrupted by fundamental changes requiring revisiting of lower levels.
The constant maintenance within communication and connection layers devours a significant amount of engineering effort. Whenever new conceptualizations are offered, there’s often either a reluctance to embrace them or an inability to integrate them due to existing communication structures. The breakthroughs that manage to rise to the coordination and cooperation layers are usually extremely narrow in scope and highly specialized. An example here would be intelligent networking, while from the application perspective it represents a communication layer, its conceptual capabilities within its domain place it up above the coordination and cooperation layers.
For many software and systems engineering teams, the breakdown of the design and development effort volume looks like the above. Most of the workload of humans and machines is spent on maintaining connections between systems and services and managing communication over various channels afforded by such connections. An example of this would be the state of play in the observability space, where today most customers and vendors are still spending an enormous amount of effort moving data around from a growing number of sources and sinks over various pipelines and protocols. Little has been abstracted in over 30 years since distributed tracing was introduced, instead of imagining new conceptualizations of what systems and services are and how they should be sensed and steered the focus remains on the best means of connecting and communicating measurement data. Whenever there is an attempt to find the signal lost to noise, such as with the introduction of service cognition, the community has steadfastly remained at the lowest layer by relabeling collected data as signals. AIOps is a pipe(line) dream.
Can it be fixed? Maybe, but if it were, the redistribution of work would need to look more like the following. Perhaps this is where accelerated adoption of automation, self-adaptive systems, and AI are likely to lead us. Design and development will be a dialog with the machine at the level of normal social-human discourse in terms of past and present situations, possible scripts, and probable scenarios that bring us to a projected situation.
While DORA metrics offer a starting point for measuring engineering effectiveness, they lack context and ultimately prove insufficient. A more critical question emerges: how do we determine the level of abstraction (layer) within which a design operates? The answer lies in understanding the domain’s specific context, which is where intelligent analysis comes in. However, some universal concepts can serve as signposts for different layers. For instance, plans, policies, and promises often indicate a design that facilitates cooperation. Meanwhile, situations, simulations, patterns, and predictions typically point towards the design operating at the cognitive layer.
A practical approach involves listing all the key concepts a design introduces and then mapping them to the appropriate layer. This is demonstrated by the layer mapping of interfaces within the Humainary toolkit.
