This article was first published in 2014

The following graphic has been used to frame various software performance optimization techniques. It is not a comprehensive inventory of all software performance optimization techniques or concerns. Still, it serves a purpose in managing the amount of effort that, in general, should be spent on each technique outside of extreme cases.

The left side is your typical localized bottom-up approach to speeding up code execution steps. Most developers involved in performance improvement efforts start on the left side because it feels less abstract and much more tangible than system dynamics or software adaptation, both of which require a much higher level of understanding of precisely what happens outside of the code editor. On the right side, the focus is on manipulating or mirroring system execution behavior to globally control, circulate, and conserve costs associated with coordination, contention, congestion, consumption, and coherence—a top-down viewpoint.

The left side addresses local execution efficiency; the right is more global execution intelligence. But in the middle, we have execution heuristics, which should be viewed as the first and fundamental step towards making software intelligence—a smartness independent of a particular benchmark environment. Once ascertained and adequately assessed, performance heuristics should feed back into many other optimization activities on either side.

It is easy to focus efforts on micro-benchmarking, though generally, the improvements reported rarely translate into measurable gains of the same magnitude within a real-world environment far noisier than a micro-benchmark.

It can prove counterproductive when it places more stress elsewhere in the pipeline processing.

A developer can waste enormous time chopping off branches here and there and never acquire a fundamental understanding of the forest they or their applications operate within. Each branch chop can be rewarding but frustrating when one steps back and sees the negligible impact in a much broader context.

Unless the forest holds a few trees it’s far too easy to lose all sense of perspective the longer one stays staring.

Heuristic refers to experience-based techniques for problem-solving, learning, and discovery that find a solution that is not guaranteed optimal but good enough for a given set of goals. Where the exhaustive search is impractical, heuristic methods are used to speed up the process of finding a satisfactory solution via mental shortcuts to ease the cognitive load of making a decision. Examples of this method include using a rule of thumb, an educated guess, an intuitive judgment, stereotyping, or common sense. More precisely, heuristics are strategies using readily accessible, though loosely applicable, information to control the problem-solving in human beings and machines. – Wikipedia

The above definition of a heuristic emphasizes experience codified in rules of thumb and strategies.

However, during countless troubleshooting engagements, it seems developers rely far more on expectation than experience in writing efficient code. This could be because there is still not enough attention given to observing how software machines execute, in the sense of motioned activity, and not some disconnected metric created to feed an illusion of control held within by operational teams in looking over many dashboards.

Looking at many of the case studies by an application performance monitoring vendor, you repeatedly encounter the “surprise” that developers experience when the product solution throws up some data into a dashboard that hints at an underlying performance or scalability issue. This is the big selling point here, but as a supposedly engineering profession, should we be surprised when we learn how our creations execute and service others?

The devotion to our software creations needs to extend further into the lifecycle. One such extension would be to give software memories that can be recalled and perceived by a developer, operator, or even the software itself for offline and online behavior and prediction training. Most software performance visualizations render aggregate data but not the motion of software activity at a level that lends well to how humans process sensory data in the physical world. To acquire the knowledge of suitable software performance heuristics, developers must experience software execution in a new, more modern manner – a simulated environment of episodic machine memory replay.