The apprentice returned the next morning carrying three lamps.
The master said,
Why do you carry three lamps?
The apprentice said,
One is monitoring.
One is diagnostics.
One is observability.
I have been told they are different,
but when I light them,
they all seem to show me trouble.
The master said,
Then set them down.
The apprentice set the lamps on the floor.
The master pointed to the first.
What does this one do?
The apprentice said,
This one watches.
It waits for a value to cross a line.
It tells me when something is too high,
too low, too slow, too many, too few.
The master said,
That is monitoring.
The apprentice said,
Yes.
The master said,
Monitoring is the lamp at the gate.
It does not know the kingdom.
It knows the bell.
When the bell rings, it says,
something has crossed the line we placed here.
The apprentice said,
But that is useful?
The master said,
Useful.
A bell is not a small thing.
A bell wakes the sleeping.
A bell gathers attention.
A bell says, look here.
But do not confuse the bell with sight.
The apprentice looked at the second lamp.
The master said,
And this one.
The apprentice said,
This one follows the path.
It gathers logs, traces, spans,
errors, timings, retries, calls, returns.
It shows what happened in flight.
The master said,
That is diagnostics.
The apprentice said,
It feels like observability.
The master said,
Because it shows movement.
The apprentice said,
Yes. It shows the system doing things.
The master said,
It shows the residue of doing.
It shows the footprints left by work
as it crossed the ground.
With enough footprints,
a careful reader may reconstruct the passage.
The apprentice said,
Is that not observability?
The master said,
No. It is evidence.
The apprentice was quiet.
The master said,
Diagnostics begins after doubt has entered.
Something is wrong, or may be wrong, or was wrong,
and now the reader descends into the evidence
to recover the condition that produced it.
Diagnostics asks:
What happened?
Where? What failed? What waited or retried?
Which path did the work take?
The apprentice said,
These are good questions.
The master said,
They are necessary questions.
But they are questions asked
after the system has already spoken
without enough meaning.
The apprentice said,
Then diagnostics is downstream asking?
The master said,
Often. Not always, but often.
It is the asking carried into the night.
The apprentice looked at the third lamp.
The master said,
And this one.
The apprentice said,
This one is called observability.
The master said,
What does it do?
The apprentice paused.
It lets me infer state.
The master said,
What state?
The apprentice said,
The state inside the system.
The master said,
From what?
The apprentice said,
From what the system emits.
The master said,
And how?
The apprentice paused again.
From the data.
The master shook his head.
Not from the data alone.
The apprentice frowned.
The master said,
A footprint does not tell you whether the traveler was lost,
fleeing, hunting, returning, obeying, trespassing,
or carrying medicine to a child.
The footprint is evidence of passage.
Its meaning comes from a model
of the world in which passage matters.
The apprentice said,
Then observability requires a model?
The master said,
Yes.
The apprentice said,
But diagnostics also has state.
A trace has duration.
A span has status.
A log has an error.
A request has a path.
These are states?
The master said,
They are states of the evidence.
They are states of the work in transit.
They are not yet the state of the system as a whole.
The apprentice said,
Explain that to me.
The master lifted the first lamp.
Monitoring says,
this value crossed this line.
He lifted the second.
Diagnostics says,
this work passed through these places
and left these marks.
He lifted the third.
Observability says,
given what the system emitted,
and given the model by which this system is understood,
this is the condition the system is in.
The apprentice said,
Monitoring perceives.
Diagnostics reconstructs.
Observability infers.
The master said,
Better.
The apprentice said,
But is inference not also reconstruction?
The master said,
It can be.
But reconstruction walks backward through evidence.
Inference uses evidence through a model.
It does not merely ask, what sequence happened.
It asks, what condition must be true for these signs to appear,
and what condition may follow if this continues.
The apprentice said,
That sounds closer to situational awareness.
The master said,
It is on the same road.
Monitoring mostly belongs to perception.
It tells you something has appeared.
Diagnostics belongs to perception
and some comprehension,
but usually after the fact.
It gathers enough evidence
for a human to understand.
Observability begins when the system’s emissions
support comprehension through a model.
It becomes stronger when they also support projection.
The apprentice said,
Projection?
The master said,
What is likely next.
The apprentice said,
Monitoring rarely tells me that.
The master said,
Monitoring tells you the water has reached the mark on the wall.
The apprentice said,
Diagnostics tells me which pipes filled the room.
The master said,
Observability tells you what condition of the house
allowed the flood,
whether the walls can bear it,
which rooms are next,
and what action would change the course.
The apprentice said,
Then much of what we call observability is diagnostics?
The master said,
Much of it is.
The apprentice said,
Why do people confuse them?
The master said,
Because diagnostics feels deep.
It has many details.
It shows hidden passages.
It lets a person descend from symptom to mechanism.
This descent feels like seeing.
The apprentice said,
But it is not seeing?
The master said,
It is reading after the fact.
The apprentice said,
And observability?
The master said,
Observability is not the pile of things you can read.
It is the degree to which the system can be understood
from what it expresses.
The apprentice said,
Expresses, not emits.
The master smiled.
You remember yesterday.
The apprentice said,
Ask what it means before it leaves you.
The master said,
Yes. Emission without meaning
produces diagnostics work.
Expression carries meaning into the world.
The apprentice looked again at the second lamp.
Then what is a trace?
The master said,
A trace is a path record.
The apprentice said
Is it a model?
The master said,
No.
The apprentice said,
But it represents the system?
The master said,
It represents a passage through the system.
That is not the same as representing the system.
The apprentice said,
Because the system is more than the path?
The master said,
The system is boundaries, purposes, capacities,
commitments, dependencies, expectations, constraints,
states, histories, and possible futures.
A trace may show that one service called another.
It does not, by itself, tell you whether the call was wise,
expected, dangerous, degraded, compensating, futile,
late, harmful, or part of a larger drift.
The apprentice said,
Unless those meanings were marked into it.
The master said,
Yes. Then the trace becomes a carrier of signs,
not merely a cord of timings.
The apprentice said,
So the representation is thin.
The master said,
A thin representation can still be useful.
A map of roads is useful.
But a road map is not a weather report,
not a law book,
not a memory of accidents,
not a knowledge of why people travel,
not a judgment of whether the bridge
can take another thousand carts before morning.
The apprentice said,
And we have mistaken the road map for the kingdom?
The master said,
Often.
The apprentice said,
Then diagnostics has representation,
but not necessarily a model.
The master said,
Diagnostics has records arranged for investigation.
Observability has emissions shaped by a model of what matters.
The apprentice said,
What makes the model?
The master said,
The model begins with distinctions.
The apprentice said,
What distinctions?
The master said,
Normal and strained.
Accepted and rejected.
Promised and missed.
Recovering and failing.
Blocked and merely slow.
Expected and surprising.
These are not just values.
They are judgments.
The apprentice said,
And judgments belong at the source?
The master said,
The first judgments do.
Higher judgments belong higher.
The apprentice said,
A component cannot know the whole kingdom,
but it can say what happened at its gate:
what it accepted, refused, awaited,
and promised but did not fulfill.
The master said,
They are the beginning of signs.
The apprentice said,
Monitoring counts and compares.
Diagnostics collects and reconstructs.
Observability expresses and infers.
The master said,
Say it again, but with the burden included.
The apprentice thought for a while.
Monitoring asks a question we already knew to ask.
The master nodded.
Diagnostics asks questions after trouble has appeared.
The master nodded again.
Observability makes the system answer better questions as it runs.
The master said,
Better still.
The apprentice said,
Monitoring depends on thresholds.
Diagnostics depends on evidence.
Observability depends on a model.
The master said,
Now you are beginning to see the third lamp.
The apprentice said,
Then why do the vendors sell all three under one name?
The master said,
Because all three are useful,
and because the third name is the grandest.
The apprentice laughed.
The master did not.
A name can lift a tool above its nature.
It can also hide the work not yet done.
The apprentice said,
What work?
The master said,
The work of deciding what the system must be able to say about itself.
The work of designing signs.
The work of marking state where state is known.
The work of making emissions carry judgment,
not merely occurrence.
The work of building the model before the incident,
not inventing it during the incident.
The apprentice said,
That is hard.
The master said,
Yes.
The apprentice said,
So we choose diagnostics because it lets us postpone the model?
The master said,
Often.
The apprentice said,
We emit first and understand later?
The master said,
Often.
The apprentice said,
And then we call the pile observability?
The master said,
Often.
The apprentice looked at the three lamps.
The first lamp rang when a line was crossed.
The second lamp lit the path already taken.
The third lamp did not grow brighter
until the apprentice began naming the distinctions
by which the system could be understood.
The apprentice said,
Then the third lamp is not a tool I buy?
The master said,
No.
The apprentice said,
It is a capacity I design?
The master said,
Yes.
The apprentice said,
And the capacity begins before emission?
The master said,
As yesterday taught you.
The apprentice said,
Ask what it means before it leaves you.
The master said,
And today?
The apprentice answered slowly.
Do not mistake the evidence of passage for the model of state.
The master nodded.
He lifted the third lamp and placed it in the apprentice’s hands.
The lamp was still dim.
The master said,
It will brighten only as the system learns what it is able to say.