The Black Box

In the 1950s, aviation had a problem.

Aircraft were becoming faster, more complex, and capable of operating in conditions where, when something went catastrophically wrong, the evidence investigators needed could disappear with the aircraft.

They had paperwork.

What they did not necessarily have was a surviving record of what actually happened inside the system in the moments before failure.

Australian scientist David Warren became interested in a deceptively simple idea: what if the aircraft carried a record of its own operation, built to survive the crash?

His early recorder captured flight data alongside cockpit sound. The idea initially met resistance. Australian aviation authorities saw little immediate use for it. Pilots had understandable concerns about a device recording their conversations.

But the principle survived.

If you want to understand a consequential failure, preserve enough of the system to reconstruct it afterward.

I wonder whether organisations running AI supported decisions are still learning the same lesson.

Organisations entering the AI age are extraordinarily good at recording who approved a decision. They are far less capable of reconstructing how that decision actually came to be.

These are not the same record, and one does not substitute for the other.

A signature records an approval. It does not preserve a sequence.

It does not show what options the system made available, excluded or privileged before presenting its recommendation. It does not show what the objective was optimising for, or when that objective was last examined. It does not show a warning that appeared once, was overridden, and was never logged again.

What gives the black box its value isn't that it assigns blame. It's that it makes a system's failure reconstructable, legible enough afterward to understand how it actually happened.

Most organisations investigating a failed AI supported decision today still have the equivalent of the captain's licence.

What is much harder to find is the equivalent of the recorder.

I wonder whether that gap is only visible once, in the moment an organisation actually needs the reconstruction and discovers it was never being kept.

Question for the Field

Imagine a consequential AI supported decision in your organisation failed badly tomorrow.

Not who approved it. You already know how to find that.

Ask instead whether you could reconstruct the sequence that produced it: what the system was optimising for, what information shaped the recommendation, what options were made available or excluded, what warnings surfaced, and what happened to them.

And then ask the hardest question.

By the time the recommendation reached the person whose name appears on the record, how much of the decision was actually left for them to make?

If you cannot answer that, you have an approval record.

You do not yet have a black box.