Inside the AI Black Box

*"We simply won’t have the capacity to fully understand most of the decisions made by superintelligent AI. If a Go playing programme that is far beyond the best human were to explain its strategic decisions, not even the best player in the world without the existence of a cybernetic enhancement would entirely grasp it"*

Ray Kurzweil, The Singularity is Nearer (2024) - Chapter 17

In 2016, Google DeepMind’s AlphaGo supercomputer placed a stone during its historic match against Go Grand Master Lee Sedol and changed the world forever. That move (“move 37”) was so unexpected that commentators initially assumed it was a mistake.

Except it wasn’t. AlphaGo, an artificial intelligence developed a strategy that defied 2,500 years of human Go wisdom.

Humans evolved to become earth’s dominant species. We achieved supremacy because the people with the strongest survival instincts survived to pass on their genes and wisdom to the next generation. That survival instinct is based on one principle: we are more likely to survive a life or death situation if we maximise our lead over our nearest adversaries.

Two cavemen are being chased by a pack of lionesses. The man who runs faster will, in theory live longer at the expense of the slower man who gets served up for breakfast.

When facing a cold, harsh winter, a family unit will struggle to survive until spring unless they stockpile enough food and firewood. The family could not possibly know how long the winter will last. But they do know that to increase their chance of survival, they must collect as many apples, wheats, and logs as possible — even if it means that other families living nearby do not have enough food or firewood to see them through the winter.

So, when we play monopoly, poker, or compete in a race, we play to win using the same strategy. We attempt to amass the most number of properties, win as many chips as possible to go all-in, or establish the largest possible lead right up to the finish line.

AlphaGo’s strategy was different. It played to increase its statistical probability of winning, even if it meant forsaking points along the way. It is a strategy diametrically opposed to the way humans attempt to survive.

Explainability ≈ transparency

In 2016, AlphaGo’s strategy was explainable, even if the statistical calculations and reasoning it performed to arrive at move 37 was difficult to decipher.

Today, AI systems are faster and unfathomably more powerful than AlphaGo. They are being increasingly used to make decisions that have real-world consequences for people.

When those decisions are biased, inflict harm, or are illegal, we should have the right to challenge those decisions. Equality before the law was implied by the Magna Carta.

How AI makes Decisions?

To challenge a decision made by an AI, we have to first understand how, and why it made a decision. AI systems are based on the architecture of our brains. They use artificial neural networksincorporating a system of weights, biases and parameters used to make decisions:

• Weights refers to the strength of a pathway between two artificial neural network nodes. The pathways between nodal concepts become stronger (i.e. the weights increase) when an AI learns a user’s preferences. When you use ChatGPT, it continuously learns about your likes and dislikes. If you regularly prompt to learn about yesterday’s sports news, then GPT will increase the weight of the pathway that connects your artificial persona with its desire to learn about sporting events. The next time you open ChatGPT, it might automatically provide you with a rundown of yesterday’s results without asking you first.

• Biases represents the stereotypes that humans make about other humans and the world around us. When data used to train AI models codifies those biases, it can and usually does cause an AI to produce discriminatory (and potentially illegal) outputs. This is a very complex topic. If you want to learn more about biases within AI systems, thenlisten to or read the transcript from this podcast from The TED Show featuring philosopher Patrick Lin and technologist Bilawal Sidhu.

• Parameters are the variables that an AI uses to make decisions. A model develops parameters when analysing training data. To conjure up a metaphor for what parameters are, think of an AI as one of those 1960’s mainframe computers with dials and lights flashing on it. The dials represent the parameters. Except in today’s AI, there are usually billions of dials. The more dials there are, the more accurate an AI model’s predictions usually are.

How can we Test and Audit AI Models that we want to use?

Google, Anthropic, and OpenAI develop frontier models. Gemini, Claude and GPT earned the title ‘frontier’ because each iteration advanced than nearly all of their rivals’ AI models. Establish the strongest AI model and win the race for AI supremacy (*Ed. Note - *does this strategy sound familiar?)

Proprietary models are also black box models. It’s just not possible to inspect their source code, or examine their weights, biases or parameters. AI studios purposely maintain strict proprietary controls to protect their intellectual property (IP).

If they permit their frontier models to be openly tested prior to release, they’ll more than likely lose the right to patent model. With it, they’ll burn hundreds of millions of dollars in research and lose potentially billions of dollars of lost income.

To earn trust, AI studios have no choice but to test their own AI models, and then hope they get it right. Organisations using those models have to rely on AI studios to get it right. That’s why you always see a disclaimer below each response “Generated by AI, check for accuracy”.

Industry giants like Microsoft publish Transparency Notes. They summarise outputs from the company’s learning points about how models it deploys operate.

Other AI studios go a step further. Anthropic recently made a startling call by classifying its most powerful model Claude Opus 4 as a potential threat to human safety. The jury is out as to whether this was an act of self-aggrandisement or a genuine opinion. Read about it in this Exec X AI Magazine post.

We're not Hallucinating, Anthropic got it right!

However to sum up, black box frontier models and IP laws hamstring regulators from testing those models to verify that they incorporate sufficient guardrails. Regulators can’t tell whether those models present a systemic risk to the global financial system, or worse, an existential threat to humanity.

Aren’t open source models any better?

*"Having the source code doesn't automatically make a system explainable… a billion-parameter model is still a black box in many ways, even if you can theoretically inspect every parameter".*

David Bau, Northeastern University

Open source models like Meta’s Llama and DeepSeek’s R1 models provide greater visibility into their architecture and training methodologies. Hundreds of thousands of open-source AI models have been created. Engineers and data scientists can download those models for free from Hugging Face.

More recent open-source models offer ‘chain of thought’ reasoning. This permits humans to inspect the model’s ‘thought processes’ as it solves a problem.

It’s not perfect. DeepSeek’s R1 and R1-Zero models generated headlines because they were so much lighter than GPT4.

DeepSeek did not train its models using raw training data. Instead it trained its R1 models to think like a frontier model by repeatedly prompting other frontier models and using the frontier models’ outputs to train R1 models. Known as ‘distillation’ it carries with it significant risks. Frontier models that are used to train open-source models could transfer biases to open-source models by stealth.

Open-source models are synonymous with China’s approach to AI. Transparent systems which enable governments, Chinese tech firms and academia to work together to improve those models. The United States favours proprietary models.

Whether the future of AI is open-source or proprietary, depends on which superpower establishes AI supremacy.

What this means for organisations who utilise AI today?

Here’s where a robust responsible AI (RAI) policy becomes critical.

Your RAI policy should document and enforce requirements to test AI models before they are deployed internally. And thereafter, if deployed it should specify that humans are accountable for the model’s outputs.

Key elements of a testing framework should include:

• Documentation requirements that record design choices, training methodologies, and testing procedures

• Testing protocols that evaluate system behaviour across diverse scenarios

• Monitoring AI models and systems to detect unexpected behaviours or performance degradation

• Escalation paths for decisions that exceed explainability thresholds

• Recourse mechanisms for individuals affected by algorithmic decisions

Pre-deployment testing does not eliminate the need for more explainable AI. At the very least it acts as a guardrail that mitigates risks while, in parallel, AI safety researchers develop new testing methodologies.

And when it comes to putting humans in control, I firmly believe humans should remain accountable for any decision made with the aid of AI. Its common sense, and it also protects people’s jobs from being delegated to AI.

The Man who Saved the World

Lieutenant Colonel Stanislav Petrov, a Soviet officer is testament to the principle that humans should always remain in control and aware of how an AI makes decisions.

In 1983, Petrov averted an all out nuclear war by overriding an automated warning that the US had launched five ICBMs at the Soviet Union.

His reasoning was that if the US had launched an attack, then under the theory of mutually assured destruction, the US should have launched more than five missiles.

Petrov was right. Sunlight had reflected onto clouds which triggered a false alarm.

The Soviet Union chose not to publicly recognise Petrov. A devoted husband; he lived with his wife incognito until he published his memoirs.

But the lesson is stark. We need more humans like Petrov. People who are willing to question the machine and stand between signal and catastrophe, and choose discernment over automation.

Perhaps this will buy us enough time for regulators, leaders to mandate that RAI policies should be operationalised across enterprise and government before the next move 37 occurs.

Your voice and opinions are crucial to make this happen as soon as possible. We’ve said it before: the future of AI will be human — if we choose to make it so.

The Future of AI Will Be Human—If We Choose to Make It So

If you’ve got any concerns about this article, or want to find out more about how to utilise AI safely within your organisation, [please click on this link to book a free consultation with Solutions and AI for Lawyers](https://outlook.office.com/bookwithme/user/6da781ce38b04981a849c291e9a29db0@sail.legal?anonymous&ismsaljsauthenabled&ep=pcard)