The Normalization of Deviance in AI

This thought-provoking essay from Johann Rehberger directly addresses something that I’ve been worrying about for quite a while: in the absence of any headline-grabbing examples of prompt injection vulnerabilities causing real economic harm, is anyone going to care?

Johann describes the concept of the “Normalization of Deviance” as directly applying to this question.

Coined by Diane Vaughan, the key idea here is that organizations that get away with “deviance” - ignoring safety protocols or otherwise relaxing their standards - will start baking that unsafe attitude into their culture. This can work fine… until it doesn’t. The Space Shuttle Challenger disaster has been partially blamed on this class of organizational failure.

As Johann puts it:

In the world of AI, we observe companies treating probabilistic, non-deterministic, and sometimes adversarial model outputs as if they were reliable, predictable, and safe.

Vendors are normalizing trusting LLM output, but current understanding violates the assumption of reliability.

The model will not consistently follow instructions, stay aligned, or maintain context integrity. This is especially true if there is an attacker in the loop (e.g indirect prompt injection).

However, we see more and more systems allowing untrusted output to take consequential actions. Most of the time it goes well, and over time vendors and organizations lower their guard or skip human oversight entirely, because “it worked last time.”

This dangerous bias is the fuel for normalization: organizations confuse the absence of a successful attack with the presence of robust security.

Tags: security, ai, prompt-injection, generative-ai, llms, johann-rehberger, ai-ethics

I have been telling people to use AI as a thinking partner, not a replacement for thinking. To let it interview them, surface ideas, enhance their output rather than generate it. I still believe that. But I have come to realise it is not enough.

Even when you use AI thoughtfully, even when you treat it as a collaborator rather than an answer machine, you are still absorbing something you did not choose.

The Wrong Debate

We are having the wrong conversations about AI. Everyone debates whether it will take jobs, whether it hallucinates too much, whether it is truly intelligent. Almost nobody is asking the question that will define the next decade: whose values are baked into these models, and what happens when we outsource our thinking to them?

A senior technical leader said something to me recently that I have not been able to shake: “We think about sovereignty when we look at Chinese models. We worry about using them. But we do not think about it when using US models. We need to understand what is going into our models because otherwise we are outsourcing our brains, our decision-making, our cultural responses to a model. Whatever goes into the training data defines the culture.”

That asymmetry reveals our blind spot. We instinctively question AI that comes from cultures different to our own. We do not question AI that reflects assumptions we already hold. Which means we absorb those assumptions without noticing.

Baked In

Every AI model is trained on data that reflects specific cultural assumptions about what counts as good communication, professional behaviour, appropriate decision-making, correct priorities, and ethical choices. These are cultural artifacts, not universal truths. But AI presents them as if they are universal.

Ask an AI model about work-life balance and you will get answers shaped by US professional norms. Ask about organisational hierarchy and you will get answers reflecting Silicon Valley flat-org assumptions. Ask about management problems and you will get a worldview about what leadership should look like, how conflicts should be resolved, what good performance means. That worldview was baked in during training.

The training data for major models comes predominantly from English-language Western sources. The values embedded in responses reflect liberal US cultural assumptions. The “correct” answers to ambiguous questions align with specific cultural viewpoints. This is fine if you are comfortable with that and aware of it. It is problematic if you are not aware of it at all.

Data vs Values

Data sovereignty is a problem we understand. We have dealt with it for years: where is data stored, who has access, what jurisdictions apply. We can audit it. We can require data stays in specific countries. We can see where the servers are.

Values sovereignty is different and harder. You cannot audit what cultural assumptions are embedded in model weights. You cannot require a model “thinks” in culturally neutral ways. There is no such thing as culturally neutral. Every choice about what to include in training data, what to filter out, how to weight different sources, what outputs to reinforce, embeds specific values.

When a UK company uses US-trained AI for HR decisions, hiring recommendations, and performance reviews, whose cultural values are shaping those decisions? When a Japanese company uses Western AI for strategy recommendations, whose business philosophy is informing that strategy? These are not hypothetical questions. This is happening now, at scale, in organisations that have not thought about it.

Hallucinated Values

We have learned to watch for AI making up citations, inventing statistics, generating false information. We fact-check. We verify. We treat AI outputs with appropriate scepticism when it comes to factual claims.

But we do not apply that same scepticism to values. AI presents culturally-specific assumptions as universal truths. It embeds biases we do not notice because they match our own. It creates echo chambers of thought that feel like common sense.

People are using AI to understand historical events filtered through training data biases, to form opinions on current events shaped by the model’s embedded viewpoints, to make personal decisions based on values they did not consciously choose, to define organisational culture reflecting Silicon Valley norms by default.

If we are not careful, this kind of hallucination will leak into our understanding of ourselves, our organisations, and our history. Not through obvious errors we can catch, but through subtle assumptions we never thought to question.

No Clean Fix

There is no clean solution here.

You cannot choose a “neutral” AI because neutral does not exist. You cannot audit values the way you audit data storage. You cannot require AI providers to disclose embedded values because we do not even have good frameworks for measuring what those values are.

Some companies try to “clean” training data by filtering out bias. But that creates new problems: whose version of clean? What gets filtered and why? Who decides what counts as bias versus legitimate cultural difference? You have not removed values from the model. You have just changed whose values are embedded.

The absence of a perfect solution does not mean doing nothing. It means being thoughtful about an imperfect situation.

Model selection is a values decision, not just a cost decision. When you choose which AI models to deploy in your organisation, you are choosing whose assumptions will influence your teams. Treat it with that weight.

Diversify your AI sources. Do not rely on one model family. Use multiple models with different training approaches as a check on each other. If they give you different answers to the same question, that difference is information. It reveals where assumptions are shaping outputs.

Audit for values, not just accuracy. Just as you would review code for security vulnerabilities, review AI outputs for cultural assumptions that do not match your organisation. Ask whose viewpoint is this representing? What alternatives might exist?

Build internal AI literacy. Teams need to understand that AI is not neutral. It has embedded viewpoints. This awareness alone changes how people interact with AI outputs.

Do not outsource judgment. Use AI to gather information and explore options. Do not use it to make decisions directly. The decision should remain with humans who understand your specific context, culture, and values.

There is no such thing as unbiased AI. Every model reflects the choices made in its creation. The question is not whether your AI has values baked in. It does. The question is whether you know what those values are, whether they align with your own, and whether you are consciously choosing to adopt them.

Most organisations have not asked these questions yet. As AI adoption accelerates, as more decisions get made with AI assistance, as more thinking gets outsourced to these systems, the organisations that thrive will be the ones that engaged with this problem early.

On an airplane, we notice even tiny changes in acceleration (including direction) but we’re completely unaware that we’re traveling at hundreds of miles an hour.

“Compared to what” is the unstated question that we ask ourselves, all the time.

Consumers, employees and peers are unlikely to think about what’s already a given. It’s the changes we notice.

AI and LLMs pose a particularly visceral threat to the typing class. Writers, editors, poets, freelancers, marketing copywriters and others are voicing reasonable (and unreasonable) objections to the pace and impact of tools like Claude, Kimi and ChatGPT.

I think we have two choices, particularly poignant on US Labor Day…

The first is to walk away from the tools. You’re probably not going to persuade your competitors and your clients to have as much animosity for AI automation as you do, and time spent ranting about it is time wasted. But, you can walk away. There’s a long history of creative professionals refusing to use the technology of the moment and thriving.

If you’re going to walk away, the path is clear. Your work has to become more unpredictable, more human and more nuanced. It has to cost more and be worth more. It turns out that the pace of your production isn’t as important as its impact. Writing a hand-built Linkedin post that gets 200 comments isn’t a productive path in a world where anyone can do that. If we’re going to put ourselves on the hook, we need to really be on the hook.

Remember the mall photographers who took slightly better than mediocre photos of kids at Sears? They’re gone now, because we can take slightly better than mediocre photos at home.

The other option is to dance. Outsource all relevant tasks to an AI to put yourself on the hook for judgment, taste and decision-making instead. Give yourself a promotion, becoming the arbiter and the publisher, not the ink-stained wretch. Dramatically increase your pace and your output, and create work that scares you.

This requires re-investing the time you used to spend on tasks. Focus on mastering the tools, bringing more insight to their use than others. Refuse to publish mediocre work.

It’s tempting to fear AI slop, because it’s here and it’s going to get worse. But there’s human slop all over the internet, and it’s getting worse as well.

Whether you dance or walk away, the goal is the same: create real value for the people who need it. Do work that matters for people who care.

If we’re going to make a difference, we’ll need to bring labor to the work. The emotional labor of judgment, insight and risk.

This originally ran in The Free Press.

Poems have always been earnest. That’s why some of them are so cringe. 

Rhapsodizing about nature. Pouring out your heart to a lover. Finding deep meaning in small things. Brooding on mortality.

But a few years ago, I was talking to Allie Esiri for the Daily Stoic podcast about her wonderful book A Poem For Every Night of the Year, which I have been reading to my sons since they were little. I mentioned that I was struck by the earnest desire for self-help and self-mastery in many of the 19th-century poems written by male authors. 

You might be familiar with some of them. 

Kipling’s If— is obviously a classic of this genre. So is Henley’s Invictus. Adam Lindsay Gordon’s Ye Weary Wayfarer is another one of my favorites. 

“Life is mostly froth and bubble,

Two things stand like stone,

Kindness in another’s trouble,

Courage in your own.”

It must be acknowledged that many of the most famous of these poems were products of the British Empire at the height of its imperial power. While I’m pleased that we no longer publish poems calling a generation to pick up ‘the White Man’s burden’ or celebrating the suicidal (and avoidable) charge of the Light Brigade, I would like to point out that there was once a time, not that long ago, when an average person would pick up their daily newspaper and find a totally straightforward, understandable poem full of advice on how to be a good person or navigate the difficulties of life. 

Perhaps because it doesn’t have any jingoism or machismo—just the source code for existence—one of my favorites of this genre has always been one that is uniquely American: Longfellow’s A Psalm of Life.

It opens quite powerfully, 

Tell me not, in mournful numbers,

     Life is but an empty dream!—

For the soul is dead that slumbers,

     And things are not what they seem.

 

Life is real! Life is earnest!

     And the grave is not its goal;

Dust thou art, to dust returnest,

     Was not spoken of the soul.

 

Not enjoyment, and not sorrow,

   Is our destined end or way;

But to act, that each to-morrow

   Find us farther than to-day.

I was a kid in the 90s. I was in high school in the early 2000s. I started my career in the citadel of hipsterdom, American Apparel. Everything was couched in irony. There was a self-consciousness, an almost incapacity to be serious. The drugs and the partying and the sex were, I suspect, hallmarks of a culture distracting itself with pleasure so it would not have to look inward and come up empty. 

Is that really what we’re here for, Longfellow asks? No, he says, we’re here to work on ourselves and to get better, to make progress—for tomorrow to find us a little bit further along than we are today. 

Do things! Make things! Try your best! You matter! That’s what Longfellow is saying.

A 13-year-old Richard Milhous Nixon was given a copy of A Psalm of Life, which he promptly hung up on his wall, memorized, and later presented at school. Perhaps it’s because of people like Nixon—or Napoleon or Hitler—that we shy away from talking about individuals changing the world these days. The ‘Great Man of History Theory’ is problematic. It’s dangerous. It’s exclusionary. The problem is that in tossing it, we lose the opportunity to inspire children that they can change the world for the better, too. 

Many of Longfellow’s poems do precisely this: There’s one about Florence Nightingale, an angel who reinvents nursing. There’s another about Paul Revere and his midnight ride to warn revolutionary Patriots of approaching British troops. He celebrates Native American heroes in “The Song of Hiawatha.” They are not always the most historically accurate accounts, but to borrow a metaphor from Maggie Smith’s poem, “Good Bones”—which, again, some pretentious folks might find cringe—when we read poems to our children, we are trying to “sell them the world.” Will we sell them a horrible one? Or we will sell them on all the potential, the idea that they could “make this place beautiful”?

One of the reasons I find so many modern novels boring and end up quitting most prestige television is that everybody sucks. Nobody is trying to be good. Nothing they do matters. I’ve even found that many children’s books fall into the same trap. They are either about nonsense (pizza, funny dragons, etc) or they are insufferably woke (pandering to parents instead of children). Academia has been consumed by the idea that everything is structural and intersectional and essentially impossible to change. History was made by hypocrites and racists and everything is rendered meaningless by the original sins and outright villainy of our ancestors.

To be up and doing, as Longfellow advises, laboring and waiting, they would claim, is therefore naive. His privilege is showing when he tells us to live in the present and have a heart for any fate. One Longfellow critic has referred to the poem’s “resounding exhortation” as “Victorian cheeriness at its worst.”

But what’s the alternative? Because it’s starting to feel a lot like nihilism. I’m not sure that’s the prescription for what ails young men these days. In fact, isn’t that the cause of the disease?

I draw on one of the lines from A Psalm of Life in my book Perennial Seller, about making work that lasts: “Art is long, and Time is fleeting.” There’s a famous Latin version of this expression: Ars longa, vita brevis. 

I find it depressing how ephemeral and transactional most of my peers are. They chase trends and fads. They care about the algorithm and the whims of the moment—not about making stuff that matters and endures. Longfellow urges us to resist the pull of what’s hot right now—to think bigger and more long-term, to fight harder:

In the world’s broad field of battle,

   In the bivouac of Life,

Be not like dumb, driven cattle!

  Be a hero in the strife!

That stanza is an epigraph in another one of my books, Courage is Calling. I return to it throughout the book because that’s what these things—earnestness, sincerity, the audacity to try, to aim high, to do our best—require: courage. It takes courage to care. Only the brave believe, especially when everyone else is full of doubt and indifference. As you strive to be earnest and sincere, people will laugh at you. They will try to convince you that this doesn’t matter, that it won’t make a difference. Losers have always gotten together in little groups and talked about winners. The hopeless have always mocked the hopeful.

It’s been said by many biographers—often with a sneer—that the key to understanding Theodore Roosevelt (who would have certainly seen Longfellow strolling through Cambridge while he was an undergrad at Harvard) is realizing that he grew up reading about the great figures of history and decided to be just like them. Roosevelt actually believed. In himself. In stories. In something larger than himself. It is precisely this idea that Longfellow concludes Psalm with:

Lives of great men all remind us

   We can make our lives sublime,

And, departing, leave behind us

   Footprints on the sands of time;

But this is not mere hero worship, because Longfellow qualifies it immediately with a much more reasonable, much more personal goal, explaining that these are,

“Footprints that perhaps another,

     Sailing o’er life’s solemn main,

A forlorn and shipwrecked brother,

     Seeing, shall take heart again.

 

Let us, then, be up and doing,

   With a heart for any fate;

Still achieving, still pursuing,

   Learn to labor and to wait.

Indeed, Longfellow would hear, not long after the poem’s publication, of a soldier dying in Crimea, heard repeating to himself with his final words, “footprints on the sands of time, footprints on the sands of time, footprints on the sands of time…” 

A Psalm of Life is a call to meaning. A call to action. A call to be good. A call to make things that matter. A call to try to make a difference—for yourself and others. A reassurance that we matter. That although we return to dust, our soul lives on.

That’s why I read it to my sons. That’s the lesson that I want to pass along, a footprint I am trying to leave behind for them now, so that they might draw on it in some moment of struggle far in the future. So that they can always remember why we are here:

Not enjoyment, and not sorrow,

   Is our destined end or way;

But to act, that each to-morrow

   Find us farther than to-day.

One foot in front of the other. One small act after one small action. One little thing that makes a difference, for us and for others.