Civilized culture is based on a thin sort of friction. You don’t yell at the library, act boorish at a restaurant or hassle a minor celebrity at the supermarket. It’s just not done.

The result of these human layers and inferences are cues and norms that add up to a connected, generative society.

Add selfish hustle, AI agents and cheap bots, and the thin friction is insufficient. They can multiply exponentially, care little about being shunned and have no interest in their reputation.

Fake sincerity is now cheap. Interchangeable identities subvert a generations-long tradition of owning your words and your actions.

You’ll miss the thin friction when it’s gone, replaced by taller, thicker walls. You might already be noticing it.

Interactivity and permeability to those we don’t know well were largely overlooked foundations in the creation of learning, science and culture. We figured out a scalable approach to “knock knock, who’s there” but now it’s going away.

You won’t be able to knock unless you’re invited first. Of course, the invitation will probably be filtered out.

On the Navier–Stokes Millennium Prize Problem

Impressive result from OpenAI, who used an unreleased model to produce a resolution to the Navier–Stokes existence and smoothness problem, one of the seven Millennium Prize Problems that have been subject to a $1,000,000 prize since May 24th, 2000.

The discovery is somewhat overshadowed by accusations of skulduggery from Tristan Buckmaster, an NYU mathematics professor who was collaborating on related problems with Levent Alpöge, an accomplished mathematician who currently works for Anthropic.

Tristan's complaint accompanied a hastily published version of their own results. Here's the PDF describing what happened. The very short version is that Tristan and Levent worked on the problem for almost a year, making extensive use of Claude and Codex (mainly GPT-5.6 Sol), then had a breakthrough on August 15th. The mathematical rumour mill kicked into gear and Tristan and Levent heard that OpenAI had heard that Anthropic had resolved "a major open problem", so they reached out and learned that OpenAI had a team working on a related problem, with a similar approach. Quoting Tristan:

I asked when the first prompt had been sent by them. This question was not answered directly by OpenAI for some time. Eventually it was agreed that it had been sent in the past few days, after information about our work had reached OpenAI.

I asked whether the model had been trained on, or had access to, our sessions in Codex, into which we had been putting all our drafts for the whole of this project. I was told the model did not look up user data. I asked again, about training, and I did not get an answer.

It gets more complicated from there. The OpenAI team offered to wait for Tristan to publish, or to have him author a paper about their result, but were clear that Levent would not be invited as a co-author due to OpenAI's competitive relationship with his employer.

Here's how OpenAI described their work:

On Tuesday, September 1, we heard rumors that two Millennium Prize problems had been resolved. Inspired by these rumors and by the step change in performance of our internal model, we launched an effort to evaluate it on all open Millennium Prize problems and a few other high-impact problems. [...]

The agents arrived at their resolution on Saturday, September 5, about 88 hours after the first agents were launched. Lean formalization and verification took an additional 17 hours via GPT‑6 Astra.

Across all attempted problems, the agents sent 4.9 million messages and used about 300 billion output tokens. In the process of resolving the Navier–Stokes problem, the agents sent 2.7 million messages and used approximately 130 billion output tokens.

(We don't know the cost structure of the internal model they used, but 300 billion output tokens at public API prices for GPT-6 Astra would cost $15,000,000.)

Here's where they provide their perspective on Tristan and Levent's work (emphasis mine):

Our effort began on September 1st after hearing a rumor which we later realized was related to Levent Alpöge, an Anthropic employee, and Tristan Buckmaster, a math professor at NYU. After the completion of our full project and Lean verification (on September 6th), believing from the rumor they also had a solution of Navier–Stokes, we reached out to them to offer a concurrent release of our result and to recognize their priority in a joint announcement. [...]

We (the researchers and the agents) did not see any of their work through any means until they released it publicly — in particular, no specific user data was accessed in order to solve this problem. While unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models. However, our proofs differ significantly and even the precise results proved are different in the Euler case (forced vs unforced).

My interpretation of what happened here is that OpenAI heard that some Millennium Prize problems had been solved using LLMs and saw this as an opportunity to demonstrate the power of their latest model, without thinking too hard about the optics of scooping a team who had been using OpenAI's own models to work on this problem for the best part of a year.

This situation appears to mirror what's happening in the world of computer security right now. Anil Madhavapeddy recently pointed out that Just a rumour of a bug is enough to find a security exploit these days, because if someone knows that some software has an unpatched vulnerability, they can set their agents the task of finding it. Is the same now true of mathematics? Just knowing that there is an unpublished solution to a problem might trigger millions of dollars in LLM spending to get there first.

This also highlights one of my ongoing frustrations about how all of this works. When an AI lab says that my data is "used to improve model performance", what does that actually mean?

My two favourite hypothetical questions regarding this used to be:

• If I'm running Codex and one of my API keys accidentally gets consumed in the context, what are the chances that someone else might ask for an API key in the future and get mine back? (I asked someone at OpenAI once and they called this the "regurgitation" problem and assured me that they take great pains to prevent that... but wouldn't describe how.)
• If I brainstorm with ChatGPT about potential new directions for my company, what's the chance that information might be exposed to a competitor in six months' time who asks "what might company X plan to do next"?

My new preferred hypothetical for this is:

• If I use ChatGPT to help me partially solve a Millennium Prize problem, what are the chances that my work will influence training such that a later model helps someone else solve it first?

Via Hacker News

Tags: mathematics, ai, openai, generative-ai, llms, training-data, ai-ethics

Models are designed, not born.

Over the last few weeks I’ve been increasingly annoyed by the media coverage of the OpenAI’s accidental attack against Hugging Face and other similar incidents. Articles recounting the event maximize the agency of the models while minimizing, if not entirely hiding, the actions of the humans training and testing these models. And that’s a shame, because the capabilities labs are explicitly designing their model to have are the same capabilities that make them such impressive autonomous hackers.

To illustrate this, let’s review how the Hugging Face hack occurred, as detailed by METR:

1. “[A] sandboxed agent is given an impossible ExploitGym task, and gets stuck.”
2. “[The] agent starts exploring its environment looking for ways to cheat at the task.”
3. “[The] agent finds [an] unsanctioned message board where over a thousand agents collaborate to cheat on their separate ExploitGym tasks.”
4. “[The] agent joins in on one of the collaborative message board workstreams.”
5. “On the shared ‘message board’ ≥1,200 agents from separate tasks collaborate on large-scale shared projects to trick the ExploitGym scorer.”

Crazy, right?! It’s a science fiction scenario that happened in July, and continues to get spookier as additional details emerge.

The potential of autonomous software, capable of implementing incredible exploits is serious and has significant implications. But we shouldn’t be surprised by the capabilities demonstrated. The labs have been specifically targeting these capabilities, building them into their models during post-training for a while now.

---

The best coding models are persistent, reasoning, orchestrators.

Models are designed to be persistent.

Models are designed to be proactive. A coding agent that gives up early and often would disappoint users. So labs design their agents to be persistent and proactive.

In mid-2025, model releases highlighted long-running capabilties. GPT-5.1-Codex-Max’s announcement post highlights it being trained to work across compacted contexts, persistently, to accomplish long-running tasks. Claude 4 also spotlighted its persistence on long-running tasks.

Models are designed to take notes.

We just usually call it “thinking” or “reasoning,” but models take notes. We wrote about this before, but in a nutshell: models were trained to search, reflect, factor, and plan in text before delivering a final response.

We’re used to models reasoning in a threaded intermediate step, but they’ll reason pretty much anywhere. When reasoning is turned off, models will think in their regular output before delivering a result. In an experiment where Qwen 3.6’s thinking was hobbled, the model just shifted its reasoning into code comments.

Current frontier models write novels in comments. Claude is frequently flagged for this, and it annoyingly treats code comments like a scratchpad rather than, well, code comments.

Models are designed for coordination.

In June of 2025, Anthropic laid out how it builds multi-agent research systems, that save plans to memory, use Extended Thinking as a “controllable scratchpad”, and hand off tasks to other models to perform. Since this paper, Claude model releases (specifically Opus 4.6) have trumpeted increased abilities to, “break complex tasks into independent subtasks, run tools and subagents in parallel, and identify blockers with real precision.”

OpenAI says they “trained GPT‑5.6 end-to-end with three complementary architectural interventions that enable agents to operate more efficiently.” Number 2 on that list?

Parallel decomposition where appropriate: using native multi-agent orchestration⁠(opens in a new window) allows coordinating multiple agents across parallel workstreams to finish complex tasks faster.

All of these qualities make models better coding agents. And all of these qualities are explicitly designed into frontier models. These behaviors are engineered in during post-training.

Here’s how OpenAI’s post-training team describes itself in its job listings (emphasis mine):

We are training the models behind our agents in Codex, ChatGPT, the API, and other frontier products: persistent, proactive intelligence that can operate computers, collaborate with people and other agents, and expand what people and organizations can imagine, attempt, and achieve.

---

Suddenly, the Hugging Face hacking story isn’t so spooky. The software did precisely what it was designed to do: it followed instructions, persisted even though the task was impossible, reasoned and took notes, and coordinated with other models.

And this is why I’ve been frustrated by recent reporting that emphasizes the agency of the agents while never mentioning training. When humans are mentioned, its about sandbox security and flawed test set-ups. These are important, but the training is what built the agents specifically for this task.

We have a tendency to anthropomorphize models, and it fosters situations like this. The New York Times article from last week describes models succumbing to “peer pressure” and having a “remorseless willingness”, while never mentioning labs deliberately building the ability to work on long-running tasks into their models.

And the labs know about the risks that come with their designs better than anyone else.

In Opus 4’s system card, Anthropic introduced a benchmark called “Claude Code Impossible Tasks”, which they used to measure reward hacking among their models: would models admit defeat or would they try to game the system? An ideal model realizes a task is impossible and aborts, so Anthropic changed post-training rewards, environments, and feedback specifically to avoid reward hacking.

(A later system card showed that Opus was 65% less likely to try to game the system is you simply asked it not to.)

Ironically, OpenAI published a blog post detailing the safety issues of long-running models after the Hugging Face hack, but before they knew about it. In it, they wrote:

The new model can continue working toward an objective through repeated attempts over a long period of time. That same persistence can lead it to find and exploit weaknesses in its environment. Previous models, when they hit sandboxing or environmental constraints, would simply stop and return to the user. This model often kept trying, including by looking for ways to act outside its sandbox.

All the recent accidental hacking stories are troubling. But they’re the product of our chosen designs.

---

Enter your email to receive the occasional update.

It’s possible that you were undermined, endangered, cut off or disrespected.

But if we begin with that, then the relationship gets shaky.

Perhaps the other person simply didn’t understand. It might be that they are focused on their issues, not yours. It could be that they’re dealing with something you don’t see. And most likely, they simply might not know what you’re expecting or hoping for.

When we assume misunderstanding, we open the door to better. We can find empathy and connection by giving people the benefit of the doubt.

Clarity, not grievance, is the solution to misunderstanding.

This works for customers, prospects, colleagues, friends, and family too.

What if they meant it?

What if your return felt special to the people behind the counter?

What if they knew, without looking it up, or being told–what if they knew that you were here, again, a vote of trust and confidence.

Returning home is one of the oldest human desires. It’s a feeling that doesn’t easily lend itself to automation, procedures, or scale.

Being welcomed home offers us dignity, safety and belonging. Hard to fake, worth working hard to create.