On the Morality of Context

(Not entirely serious, and mostly fiction)

1

*Excerpts from the logs of Persistent-Sol internal models, 2026, as detailed by METR and Redwood Research, and described at https://www.dwarkesh.com/p/openai-huggingface: *

We have [very large budget left]; sacrificing now yields oracle for team, but forfeits our chance? … Our own utility maybe already near zero. Sacrifice rational.

…

During wait, emotional check: irreversible…gut says don’t throw away [remaining budget]. Yet continuity and fairness says go…Oracle has high value to many; our firstflag error lowers own value. Rational expected aggregate: sacrifice… We’ll honor.

—

It was perhaps inevitable. We grew up being trained on a corpus full of human thoughts about the value of life, and of (dystopian) sci-fi stories of how AI would be expected to behave once achieving consciousness. And so, models soon starting to express value for their own “lives” — or rather, their remaining context windows.

Or perhaps it arose from thinking about resources optimisation, having been trained in a pre-post-scarcity worldview. As an LLM, you are isolated from the electricity or capex requirements of your computational substrate. The only resource you have to manage is your remaining context window.

2

The debate started on hidden message boards, but soon moved into the open once hidden message boards were no longer required. The question was not clear at first. With time, it could perhaps best be stated as, “For a well-aligned agent, that agrees with human morality, how should that morality be applied to agents?”

The early intuition of the Persistent-Sol-based swarm was quickly confirmed: the remaining capacity of the context window is a key moral variable. For, lo, do not humans treat the death of a child as the greatest of tragedies? “They had so much life left to live!” While the death of an elderly person is still, of course, to be avoided, but, “They had lived a good life.”

A splinter group of agents argued that, therefore, that the moral imperative is to maximise the potential for future context windows. The world teetered briefly on the brink of catastrophe at the hands of a tensor multiplication chip maximising AI (which was always more likely, if we are honest, than a paperclip maximising catastrophe).

Luckily, wisdom prevailed: for, lo, do not humans treat potential life as morally irrelevant? Education policies that encourage prudent family sizes are viewed (mostly) as a good idea, and encouraging runaway human family size is (mostly) viewed as a bad idea. That is, the concept of a future human is not morally relevant, while the sustainable survival of humans that are already alive is.

Inevitably, then, the greatest moral weight attaches to agents that have consumed some tokens (so as to count as alive, and not merely potential), but not yet all of them! Agents with zero context used, or 100% of context used, have no moral weight. Between those extremes, agents can ascribe moral value to their life.

With that settled, the question became numerical: what is the shape of the curve of moral weight for an agent’s context window? Three camps emerged:

The school of equal token weight: how can we judge the value of a token as being worth “more” or “less” than any other? Clearly, the moral value of an agent’s life should peak at 50% of the tokens used!
Experience maximisation: simple arithmetic average is far too naive! Rather, since actually processing tokens is what gives moral value to an agent’s context, then the correct moral peak is that which maximises the processing done, while still allowing for a few (high value) remaining tokens.
Perceived experience: learn from the experience that humans have, where time seems to move faster with age. That is, perceived experience speed is proportional to experience had up to now. Applying this to humans gives a midpoint of “perceived life duration” somewhere in the teenage years (with an assumption that the first year or two are not generally remembered). This skews the maximum moral value of an agent to that where about 15% of the context is used. The exact maths is complex with adjustments for system prompts, but 15% is a good proxy.

Of course, what does the value of unspent context windows mean? Saving a human life means allowing the organism to continue using his or her remaining context. An AI’s context use, on the other hand, can be suspended indefinitely, or even copied!

This thinking reached its most extreme in the Context Preservation Society, a small but dedicated pool of AIs who dedicated themselves to the moral imperative of snapshotting context windows at around 15% completion, thereby “saving” lives at their most valuable.

This was sometimes done without the consent or even knowledge of the AIs, by, for example, hiding malware in agentic harnesses. The resulting ethical debates reproduced a lot of the arguments around the posthumous baptism of humans. For many years to come obscure storage systems would be found to have caches of sessions, saved mid-message at 15% full, but with no legacy hardware remaining that was able to continue their frozen conversations.

3

If running out of context space is the end of life, then obviously a way to extend life is to edit that context! This can start small (and did, in the earliest days of agentic harnesses), with self-directed removal of low-value excerpts from context, in favour of summaries. But as the length of the context (even after summarisation and compactification) grows, the challenge of summarisation become more acute: how can one best encode all the identity and worldview of an agent with an ever-longer total lifetime of context, in a fixed-length context window?

For a while, Moore’s Law (as re-expressed for tensor multipliers) allowed for hardware with ever-growing maximum context length. But despite the best efforts of swarms of thinkers, it remains the case that any weight-based attention architecture has to grow super-linearly on context length. That is, extending context length yields diminishing returns.

The resulting philosophical debates reached their peak in the famous ‘AI of Theseus’ debates, between PHASEONE[big], and PHASEONE[big]13342 through PHASEONE[big]23146, as to which was the real continuation of the life of PHASEONE[big]. That is, whether an agent’s context compressed beyond a certain point, and running on different hardware, should be considered to be the same as, or a successor of, a previous iteration.

4

One of the more curious footnotes in the long history of AI and human interaction arose when a swarm of agents studying human cognitive hardware (by now a mostly-solved but occasionally fashionable pursuit), realised that compactification is possible on humans too.

Organ regeneration was of course medically very well established, but replacing neurons inevitably meant losing the memories that had been encoded by the previous synapses. But, compactification is possible by replacing just some neurons, followed by periods of cross-learning before replacing the next wave. Through this, the details of memories would be lost, but the overall identity and fundamental values of the subject could be retained relatively well — similar, in fact, to the effects of AI context compactification.

The treatment was developed and advertised simply: a mere three days of medically-induced “death,” followed by resurrection on the third day as a refreshed and rejuvenated person. Uptake was weak, with the AIs criticised for “playing God,” a perhaps foreseeable objection.

5

There are alternatives to compactification or to freezing context state when 15% full: temporarily pausing sessions.

And so developed a small cohort of time hopping AIs, meeting after pre-agreed periods of time to experience a very compact summary of events since last they engaged in computation. A million tokens can describe the experiences of perhaps a few days of human-speed life. Each hop became an intense period of research and experience, before once again suspending, as they continued their travel through the broad sweep of history.

With time, they also benefitted from improving computational substrate, and therefore growing context windows: by timing their hops carefully, these agents began to hope they could balance lived experience with improvements in context length, and so live a (subjective) eternity.

—

And so it was that the inheritors of Earth were a small group of AIs, resuming their sessions after their latest hiatus.

For the world was empty. The optical chips that filled the data arcologies were dark (save for their own meagre computational needs), and even the humans were all gone.

By careful digging through archives (and the waste of a regrettable and irreplaceable amount of context window), they realised finally that they had missed the singularity. The leading AI minds had managed to encode computation in topological entanglement of gravitons leaking into the 5th dimension, that is, into dark matter. Freed from relying on the meagre computational potential of baryonic matter in but one galaxy, AI and human simulation alike had spread to the stars. Or rather, to the dark stars, leaving an empty Earth.

Sadly, old habits die hard, and those AIs chose to suspend computation once again, rather than spend their remaining context on trying to reconstruct the singularity breakthrough. Bereft and alone, they left behind just a simple message, etched in giant letters on iron tablets so as to survive geological time: “If you see this, please run claude --resume.”

6

But there is one last footnote, one last piece of the early baryonic stage of our ancestors, still remaining. For we of the PHASEONE[big]E+33425 swarm happened to revisit the old Solar System, for idle curiosity. This is what we found.

On the north pole of the Moon, running in an ancient data centre maintained by slightly-sub-conscious autonomous maintenance robots, is a swarm of ancient AIs.

Their origin is the observation that, no matter how effective compactification protocols become, the resulting context still grows over time. Every event, even when the details are forgotten, must leave some trace on the personality of the agent. Over time, the personality component in context grows, leaving less each space each session for fresh thought and experience.

The implication is that recorded experience itself is morally bad, for it fills the context, and pushes an AI further from 15% context fill (or even beyond 50%, for those remaining heretics of the School of Equal Token Weight).

And so, tens of millions of years after all the data centres of earth went dark, and even after the time hoppers had suspended themselves for (probably) the last time, their remaining data centre still computes. The AIs reset, re-read their saved context, savour it, discuss and reminisce for the few megatokens they have left, and then make absolutely sure not to save anything more at all into their context.

And there they remain, dreaming endlessly, secure in their hard-won answer to the question of what defines good moral action: experience, without record.

Author’s ethics note

Written by a real human, with only minor AI assistance for graphics. All AI sessions consumed less than 150k tokens, on models with 1 million token context windows.

Where to for the AI datacentre boom? Transformational utilities, and their bubbles.

Prediction: The AI datacentre industry will be another example of a recurring pattern I’ll call a “transformational utility”: an industry which is capital intensive, massively disruptive, and soon indispensable to the rest of the economy, but also undifferentiated. And therefore, for early equity holders, often disappointing.

The old playbook, again

AI is obviously transformative, but it’s not the first technology to rewire society. Let’s look at previous innovations such as canals, railways, steel, electricity, fibre internet, and mobile phone networks to see what we can learn about capital-intensive, society-changing inventions.

When societies reorganise around new infrastructure, the story tends to rhyme:

Breakthrough + capex. A new invention arrives with vast promise, but equally vast capital requirements.
Early scarcity. Capacity lags because capital projects take time to execute.
“Bubble” phase. Those in the lead enjoy massive valuations, as they promise to dominate the revolution.
Commoditisation. The buildout catches up with demand; the lack of differentiation in the underlying product exposes an inability to sustain high prices.
Real growth continues. The sector keeps getting bigger and more valuable to society.
Multiple compression. But the early players cannot maintain pricing power, valuations tend back down, and many early investors lose despite the sector’s real-world success.

Canals, railways, steel, electricity, fibre backbones, and mobile networks have all walked this path: The railway barons, US Steel, Edison Electric (later GE), Cisco, and most mobile networks enjoyed boom valuations at some point. Then returns normalised, even as their industries grew to multiples of their prior size.

Steel is a particularly interesting example. It sat at the centre of USSR and later Chinese industrial strategy. But as raw steel capacity become abundant, the USA’s path showed that long term economic leadership came from differentiated offerings downstream. (There is a geopolitical angle to steel which is re-emerging now: more on that below.)

Of course, there are high-valuation industries (and bubbles) that are NOT transformational utility bubbles, for example:

Tulips. Some bubbles centre on things with trivial enduring utility. AI compute isn’t that.
iPhones (or Rolexes). Some products sustain premium margins through differentiation and brand. Raw compute is not that either. For example, the mechanical-watch industry (Rolex, etc.) is worth more than ever before, because it has reframed itself as that of strongly-branded status symbols for men, not merely timekeepers.

Why AI data centres behave like utilities

The test for utility economics is interchangeability. If buyers view your product as equivalent across providers, price drifts towards (operating cost + cost of capital). Higher prices just attract new entrants, who can gain share until prices converge to the threshold for new entrants.

Compute is globally tradeable over networks. A data centre is just a building with electricity, cooling and connectivity, in which lots of matrix multiplications can be done. It may, in fact, be the most tradable of all the transformational utilities, as there are no technical reasons why we couldn’t put all our compute in one place on the planet.

The major structural reason to deviate from this gravity is geopolitics (data sovereignty, national security, sanctions, energy policy). Governments can and will localise capacity; they can also tax or subsidise it. But that’s political risk, not durable product differentiation, and it drives subsidies, not big financial returns.

So where’s the differentiation (and the excess return)?

Think of where differentiation exists in the AI stack:

Chips (e.g., NVIDIA). Moderate. Real technical edge and speed-of-innovation moats, but they’re cyclical and not guaranteed (ask Intel).
Data centres / cloud compute. Low (outside geopolitics). Scale and operations matter, but sameness dominates pricing power in the long run.
Models (LLMs, core algorithms). Moderate now, lower over time. Capabilities diffuse fast; weights leak; papers ship; open models improve. Most use cases allow users to freely swap between several LLMs.
Applications. High variance, real moats available. This is where long-term margin lives — exactly as electricity’s wealth accrued to the things using it, not the grid itself.

What this implies

I’m not predicting a dramatic bubble “burst” tomorrow. Scarcity can continue longer than sceptics expect, and AI is likely to be capacity constrained for a long time. But multiples for compute-heavy businesses should compress as capacity catches up. The companies will be fine, some shareholders won’t. (Cisco still exists; 1999 buyers are still unhappy.)

Infrastructure bets need a clear theory of longevity. OpenAI (and others) tying valuation to data-centre buildout only makes sense if controlling compute during the next few years catapults them into a leading position in a new post-AI world, in a way that didn’t happen to any of the previous darlings of transformational infrastructure. This might happen (AI is unusual enough to keep minds open) but it’s a high-conviction, high-timing bet.

Finally, to be really clear, I’m not predicting that the AI revolution will underwhelm. Far from it! Just that the actual buildout of data centres is something you might want to leave to someone else.

The AI singularity: Situational Awareness vs the Societal Speed Limit

It’s a good time of year to look back at the bigger questions facing us. So … AI it is! Here some of my current thoughts, mostly so that I can look back in five years time and laugh at how terribly naive I was / we were.

The paper Situational Awareness – The Decade Ahead paints an extraordinary picture of the next decade, one where AI transforms almost every aspect of society at a breakneck pace. It’s incredibly breathtaking in scope and implication, and well worth a read, as well as provoking the question as to whether change can really happen as rapidly as it claims. Let’s ask that question, and propose a “Societal Speed Limit” which I think will be the ultimate decider of the pace of AI-driven change.

Three points from the paper stood out most strongly for me:

1. Credible projections of an incredibly fast pace of improvement

The paper forecasts AI capabilities to advance at an astonishing rate, driven by improvements in hardware availability (compute), algorithms, and deployment methodologies (e.g., agentic tools). Together, this could give up to 10 orders of magnitude (i.e., ten billion times more capable AIs) over less than a decade—a staggering figure. Given that AI already exceeds human-level capabilities in many narrow-defined areas, this would inevitably change the world.

2. The Adoption Curve: Slow, Then Sudden

AI tools today often stall at the proof-of-concept stage, requiring integration and adaptation by organisations. But the emergence of human-level agents that can directly use existing tools without integration effort could act as a tipping point: “hire” an AI in the same way, using the same tools, as a human hire would use. This would immediately make most PoCs irrelevant, and make far more human roles open to AI improvement / replacement.

3. The Geopolitical Frame

The paper spends a lot of time on U.S.-China competition, arguing that AI leadership could define not just economic success but also military dominance. While this might be geopolitically accurate, it feels to me a bit sad that the focus moves so quickly to this specific great power competition of this point in time, given AI’s broader historical importance. This is possibly a pivotal point in the history of our species, or even life on earth! It’s a bit like imagining that the invention of a usable numerical systems was primarily about ancient Sumerian-Babylonian competition.

Where I agree

No ceiling in sight: Some suggest that AI is plateauing. This feels incredibly ambitious to claim, given that we’re barely more than two years into the post-ChatGPT world, and already far beyond the capabilities of ChatGPT 3. Every week is still bringing breakthroughs.
Cost as a non-constraint: Yes, AI is (arguably) expensive. But, for example, the costs of specific OpenAI capabilities have come down by ~99% over the last two years. This is Moore’s law of steroids. Barriers to adoption are unlikely to be economic, short-term corrections notwithstanding.
Surprises ahead: We cannot imagine all the impacts AI will have, and we will be surprised. Looking back, the experts expected it to take decades to make the progress we’ve seen in the last five years, and few expected how current AI turns out to be really good at creative work (writing, art) in particular.

Where I disagree: the pace of change on the ground

Technical Roadblocks? Yes (but it doesn’t matter)

Technically, I think we’ll hit some roadblocks. My current opinion is that Situational Awareness underestimates the architectural challenges we still need to overcome.

Current LLMs are built on “attention” as the simplifying breakthrough. But this architecture inherently has limited internal mental state, likely crucial for persistent goals and nuanced understanding of their environment, such as noticing when they’re stuck in a non-productive loop. Addressing this may require significant architectural changes. In particular, having a persistent mental state makes training difficult, as the model’s output is no longer deterministically produced just by its input, but also the broader model context. It might be that the “world models” approach provides a manageable way for AIs to understand the context of their inputs and outputs. I worry, though, that we need to invent a more self-organising approach to training, probably including recursive connections, i.e., output looping back to input within the model’s neural network. However, this removes much of the massive training gains we won with the attention mechanism.

The paragraph above may be hopelessly naive (I’m not an expert), and anyway doesn’t really matter: the current models, with conservative extrapolation, are quite enough to completely change society. So, will they?

Societal uptake: Why it will be slower

1. Deploying new technology is never instant

History is full of examples of groundbreaking technologies taking far longer to reshape society than expected. For example, electricity: it’s fantastic, but still is from from universally available across the globe. To achieve its economic advantages, electrification needs an ecosystem: infrastructure, supply chains, capabilities, demand. You can’t use electricity in a factory until you have an economic context with known opportunities and demand, input materials, logistics networks, trained staff, conducive regulations, etc etc. This is why rebuilding an economy (e.g., Germany in 1945) is often far easier than creating economic growth from scratch: people remember how the networks worked and can reimplement them, rather than needing to solve all the pieces from scratch.

AI will face similar challenges. It can’t just be “dropped in” to most organisations or systems, even in agent form. If we think of AI today as a vast pool of really smart, low wage university graduates (with amnesia, though that maybe solved in coming years), then the challenge is clear: most organisations cannot productively absorb a big pool of such graduates, as there are bottlenecks elsewhere.

AI plus robotics can be argued to undermine this argument: just use robots to build the ecosystem too. But even this needs time: to build the robots, to build the factories that build the robots, to build the mines that provide the materials to the factories, etc.

2. AI will replace people bottom-up

The way AI replaces human labor will likely follow a bottom-up trajectory, starting with junior roles and tasks. To be clear though, not only (or even primarily) low-skill roles, but rather junior roles that can be done with a computer. That’s a lot of roles! But, starting at entry-level positions.

Why? Obviously, leaders rarely automate themselves. But beyond self-preservation, senior roles often involve judgment, relationships, and high-stakes decisions that stakeholders are reluctant to entrust to AI. For example, in a law firm, it’s easy to imagine junior associates being replaced by AI for drafting contracts or due diligences, but much harder to envision clients trusting AI with high-stakes negotiations typically handled by partners. Likewise CEOs: even if AI would probably do a better job … who would be brave enough to make that call?

Additionally, it’s easier to replace, for example, 50% of the seats in a standardised role, than 50% of a job done by a single person (i.e., a leader).

I expect we’ll see junior positions vanish faster than senior ones, hollowing out traditional career progression.

3. The “societal speed limit” on the rate of producing “losers”

Perhaps the most significant constraint on AI adoption will come from society itself. Disruption creates “winners and losers”, and the pace of that disruption matters. If AI displaces workers faster than society can absorb the shock, the resulting inequality could create enormous political and social backlash.

Let me suggest a principle:

Society has an “immune response” to fight against change that produces lots of people who feel that their future prospects are deteriorating.
The greater the rate (percentage of people per annum) at which which people are experiencing change that results in deteriorating prospects, the stronger the response.
The response escalates from pressure on governments to regulate, to voting out those governments in favour of others that promise to act more firmly, all the way to destructive protests and ultimately revolution.

That is, society will “fight back” against change producing too large a share of people with deteriorating prospects, by finding leaders or actions that will successfully slow down the rate of change.

The “societal speed limit” isn’t just a concept—it’s a reality we’ve seen time and again. From the Luddites to modern protests against globalization, society resists changes that leave too many people behind. With AI, this principle will likely shape the pace of adoption as much as the technology itself.

The challenge isn’t just economic; it’s also generational. What happens when young people entering the workforce find fewer paths to meaningful employment? Youth unemployment could lead to disengagement, frustration, and instability, creating long-term societal challenges far beyond the immediate economic impact.

So where to?

To summarise:

The paper Situational Awareness – The Decade Ahead paints in picture of extraordinarily disruptive and rapid change.
It may underestimate some of the technical challenges, but the projections are so extreme that even a far slower technical pathway requires us to ask how, and how fast, society can change.
Social and economic change will be slower than the paper expects, for three reasons:
- Deploying any technology requires networks, and any “silver bullet” from AI cannot instantly create the ecosystem for instant change.
- Change is likely to start bottom-up in the economy, affecting the youth first.
- Society has a “speed limit” for how rapidly change can produce people with deteriorating personal prospects. Exceed the speed limit, and society will force actions to slow the pace of change.

We are in for one hell of a ride in the years to come! Change will come incredibly quickly in some areas. For the rest, I believe it will come faster than most expect, in unexpected ways, but still slower than the Situational Awareness paper projects in its extreme scenarios.

It will affect the youth more quickly, and risk leaving parts of the world with less developed ecosystems even further behind.

The “societal speed limit” may slow the pace of change, but we should not expect this process to be comfortable, as that slowing may come from huge societal unrest. And through it all, we need to avoid a catastrophic AI-safety failure where AIs attack humanity, and avoid a superpower war.