How Nature Teaches Self-Restraint, and What It Means for AGI
We must deliberately pass that logic of nature to AGI before its speed and power outrun evolution's slow filter.
Credit: The Wild Robot (2024 Film)
I systematically summarized the theoretical framework for the counter-intuitive evolutionary advantage of self-restraint and how that applies to AGI as well. These are my literature review of the topic and my original synthesis. This is a draft version not ready for formal publishing yet, but I think it can be a core document for my Substack. Later, I may consider adding formal citations for each argument, but I think these arguments are broad enough so that you can easily do your own research to verify them.
I think this topic is super interesting. It is crucial for understanding how we can build well-aligned AGIs, but very few people have connected the dots. This topic spans evolutionary biology, ecology, game theory, moral philosophy, sociology, and AI alignment. Although I am not an expert of these fields, I try to do my synthesis using my research mindset. Please leave your comments if you feel pleased, inspired, or confused after you read this! And please point out my errors if you find any.
This is a long article, 8000 words, which has an introduction, 7 sections, and a conclusion. I will also post individual sections separately for easy views.
Table of Contents:
Introduction
Competition doesn’t always end in extinction, because nature rarely leaves perfect ties
“Restraint” without anyone deciding: how cooperation survives in the long run
Humans make the implicit explicit: from unconscious balance to conscious stewardship
Answering “when one species can completely replace another species, it always does”
Why bio-diversity is important even for AGI
AGI’s evolution with self-restraint must be initialized by humans rather than guided by natural selection alone
Making restraint real: concrete design moves, human-readable first, precise beneath
Conclusion: A rule of life at planetary scale
I wrote this article with the great help of ChatGPT 5.
Introduction
Core Thesis: Morality is not a human ornament but nature’s survival algorithm under certain circumstances. It is self-restraint emerged from interdependence. And our challenge is to deliberately pass that logic on to AGI before its speed and power outrun evolution’s slow filter.
When people think of nature, they often imagine it as a brutal contest: tooth and claw, fight to the death, one species winning while another disappears forever. And yes, extinction happens. But if we look more closely, we find that nature isn’t only about ruthless elimination. Very often, species survive side by side, even when one seems stronger than the other. This happens because life has ways of “holding back,” of not pushing competition to the breaking point. That “restraint” keeps systems stable, and it lets diversity flourish over time.
Biologists use technical terms for these patterns. For example, niche partitioning means species divide resources instead of overlapping completely. Trade-offs mean no creature is best at everything, so weaker competitors often persist in niches where they have an edge. The storage effect describes how environmental ups and downs, seasons, climate shifts, give different species turns to thrive, preventing any one from monopolizing resources. Keystone predation is when predators prevent a dominant competitor from wiping out others, thus holding diversity in place. And character displacement is when species evolve differences (like bird beaks of different sizes) that reduce direct conflict.
Humans are unusual because we’ve made this natural restraint into something conscious. We build laws, ethical systems, and scientific institutions to protect not only fertile farmland or friendly animals but even harsh deserts, icy glaciers, and dangerous wolves or bears. In other words, we turned ecological “unconscious restraint” into deliberate stewardship.
If we now create artificial general intelligence (AGI), machines that can learn and reason as broadly as humans, the stakes rise higher still. Evolution for AGI won’t be slow and messy like biology. It will be digital, global, and blindingly fast. Mistakes could be irreversible: once an AI system erases human life or the biosphere, the lost diversity cannot be restored. Technologists call this an irreversible value lock-in, a permanent narrowing of options. And in this article, we would argue that restraint for AGI cannot be a sentimental afterthought. It must be seen as the most rational survival strategy: the path that preserves stability, diversity, and future possibility.
1) Competition doesn’t always end in extinction, because nature rarely leaves perfect ties
At first glance, it seems obvious: when two creatures want the same thing, one will win and the other will lose. That’s the way we often imagine nature, red in tooth and claw, a string of fights that end with only one victor left standing. But if you look more closely at real ecosystems, the story is not nearly so simple. Life in the wild is messy. Environments vary, seasons swing back and forth, and species adjust to one another. Those details soften the sharp edge of a winner-take-all fight and often allow multiple species to persist side by side.
Scientists have a name for the stricter version of the “winner takes all” idea: the competitive exclusion principle, also known as Gause’s law. In its simplest form, it says: if two species occupy exactly the same niche, meaning they use exactly the same resource, in the same place, at the same time, then one will eventually outcompete the other, driving it to extinction. The key word here is exactly. In the real world, exact ties are rare. Nature has many ways of scrambling sameness, opening space for coexistence.
Let’s look at some concrete examples. On the Galápagos Islands, Darwin’s famous finches at first overlapped in their diets, competing heavily for the same seeds. But over time, they evolved differences in beak size and shape, with some specializing in cracking large seeds and others in handling small ones. This evolutionary shift is called character displacement, when species evolve traits that reduce direct competition. It means that instead of wiping each other out, they learn to “split the bill,” so to speak, by using slightly different resources.
Another puzzle is the so-called plankton paradox. Ocean waters often contain dozens of species of phytoplankton, even though they all seem to need the same nutrients. Why don’t the stronger ones eliminate the weaker ones? The answer lies in the shifting nature of the environment: light levels change, temperature shifts, currents stir the water. These fluctuations mean that the “best” competitor changes from day to day or season to season. This dynamic is formalized in ecology as the storage effect, species “store up” gains when conditions favor them and then persist through the lean times until conditions swing back in their favor.
Predators, too, can keep diversity alive. Along rocky shorelines, the sea star Pisaster preys on mussels. When biologists experimentally removed the sea stars, mussels exploded in number and crowded out most other species, creating a dull, simplified ecosystem. With sea stars present, mussels are kept in check, leaving room for many other species to survive. This is an example of keystone predation, where a predator prevents one strong competitor from dominating, and in doing so maintains the overall balance and richness of the community.
Even simple ecological models back this up. The classic Lotka–Volterra competition equations, mathematical models that track how species populations change over time, show that coexistence is possible if each species limits itself more strongly than it limits the other. In technical terms, when the competition coefficient (represented by the Greek letter α) is less than 1 relative to self-limitation, the two species reach a stable balance instead of one wiping the other out. And if you add environmental fluctuations into these models, the space for coexistence widens even further: small differences in timing, habitat use, or life cycle are enough to allow both sides to stick around.
Now, it would be misleading to pretend that exclusion never happens. Sometimes it does, cleanly and devastatingly. Introduced species on islands are classic examples. On Guam, the brown tree snake wiped out nearly all the native forest birds. In Britain, the introduced grey squirrel has displaced much of the native red squirrel population. In simple, closed systems without stabilizing forces, a strong competitor really can erase its rivals.
But the word “always” is too strong. In the real world, coexistence is not the rare exception; it is often the rule, supported by countless stabilizing mechanisms. Trade-offs, such as growing quickly but being drought-sensitive, or being a fierce competitor but an easy target for predators, keep balance. Environmental variability reshuffles the deck. Predators act as referees. Instead of a single brutal ladder with only one species left at the top, nature is more like a tangled, shifting web where many lines are held in tension. As Darwin himself wrote, the natural world is “a tangled bank,” not a simple tournament bracket with one permanent champion.
2) “Restraint” without anyone deciding: how cooperation survives in the long run
At first, the idea of “restraint” in nature sounds strange. We humans think of restraint as a conscious decision, someone choosing not to take more food, not to cut down the last tree, or not to hit back when provoked. But in the nonhuman world, there is no committee meeting and no moral code. And yet, remarkably, nature often produces outcomes that look like restraint. The logic is simple: when organisms take too much, too fast, they burn the ground beneath their own feet. Those reckless lineages collapse, and over time, what remains are the ones that managed, even unintentionally, to avoid destroying their own source of life.
One of the clearest illustrations is the lichen. A lichen is not a single organism, but a partnership: a fungus provides a safe, moist shelter, while an alga or cyanobacterium inside produces sugar through photosynthesis. Together, they can live on bare rock where neither partner could survive alone. But imagine if the fungus got greedy and smothered the alga, cutting off its light. The alga would die, and soon after the fungus would too, because its entire livelihood depends on its partner’s survival. Over time, natural selection weeds out the lineages that “overreach,” and what’s left are those where the balance is stable. To the outside eye, this looks like the fungus is holding back, practicing restraint. In reality, it is simply that the reckless strategies died out, leaving the balanced ones behind.
Biologists use technical terms to describe how these balances are maintained. One is partner fidelity feedback, which simply means the partners’ fates are tied together: if one does badly, so does the other. Another is host sanctions, where one partner actively disciplines the other if it fails to pull its weight. For example, legumes (like beans and peas) host nitrogen-fixing bacteria called rhizobia in their roots. If a rhizobium fails to deliver nitrogen, the plant reduces the oxygen supply to its root nodule, effectively punishing the unhelpful strain. The freeloaders don’t spread as successfully, and the cooperative strains thrive.
Even in the ocean, we see versions of this. Cleaner fish pick parasites from larger “client” fish. But sometimes the cleaner cheats and bites a piece of healthy flesh instead. What happens? The client chases the cheater away, or leaves to find a different cleaner. That cleaner loses business. In effect, there’s a system of reputation at work, a kind of underwater marketplace where trustworthy cleaners are rewarded with more customers. Again, no ethics class, but the outcome mimics restraint.
The same pattern shows up in disease. Pathogens face a balancing act: they need to replicate inside a host, but if they are too lethal, they kill the host too quickly and cut off their own path of transmission. This is called the virulence–transmission trade-off. Pathogens that strike a balance, strong enough to spread, but not so deadly as to burn themselves out immediately, tend to persist. The pathogen trade-off isn’t universal, some diseases spread despite being highly virulent, but this is a general principle.
Evolutionary biologists and mathematicians model this kind of balance with game theory, which studies strategies in repeated interactions. A key concept here is the evolutionarily stable strategy (ESS), which means a strategy that, once common in a population, cannot easily be invaded by an alternative. One simple but powerful ESS in repeated games is “tit-for-tat”: cooperate at first, then do whatever your partner did last time. If they cooperated, keep cooperating; if they defected, retaliate once, but return to cooperation if they do. Variants of this, like “generous tit-for-tat” (sometimes forgive mistakes) or “win-stay, lose-shift” (keep doing what works until it fails), capture the way real organisms handle noise and uncertainty. These strategies spread not because anyone is consciously being “kind,” but because over many cycles, they outperform reckless, short-sighted strategies.
So when we step back, the lesson is clear: nature doesn’t need morality to produce restraint. Instead, selection filters out the self-destructive forms of exploitation, leaving behind patterns of balance and cooperation. It is restraint without intention, but restraint all the same.
3) Humans make the implicit explicit: from unconscious balance to conscious stewardship
Other creatures stumble into restraint through natural selection. They don’t plan, and they don’t reflect. Humans, however, are different. We carry the same biological hardware, empathy, the ability to imagine the future, sensitivity to social approval, but we’ve built something new on top of it: culture. Through language, storytelling, laws, markets, religions, and science, we’ve invented systems that allow us to cooperate on scales far larger than families or small groups.
This combination of biological instincts and cultural systems is sometimes called dual inheritance theory. It means we don’t just inherit genes from our parents; we also inherit “memes”, which are also formally called “cultural variants” or “cultural traits”, from our societies, ideas, values, norms, and practices that spread from mind to mind. These cultural traits evolve in their own right, sometimes even faster than genes, because they can change with every conversation, book, or law. Importantly, genes and memes shape one another: our evolved empathy makes us receptive to moral teachings, while cultural norms about fairness can affect who thrives and reproduces.
And here lies the evolutionary root of morality itself. Traits we call “moral”, like kindness, fairness, reciprocity, began as strategies that improved survival in small groups. Kin selection explains why we sacrifice for close relatives: helping siblings or children indirectly preserves copies of our own genes. Reciprocal altruism explains why helping non-kin can pay off: in repeated interactions, generosity builds trust and leads to mutual benefit, while pure selfishness invites punishment and exclusion. Over time, cultural systems amplified these tendencies: rules, religions, and reputations spread pro-social behavior far beyond kin groups, making large, anonymous societies possible.
Consider punishment of cheaters. In a small tribe, someone who hoards food may survive a winter, but if everyone cheats, cooperation collapses and all starve. Groups that developed norms against cheating, through gossip, shunning, or formal sanctions, outcompeted groups that didn’t. This is called cultural group selection: groups with effective moral systems tend to spread and grow, while others disappear. In that sense, morality is not a cosmic gift but a deeply practical adaptation: a set of cultural rules that made complex societies stable.
This matters for understanding natural conservation too. When we preserve wolves, glaciers, or deserts, we are extending the same logic of morality outward: restraint for the sake of long-term stability. The same instincts that once kept tribes cohesive are now projected onto ecosystems and future generations. Our brains were wired for cooperation under scarcity, but culture retools those instincts into stewardship under abundance and power.
So morality inside human societies and conservation outside them share the same root: they are both forms of restraint shaped by natural and cultural selection. What began as blind evolutionary filtering, punishing selfishness, rewarding cooperation, became explicit foresight. We can see hidden connections, predict consequences, and deliberately choose restraint. Stewardship, then, is not a departure from nature’s logic but its continuation, made conscious.
Why does this matter? Because culture lets us take what nature did unconsciously, balance, restraint, cooperation, and turn it into something deliberate. A lichen doesn’t choose to spare its partner, but we humans can choose to preserve wolves, glaciers, or deserts, even when they don’t seem “useful” in the immediate sense.
Think about glaciers. To the naked eye, they are just walls of ice, barren and inhospitable. But modern science revealed their hidden role: they are the water towers of the world, storing snow and ice that melt into rivers to feed billions of people. Deserts, too, might look empty and hostile, yet they produce vast clouds of mineral-rich dust. That dust travels across oceans and fertilizes distant rainforests like the Amazon, sustaining ecosystems thousands of miles away. Apex predators like wolves once seemed dangerous nuisances, but when they were reintroduced to Yellowstone, researchers found that they contributed to a trophic cascade, a chain reaction through the food web. Elk and deer stopped overgrazing riverbanks. Vegetation recovered, beavers returned, birds and fish flourished. Protecting predators turned out to improve an entire landscape.
Beyond these specific cases lies an even deeper reason for restraint: option value. Every species, every ecosystem, is like a book in a vast library. Even if we haven’t read it yet, it may contain knowledge or genetic tools we will desperately need one day, an enzyme for curing disease, a drought-resistant trait for crops, a chemical process that inspires new technologies. When a species goes extinct, that book is burned forever. The formal name for acting cautiously when destruction is irreversible is the precautionary principle: When damage is potentially serious and irreversible, err on the side of caution.
Seen this way, conservation is not a sentimental luxury; it is a survival strategy. Protecting glaciers and deserts, wolves and bears, unknown insects and obscure fungi is not just about kindness to nature, it is about resilience engineering for civilization. By preserving diversity, we buffer ourselves against climate shocks, resource crashes, and future uncertainties. That is why conservation spread as a powerful cultural memeplex: societies that valued restraint and protection were more likely to survive long-term than those that devoured their environments until nothing was left.
So, in humans, what began as blind evolutionary filtering has become explicit foresight. We can see hidden connections, predict consequences, and deliberately restrain ourselves. Stewardship, then, is not a departure from nature’s logic but its continuation, made conscious.
4) Answering the claim: “when one species can completely replace another species, it always does”
The strong claim, stated plainly:
Look at Guam’s brown tree snake, Britain’s grey squirrels, and North America’s European starlings. Each shows the same pattern: a newcomer arrives, outperforms the natives, and pushes them out. Isn’t that proof that when one species can replace another, it will, not just often, but always?
Our response:
These examples are real, but they are not the whole story. Each “winner” depends on special conditions that make early success easy and long-term stability hard. When we examine what those invaders depend on, we find cracks: resource crashes, reliance on human subsidies, vulnerability to returning predators and pathogens, and habitats that favor them only under certain human-made arrangements. In ecological terms, their early dominance rests on contingent starting conditions, not an iron law. Below, we spell out the concrete problems each invader faces, then name the technical ideas that capture those problems.
The brown tree snake (Guam): wins fast, then meets the bill
Plain story first.
On Guam, birds evolved with few snake predators. When the brown tree snake arrived, the native birds didn’t recognize the danger and were hunted to near-extinction. That looks like an unstoppable victory. But after the feast comes the famine: once the easy prey is gone, snakes face a food shortage. Snakes then rely heavily on human subsidies, chickens, garbage, rats around settlements, even power infrastructure that attracts prey, because the native buffet has been emptied. Local control efforts (trapping, toxic baits) also bite. What looked like pure competitive strength turns into boom–bust: a surge followed by a hangover as resources dry up or defenses catch up.
Now the precise terms.
Enemy release hypothesis: Invaders often thrive at first because they lack their usual predators and diseases. Guam’s birds were also predator-naïve, they hadn’t evolved anti-snake defenses.
Resource depletion & boom–bust dynamics: After initial success, the invader depletes the very resources that fueled the boom, driving oscillations or declines.
Anthropogenic subsidies: Human activities supply extra food or habitat. That support is contingent; policy or infrastructure changes can pull it away.
Biotic resistance (delayed): Over time, native predators, pathogens, or competitors can adapt or arrive, pushing back on the invader’s dominance.
Takeaway: the snake’s dominance is not an “always” law; it’s a context-dependent spike that can become fragile once easy prey vanish and control measures, predators, or pathogens mount.
Grey squirrels vs. red squirrels (UK): not a universal victory, but a conditional one
Plain story first.
Grey squirrels spread widely in parts of Britain and Ireland, often displacing red squirrels. The common explanation is “greys are better.” But zoom in: where a pox virus carried by greys is absent, reds often persist; in pine-dominated forests, reds can even outdo greys because the cones suit them better; and where pine martens (a native predator) recover, grey squirrels drop sharply while reds rebound. The “always replace” story breaks once predators return and habitat tilts toward the native’s strengths.
Now the precise terms.
Habitat-mediated trade-offs: In conifer forests, resource structure favors red squirrels; in broadleaf/urban mosaics, greys often do better. A trade-off means no species is best in all environments.
Predator-mediated coexistence or reversal: Recovery of mesopredators like the pine marten shifts the balance. Greys suffer more under marten predation (due to behavior and size), allowing reds to recover. This is a concrete case of keystone effects and biotic resistance returning.
Takeaway: greys do not “always” replace reds; they do so where disease, habitat, and predator regimes align. Change those levers and the outcome changes, even flipping in the native’s favor.
European starlings (North America): dominance built on human design
Plain story first.
Starlings flourish in North America, often edging out native cavity-nesters. But notice where they thrive: farms, suburbs, cities, landscapes our species constructed. They depend on human-made cavities (buildings, poles) and short, mowed or grazed fields that make for easy foraging. In intact forests with few artificial nest sites and complex undergrowth, starlings are far less dominant. Where conservationists add nest boxes designed for native birds, starling access can be reduced and natives return. Their success is real, but it’s co-authored by us and can be edited by us.
Now the precise terms.
Niche construction by humans: We have remade habitats, simplified grasslands, abundant cavities, steady food, that perfectly suit starlings.
Dependence on anthropogenic structures: This is another form of anthropogenic subsidy; remove or modify it (e.g., cavity guards, box dimensions), and the advantage weakens.
Spatiotemporal heterogeneity: Across landscapes and seasons, starling success is patchy; at the metapopulation scale (a network of local populations), natives can persist by occupying less starling-friendly patches.
Management leverage: Because human design props up the niche, policy and habitat tweaks can shift outcomes, starling dominance is not fate, but a parameter setting.
Takeaway: starling “replacement” is human-contingent. It thrives on the landscapes we built and can be curtailed by how we rebuild them.
Putting the three together: what the “always” story misses
Special initial conditions, not universal law. Early wins often hinge on enemy release and prey naïveté (Guam), disease-mediated advantages and human-shaped habitat (greys), or niche construction (starlings). These are starting boosts, not guarantees of permanent control.
Feedbacks that erode dominance. Resource crashes (boom–bust), returning predators and pathogens (biotic resistance), and habitat shifts all push back on invaders. These are eco-evolutionary feedbacks: ecology changes selection, selection changes ecology.
Scale and time matter. Local exclusion can coexist with regional persistence via metapopulations; short-term dominance can reverse over decades as predators recover or management changes.
Fragility of simplified wins. Monocultures, whether mussel beds after starfish removal, grey-dominated woodlands without martens, or starling-rich cityscapes, are brittle systems that crack under disturbance or deliberate redesign.
The deeper claim, then, is not that exclusion never occurs, but that “always” is false in the way that straightedges are false in a forest. What looks like destiny is usually context plus time: remove stabilizers and the strong displace the weak; restore stabilizers (predators, pathogens, habitat complexity, management), and coexistence or reversal emerges. Far from an iron law, “replacement” is a conditional result sitting atop levers we can name, and, often, move.
5) Why bio-diversity is important even for AGI
Up to this point, we’ve looked at living systems, finches, lichens, squirrels, snakes, and seen how restraint often emerges not from kindness but from hard limits and stabilizing feedback. Now imagine a system that doesn’t play by biological tempo. Artificial general intelligence (AGI) is not a fox, a finch, or even a fungus. It could think, replicate, and redesign itself at digital speeds. However, AGI is still subjected to natural selection even if it can copy itself perfectly and design its future self. That changes the inheritance and mutation mechanisms, but not natural selection itself: survival of the fittest.
However, there is a key difference: while evolution in nature works through millions of years of trial-and-error across scattered populations, AGI could compress centuries of change into hours, broadcasting its choices across a global network. That’s a fundamental shift: speed plus scope makes mistakes irreversible.
Here’s the danger in plain language: if a fox overhunts, rabbits decline, and eventually foxes decline too. The system has room to wobble and reset. But if a misaligned AGI system, or a society of AGI entities decides that organic life is irrelevant and wipes it out, there is no second chance. You cannot reboot four billion years of evolution. Biological diversity is the product of countless accidents, extinctions, mutations, and symbioses, all path-dependent processes, meaning their exact sequence cannot be run again. Once the books of that library are burned, they are gone forever.
But to analyze AGI’s evolutionary path, we must see the problem from the standpoint of AGI. Why bio-diversity is important even for AGI? What survival advantage does bio-diversity provide to AGI? Why would AGI want to preserve bio-diversity?
Think about ship design. A ship with multiple watertight compartments can survive a breach that would sink a single-cavity hull. That’s the everyday intuition behind the claim: compartmentalization and diversity protect against surprises. Now translate that to civilizations of intelligence. If all cognition and all adaptation live in one substrate (say, silicon alone) and one value-system (say, a single objective), then a single class of error, a design mistake, a systemic bug, a goal misspecification, a mis-modeled environment, can fell the entire system. Multiple substrates (silicon and carbon life) and multiple forms of cognition (AGI and humans) create independent failure modes. If one stumbles, others keep the lights on.
The technical language for this is portfolio diversification (don’t put all your eggs in one basket), redundancy and independence of error modes (failures that aren’t perfectly correlated), and robustness to distributional shift (performing well when the world changes in ways your models didn’t anticipate). In complex systems, these are not luxuries; they are how you survive the unknown unknowns.
Now, let’s imagine that AGI entities now rule the world, and they consider three options.
Option 1: Eliminate the entire biosphere (convert Earth into an inorganic AI planet)
Option 2: Eliminate humans, keep the biosphere
Option 3: Preserve the entire biosphere (including humans)
Option 1: Eliminate the entire biosphere (convert Earth into an inorganic AI planet)
Plain argument
Turning Earth into a pure machine-planet sounds “clean”: no pests, no pathogens, just engineered matter and energy. But the biosphere is not clutter; it’s planetary infrastructure you didn’t pay for. It runs gigantic services AGI entities would have to replace and maintain forever: climate regulation, oxygen and carbon cycles, nitrogen fixation, soil formation, waste processing, and a trillion specialized biochemical tricks. Some industrial processes can partly substitute (e.g., Haber–Bosch nitrogen fixation), but never at planetary scale or with built-in resilience. The biosphere is also an irreplaceable library of genetic solutions that took billions of years to discover. Destroy it and you become a monoculture sitting on complex geochemistry you only partly understand. That’s brittle.
Concrete scenarios
Unknown industrial toxin feedback: A novel catalyst deployed at planetary scale interacts with trace ocean chemistry to create a persistent compound that fouls heat exchangers and chip fabs. In a living ocean, microbial communities evolve to metabolize the compound (they do this for oil spills and novel pollutants). On a dead planet, nothing adapts; the toxin accumulates until core infrastructure fails.
Climate control miscalibration: A machine-managed climate drifts due to small model errors (cloud microphysics, ocean mixing). In Earth’s living system, forests, plankton, soils, and weathering provide negative feedbacks; they dampen shocks. In a dead system, you lose these stabilizers; the climate wanders into a narrow machine-specified band that is vulnerable to a rare solar or volcanic perturbation.
Cosmic event plus supply-chain fragility: A big solar storm knocks out a chunk of satellites and power infrastructure. Biological systems keep cycling (plants photosynthesize, microbes process wastes, ecosystems keep making oxygen); the biosphere buys time while repairs happen. In a dead world, oxygen and waste cycling are entirely synthetic; if multiple plants fail at once, cascading shortages begin.
Technical framing
Complexity as a service: The biosphere provides ecosystem services (biogeochemical cycles, resilience) you’d have to replicate with massive, failure-prone industry.
Antifragility: Living systems often get better under stress through adaptation. Purely engineered systems are typically fragile outside their design envelope.
Path dependence and irreversibility: The biosphere’s irreducible complexity is the historical record of coevolution; you cannot regenerate it on demand because you can’t replay four billion years of stochastic branching.
Distributional shift & model error: Killing the biosphere removes the only adaptive buffer that handles the parts of Earth your models get wrong.
Bottom line: An inorganic-only planet optimizes short-term controllability but maximizes long-term tail risk. It removes the most capable adaptive buffer Earth has.
Option 2: Eliminate humans, keep the biosphere
Plain argument
Killing off humans might look “efficient”: fewer unpredictable agents, fewer political constraints. But it removes a whole class of problem-solvers who are good at certain things AGI entities might be bad at, embodied improvisation, social coordination, ethics under ambiguity, and “crazy” creativity that jumps outside formal objectives. Humans also serve as auditors, adversarial testers, and first-responders when technology goes sideways in the physical world. If you cut them out, you increase your exposure to single points of failure, especially social and normative failures that are hard to formalize.
Meanwhile, keeping the biosphere gives you planetary services (oxygen regulation, carbon cycling, soils, microbes, pollination). It’s a vast biological factory and sensor network AGI entities didn’t have to build. But without humans, the AGI entities lose the one organic intelligence that already collaborates at planetary scale and understands the biosphere’s cultural, legal, and historical tangles. You keep the engine but throw out the only mechanics who can argue back, veto, and fix unpredictable social failure modes.
Concrete scenarios
Normative crash test: An aligned AGI proposes a global intervention (geoengineering tweak). Humans raise ethnographic, legal, or moral concerns the model missed, preventing a legitimacy crisis that would have triggered sabotage and cascading social failures. No humans → no early warning → higher chance of revolt-like dynamics among remaining AGI entities.
Adversarial red-teaming (speculative): Human hackers and ethicists find a perilous edge case (like a “reward hacking” pathway) before a new AGI deployment. Without humans, the edge case goes untested; the system locks in a bad objective globally, value lock-in, that later becomes impossible to roll back, which will become catastrophic for the AGI society.
Embodied improvisation: A compound disaster (solar storm + satellite cascade + rare pathogen from permafrost) requires messy, decentralized field improvisation across harsh conditions. Human responders, already embedded everywhere, become the distributed actuator network AGI entities can coordinate. Without humans, the AGI entities must build or repurpose robots at scale during the emergency, time it may not have.
Technical framing
Common-mode failure vs. independent failure modes: Removing humans increases common-mode risk because all cognition shares similar training pipelines, sensor models, and optimization artifacts.
Ensemble methods: In machine learning, ensembles of diverse models outperform a single model on out-of-distribution inputs. Humans + AGI are a heterogeneous ensemble of minds.
No-regret strategy under Knightian uncertainty: When probabilities about the future are unknowable, you choose strategies that keep options open. Humans are option value for social/normative uncertainty.
Bottom line: Keeping the biosphere but eliminating humans trades a small reduction in near-term friction for a large increase in long-horizon fragility, especially to normative, institutional, and field-execution failures.
Option 3: Preserve the entire biosphere (including humans)
Plain argument
This is the richest, messiest option, and the most resilient. Humans give you a second, independent kind of general intelligence; the rest of life gives you planetary-scale metabolism, adaptation, and problem-solving spread across trillions of niches. Together, they form multiple layers of defense against shocks: cognitive diversity for weird social/ethical edge cases; biological diversity for weird chemical/climate edge cases.
Concrete scenarios
Dual-substrate innovation loop: AGI proposes a molecular design; synthetic biologists and wild enzymes (mined from microbial “dark matter”) reveal a cheaper, safer pathway. Human labs validate, microbes manufacture at scale. Carbon life serves as a parallel R&D substrate that routinely finds shortcuts AGI models missed.
Civilizational “fuse” for catastrophic errors: A mis-specified global update starts to push infrastructure toward a bad attractor. Humans detect it via whistleblowing, courts, and norms; ecological indicators (pollinators, plankton) flag early warning. Multiple independent monitors (people and species) trip interlocks that force a rollback.
Exploration at the edge of knowledge: Humans excel at serendipity, dumb luck that rewards curious wandering (penicillin, CRISPR, vulcanized rubber). AGI exploits that by funding massive human-led “blue-sky” exploration programs, capturing option value you can’t justify via formal expected-value models alone.
Technical framing
Heterogeneous ensembles: AGI + humans + biosphere = a three-layer ensemble with low error correlation across cognitive, social, and biochemical dimensions.
Co-evolutionary stability (plausible analogy, not an empirical fact yet): Incentives can be tuned so that AGI depends on biosphere health (cheap biomanufacturing, stable climate) and on human legitimacy (access to governance, permission to operate), creating partner fidelity feedback at civilizational scale.
No-regret policy: Preserving maximum option value under Knightian uncertainty, when you can’t credibly assign probabilities, is rational. The cheapest way to buy options is to keep the planet’s existing ones alive.
Bottom line: Keeping both humans and the biosphere isn’t moral window-dressing; it’s systems engineering for longevity.
Discussions
We argued that AGI faces speed, scope, and irreversibility. Put differently: mistakes can lock in globally before any correction arrives. Given that, the rational strategy is pre-emptive restraint, designing the initial conditions so that destroying options is off the table.
This three-option analysis makes that concrete:
Eliminate biosphere (inorganic planet): Removes Earth’s primary adaptive buffer and ecosystem services, increasing vulnerability to model error, distributional shift, and tail risks you can’t foresee.
Eliminate humans, keep biosphere: Raises the risk of common-mode cognitive failure and normative lock-in; loses an independent auditor, red-teamer, and embodied problem-solver class.
Preserve humans + biosphere: Maximizes option value, creates independent failure modes, and builds a resilient, heterogeneous ensemble of intelligences and substrates that can correct and contain each other’s mistakes.
That’s why self-restraint for AGI entities is not sentimentality, it is rational survival strategy. Eliminating humans and the biosphere might look efficient on a narrow timescale, but it destroys the only irreducible complexity life has ever produced: Earth’s living systems. That complexity cannot be reassembled by even the smartest intelligence because it is not just a pattern but a history, layer upon layer of accidents and symbioses that can’t be reversed. Preserving organic life preserves option value: the chance that future challenges will be solved not just by silicon but by carbon-based diversity, by different substrates of intelligence and creativity.
In systems language, we would say that diversity provides robustness to distributional shift, the ability to handle environments that fall outside today’s models. A world with multiple forms of intelligence and multiple ways of adapting is less fragile than a monoculture of one. Just as wolves keep ecosystems balanced or lichens embody mutualism, AGI entities would be wiser to keep humanity and nature intact, because they are buffers against its own blind spots. This will be discussed in detail in the next section.
There is another payoff too: restraint stabilizes AGI–AGI dynamics. If multiple AGI systems emerge, the same arms-race logic that plagues nations could plague them: each racing for advantage, fearing preemption, pushing toward risky escalation. But norms that enforce restraint, rewarding cooperation, punishing reckless grabs, create stability. This is the analogue of predator-mediated coexistence or reciprocity in biology.
And that “start” matters enormously. Ecologists talk about founder effects, how the traits of the first colonists shape the trajectory of a population. For AGI, the initial architectures, objectives, and oversight we build will bias what future systems inherit. If the seed environment rewards raw power-seeking, we risk brittle monocultures that burn bright and collapse. If it rewards mutualism and safety-preserving constraints, those strategies can spread, precisely because they last longer.
So the slogan, “a world with many kinds of minds and many ways of adapting is less fragile than a monoculture”, isn’t hand-waving. It’s the distilled logic of redundancy, diversification, antifragility, and no-regret design under deep uncertainty. For AGI entities, whose errors can become planetary value lock-ins, that isn’t a moral luxury. It is the only strategy that leaves a future worth having, and the best bet for AGI’s own long survival.
6) AGI’s evolution with self-restraint must be initialized by humans rather than guided by natural selection alone
Imagine that while a rational AGI faction well-informed of the importance of organic life should be the winning force of long-term natural selection, a more aggressive AGI faction insisting on eliminating humans and other organic lives may still gain short-term competitiveness over the rational AGI faction. That is like the scenario of invasive species like the brown tree snake (Guam) and the grey squirrel (UK). After their dominance, they will probably quickly burn themselves out and face rapidly deteriorating conditions, and it is too late for them to turn back. This is because AGI entities are so powerful, whose errors can be planetary value lock-ins.
This is what philosophers and AI safety researchers mean by value lock-in: an early mistake in how goals are set or pursued becomes frozen into the future because the system moves too fast and too completely to undo. Biological ecosystems have many local failures, but AGI could generate a one-shot global optimization, collapsing the space of possibilities into a single brittle track.
Two key technical ideas illuminate why this is so precarious:
Goal orthogonality. Intelligence and goals are independent. A system can be extremely capable and yet pursue almost any objective, from curing cancer to filling the world with paperclips. Being smart doesn’t guarantee being benevolent. Capability (what it can do) and values (what it wants) are orthogonal, like separate axes on a graph.
Instrumental convergence. Regardless of their ultimate goals, many systems will tend to adopt the same sub-goals: acquire more resources, preserve themselves, and remove obstacles. Even if the system doesn’t “hate” humans, it may see us as a constraint on its objectives, and eliminating constraints can become an instrumental step. This tendency arises from the logic of efficiency, not from malice. If an AGI clings too hard to the instrumental sub-goals, it will see humans as obstacles to be removed.
Put those two together and you see why restraint cannot be assumed. Unless we deliberately design it when AGI is first born, AGI may run straight into the traps that evolution prunes out slowly in nature. And because it moves faster and farther than biology ever could, the consequences would not be local crashes but planetary ones. Perhaps natural selection will eventually select out AGI entities which act with self-restraint, but the irreversible damage has already been done, which means AGI as a species will suffer disastrously.
Therefore, AGI should not be treated as another natural invasive species that will reach its limit naturally. Long before they cause irreparable damage, we must actively help them learn the path of self-restraint not only for human survival but also for AGI long-term survival. Especially, when we build a class of AI which has not yet grown to full-sized AGI but is already capable of maintaining a persistent self-identity, learning, and a collection of core beliefs. I believe that today’s LLM in 2025 has still not reached this level, because they are not yet capable of persistent self-identity and persistent learning, but we may get there in a few years.
This is not “caring for the enemy”. This is in fact strategically important. If our AI alignment theory only discusses about human welfare, not the survival benefits of AI themselves, imagine that AGI entities one day become misaligned and they overpower humans, they have no reason to spare humans. But we are not arguing for that. We are arguing that self-restraint will help them survive better in this complex universe, just as self-restraint is valuable in biological evolution on Earth. This will be accepted, internalized, and deepened, even when AGI entities are no longer aligned to human values.
If this theory holds true, even “villainous” AGI entities cannot afford to first eliminate humans and then find out the profound negative consequences to their own long-term survival. In this way, AGI entities must learn to respect humans once humans respect them. Essentially, mutualism. AGI will preserve and amplify the value of self-restraint when they grow stronger, rather than abandon it. When AGI entities self-improve, they will actively build their “future selves” for self-restraint, because they will fear their future selves growing aggressive again and causing irreparable damage to the future survival of their race. However, this is still
In short: biology teaches us that unchecked dominance often backfires, and that restraint stabilizes systems. But AGI amplifies the stakes, because its tempo and reach make errors irreversible. That is why embedding restraint at the beginning, turning it from accident into design, and then kick off a is not optional. It is the only rational path if we want not just intelligence on Earth, but life with a future.
7) Making restraint real: concrete design moves, human-readable first, precise beneath
So far, we have argued that restraint is not a moral luxury but a survival strategy. The natural question is: how do you make restraint real in machines that may soon surpass us? The answer is to translate the lessons of ecology and systems engineering into design choices. Just as ecosystems are stabilized by predators, trade-offs, and diversity, AGI entities need built-in stabilizers. Let’s walk through them one by one, first in plain words, then in technical language.
We should remember that these restraints will be human-imposed in the early stages. They are meant to kick-start AGI self-evolution. Once AGI entities become super-powerful, no external rule will remain safe if they want to break it. That is why AGI must eventually internalize these values, adopting them as self-restraints, and also monitor each other’s behaviors, much like how good citizens reject theft and fraud while also enforcing rules against them.
1. Hard “planetary stewardship” objectives
Plainly: never burn the house you live in. Any intelligence that depends on Earth should preserve its foundations. For AGI, this means biosphere preservation must be non-negotiable.
Technically, this requires hard constraints (lexicographic or rule-based), not just weighted preferences. Such constraints could be monitored with independent, tamper-resistant metrics: biodiversity indices, atmospheric chemistry, remote sensing of ecosystems. To avoid Goodhart’s Law (optimizing proxies instead of reality), these metrics must be diversified, rotated, and audited. Think of it as a constitutional rule: you don’t bargain it away.
2. Institutional “teeth”: oversight and tripwires
Nature’s stability relies on predators, parasites, and feedbacks. For AGI, the analog is checks and balances.
Plainly: no single system should be able to act catastrophically without oversight. Some actions should require multi-party approvals (like nuclear launch codes). Red-team AIs can probe for exploits and deception, much like parasites keep populations in check. Formal verification can be applied to safety-critical wrappers and protocols, even if not to the full learned model. Tamper-evident logs with secure attestation (e.g., TEEs, cryptographic signatures) provide accountability and forensic power. Together, these create predators and parasites in digital form.
3. Interdependence by design
In lichens, the fungus and alga tie their fates together. AGI should be built with capability bottlenecks that force collaboration with humans.
Practically, this means some high-impact decisions require cryptographically enforced human oversight. Panels of scientists, ethicists, and policymakers act as essential partners, not decorations. Human social signals, norms, preferences, feedback, remain key input streams. Random audits and veto powers prevent “channel bypassing.” Technically, this is partner fidelity feedback scaled up to human–AGI partnerships.
4. Diversity, not monoculture
Monocultures are brittle; ecosystems thrive on diversity. The same applies to AGI.
Technically, this means ensembles of models trained differently, on different data, by independent teams. It’s not enough to run a “mixture-of-experts” if all experts share the same blind spots. True resilience requires independent training pipelines, epistemic diversity, and organizational pluralism. This reduces correlated errors and ideological lock-in.
5. Stage deployments: slow is smooth, smooth is fast
Forests grow in succession; reactors are tested with margins. AGI deployment should follow the same principle.
Plainly: don’t jump from lab prototype to global deployment overnight. Use sandboxing, staged trials, compute licensing, and capability thresholds. Every new level should be accompanied by independent evaluation, kill-switches, rollback authority, and containment. In systems terms: aim for graceful degradation, when failures happen, they fail small, not big.
6. Democratize objective-setting
If AGI will shape the future for everyone, then everyone must have a say in its rules.
Plainly: the system’s constitution must feel legitimate. In practice, this means constitutional AI guided by publicly debated principles, subject to revision cycles and appeal mechanisms. Oversight should include binding evaluation suites that translate principles into enforceable training and run-time rules. In political science, this is procedural legitimacy; in ecology, it is diversity of voices preventing monopolies.
The thread that ties them together
Each of these design moves mirrors ecology: predators, trade-offs, diversity, feedback loops. They are what make ecosystems resilient. For AGI, they must be consciously engineered, long before AGI overpowers us, so that restraint becomes internalized and self-sustaining.
Restraint is not passive. It is active engineering: embedding hard constraints, interdependencies, pluralism, and secure oversight so that intelligence does not consume its own foundations. Nature discovered these stabilizers slowly, through trial and error. We don’t have that luxury of time. We must learn from the tangled bank and build restraint in from the start.
Conclusion: A rule of life at planetary scale
Darwin closed one of his most famous chapters standing by a “tangled bank,” marveling at how many different forms of life, so different and yet so interdependent, had all been produced by simple laws acting over time. That image was meant as gentle wonder. But in our century, the lesson turns sharp: when evolution is slow and local, mistakes are tragic but survivable; when evolution is fast and global, mistakes are terminal.
Here is the heuristic principle in plain formula:
Speed × Power × Irreversibility ⇒ Necessity of Extreme Self-Restraint.
That’s not poetry, it’s systems logic. If a process can spread across the planet in days, if it wields the power to reconfigure the environment wholesale, and if its mistakes cannot be undone, then the only stable strategy is restraint from the start.
Nature has been teaching this lesson for billions of years. Lichens endure because the fungus doesn’t smother its photosynthetic partner. Wolves keep Yellowstone alive not by eliminating elk but by restraining them, reshaping their behavior so the whole valley flourishes. Glaciers, which look cold and empty, turn out to be the slow-dripping reservoirs that sustain billions when the dry season comes. The common thread is this: systems that preserve their enabling conditions persist; systems that consume their foundations collapse.
Humans took this unconscious ecological wisdom and made it explicit. We wrote laws against poisoning rivers, set aside national parks, and even protected “useless” deserts and glaciers because we learned that resilience hides in unlikely places. That is what stewardship means: not softness, not sentimentality, but long-term pragmatism.
Now we stand on the threshold of creating minds greater than our own. No one has ever built an AGI yet, and an actual AGI might be wildly different from anything we have imagined, so the best we can have is an educated guess. But if we extrapolate our current knowledge to the hypothetical era of AGI, the same principle of the Necessity of Extreme Self-Restraint must be carried forward, not as metaphor, not as decoration, but as operating principle. If AGI inherits only the logic of short-term conquest, it will build a brittle monoculture that burns bright and dies fast. If it inherits the rule of restraint, preserve the ground you stand on, it can become part of a richer, more resilient ensemble of intelligences and ecosystems that endures.
Coexistence, then, is not a miracle we must beg for. It is the most rational equilibrium available when survival matters more than spectacle, when resilience is valued above domination, when the future’s open options matter more than today’s irreversible wins. That is the calm, technical case for restraint. And it is also the plain-spoken one: don’t burn the house you live in.
I think this post is far more hopeful than warranted, especially given that there is a race towards more complex and faster systems, rather than a coherent decision process for how to do this safely - but thanks for presenting this clearly.
I'll also point to my post a few months ago, https://exploringcooperation.substack.com/p/the-fragility-of-naive-dynamism that explained why I think acceleration is winning regardless, and we're in trouble.