Against optimism
The whiggish blindness of Dario Amodei
Dario Amodei runs Anthropic, one of the larger AI startups: they publish Claude, one of the very best, which I use for coding help1. He’s obviously very intelligent, well-informed across a wide range of topics beyond and outside computer science, thoughtful and – unlike so many of his peers – not in the least sociopathic. Last autumn he published a 16,000 word essay on the future of AI which stands now as a kind of Ozymandian relic of the world of yesterday.
What he has to say about the technical possibilities of AI is fascinating and thought-provoking and much of it may well come true. But his view of the political and social frame within which it will be developed – well, it’s like looking back on 1913 from a vantage point ten years later. He thought then he was a pessimist: in fact he wrote
Unfortunately, I see no strong reason to believe AI will preferentially or structurally advance democracy and peace, in the same way that I think it will structurally advance human health and alleviate poverty ... the triumph of liberal democracy and political stability is not guaranteed, perhaps not even likely, and will require great sacrifice and commitment on all of our parts, as it often has in the past.
He really had no idea. Read that passage again and remember that the American electorate has rejected anyone who offered them sacrifice and commitment ever since Ronald Reagan beat Jimmy Carter in 1980 and most recently chose a complete psychopath who thought that not catching the clap in 1980s New York was the equivalent of risking his life in Vietnam.
There is a particular quality to his whiggish optimism which I have come to think of as the fallacy of Dawkins’ metaphor. People who don’t understand biology think of DNA as an active, almost purposive agent which directs the rest of the system when it is more nearly or just as much a passive reserve of strategies and capabilities which the system uses as and when it requires for purposes of its own; in the same way, people who don’t understand society think of it as something driven by technology, and if you don’t understand people, you think they are driven by their intellect and conscious purposes.
Amodei clearly does understand biology so he wouldn’t make those mistakes about literal DNA, but the gnostic view underlying it, in which spirit or intellect animates brute matter, seems to permeate his thinking. He takes for granted that the threat to liberty and decency is primarily technological: “AI seems likely to enable much better propaganda and surveillance, both major tools in the autocrat’s toolkit.”
He hopes for
“a world in which democracies lead on the world stage and have the economic and military strength to avoid being undermined, conquered, or sabotaged by autocracies, and may be able to parlay their AI superiority into a durable advantage… The vision of AI as a guarantor of liberty, individual rights, and equality under the law is too powerful a vision not to fight for.”
This reminds me of the people who believed that the security of encrypted communications under a dictatorship was about the algorithms used, and not whether the secret police would rape your relatives in front of you if you did not give up the password2.
So, the first mistake is that he didn’t last autumn foresee the onrushing Trump coup. There is the more general contradiction that he believes both that advanced AI will confer world domination on whichever power bloc gets it first, and that this power will be used to the general good. That is not how Europe used the benefits of the last industrial revolution.
Perhaps this blindness comes easily when you live in San Francisco, with your eyes fixed on horizons filled with promise, not the streets all paved with dying junkies.
It’s not as if Amodei is a bad or callous man, who thinks that there can’t be a problem. He’s obviously someone who cares about the world, with whom it would be fascinating to talk. He knows there can be problems; he’s just confident they can be overcome:
“With advanced health interventions and especially radical increases in lifespan or cognitive enhancement drugs, there will certainly be valid worries that these technologies are ‘only for the rich’ [but] developed world political institutions are more responsive to their citizens and have greater state capacity to execute universal access programs—and I expect citizens to demand access to technologies that so radically improve quality of life.”
Apparently no one has told him about the American health care system.
Modern technological medicine is wonderful. It makes my life possible – my heart would have killed me without a valve replacement; I can see to type this because I wear spectacles and every morning I put in hearing aids so as not to be maddened by tinnitus. No doubt the pills that I take every day will also prolong my life. But access to all these things are dependent on social arrangements as much as on their technological possibility. I could never have afforded them had I lived in the USA, or almost anywhere in fact outside of Western Europe. The DNA is nothing without the chromatin, and the effects it has are entirely dependent on the surrounding cells.
I suppose my deepest disagreement with Amodei is that he does not believe in original sin, or what Francis Spufford calls the human propensity to fuck things up. No matter how smart the drugs we take, and no matter how thoroughly we were screened as embryos, we will always be dissatisfied, because our hearts want things which are incompatible with each other and with other people’s conflicted desires too.
Amodei looks forward to a world in which
“AI-accelerated biology will greatly expand what is possible: weight, physical appearance, reproduction, and other biological processes will be fully under people’s control. We’ll refer to these under the heading of *biological freedom:* the idea that everyone should be empowered to choose what they want to become and live their lives in the way that most appeals to them. There will of course be important questions about global equality of access …”
Leave to one side the question of whether we ought to want or work towards this meat lego gnosticism3. Suppose for the sake of argument that it’s a good thing that “everyone should be empowered to choose what they want to become and live their lives in the way that most appeals to them” – why does he think this is even logically possible?
How can anyone look around the world and suppose that everyone living as Elon Musk does, or Donald Trump, is even possible, let alone desirable? From Shakespeare to Springsteen, the message is the same: Appetite is an universal wolf and “Poor man wants to be rich; rich man wants to be king. King ain’t satisfied till he rules everything”. The freedom of kings is not the kind of freedom that progressive optimists imagine.
Putting these grumbles aside, the essay’s worth reading because he has some powerful and convincing arguments for the things that powerful AI can do. The one that impressed me most should be obvious but is not: once you have your model built and trained, it’s software, and can be almost infinitely duplicated. These pretrained duplicates are much less demanding of hardware than their progenitor. So you can build a society of co-operating AIs for not much more than it cost to build the first one, and with a huge gain in power. Amodei imagines a million of them, all co-operating, and calls this “a country of geniuses” The implications are far reaching and most of us have not thought them through at all.
I can think of hundreds of scientific or even social problems where a large group of really smart people would drastically speed up progress, especially if they aren’t limited to analysis and can make things happen in the real world (which our postulated country of geniuses can, including by directing or assisting teams of humans).
A surprisingly large fraction of the progress in biology has come from a … few discoveries per decade which have enabled both the bulk of our basic scientific understanding of biology, and have driven many of the most powerful medical treatments.
It’s my guess that powerful AI could at least 10x the rate of these discoveries, giving us the next 50-100 years of biological progress in 5-10 years.
Amodei’s definition of “powerful AI” is something that is “smarter than a Nobel Prize winner across most relevant fields – biology, programming, math, engineering, writing, etc. This means it can prove unsolved mathematical theorems, write extremely good novels, write difficult codebases from scratch, etc.”
One of these things is not like the others. Writing extremely good novels requires a kind of intelligence and awareness of the world which seems to me entirely distinct from excellence in coding or in mathematical insight. There have been writers of great cogency and force who also excelled in mathematical and logical reasoning – Sydney Brenner comes to mind; and Bertrand Russell even won the Nobel Prize for Literature, but that was for a book denouncing monogamy, not for a novel exploring what adultery might mean in human lives – and Russell’s record as a husband and father was distinctly spotty. So was Tolstoy’s, for that matter, but people still read Anna Karenina, while Marriage and Morals is almost entirely forgotten.
I do not think that pure intelligence of the sort that Amodei recognises can do justice to the richness and complexity and of our lives in the ways that the imagination of great artists can do. I do not believe that the human loss function can be modelled in software.
But my own beliefs about this don’t matter, and neither do yours, gentle reader. We don’t have the money to make them matter. The people who do dream at best like Amodei and at worst like Elon Musk. Not that their money will buy them control over the technologies that they unleash and in ten years their dreams won’t matter either. But that’s another story.
Though when I asked it to improve the little routine I use to delete to the beginning of a misbegotten sentence when I am writing here, it made a completely elementary mistake, and when it had fixed that failed to see (as I had) the simplest and most elegant way through the problem.
This was the technique employed by Assad’s secret police in Syria.
Mary Harrington’s phrase
I agree with you about novels. (I would say that, wouldn't I?) But I'm not convinced, either, by Amodei's idea that AIs could do the work of science, or at least the genuinely pathfinding, breakthrough-making part of science. Doing pattern-recognition on a more-than-human scale, yes; systematically exploring vast possibility spaces we couldn't get into before, yes. But those are both ways in which AI can tease out unnoticed properties of data we've already got, things we already know (but don't know we know, etc). I like Henry Farrell's characterisation of AI as a social technology of knowledge, a way of conveniently indexing and making available a digest of what's already thought. (Errors included.) What that won't do, since it isn't a matter of probabilistic links between words and other symbols, is give us things that haven't been thought before. (Though it may show us things
*implicit* in existing thinking that had been hiding in plain sight.) To me, the danger here is not one of AGI taking over the world, or any of that shit, but of AI hopelessly disrupting the models of apprenticeship and human effort without which you can't get to the genuinely new stuff. Without the effortful mastery of the state of the art as it presently exists in any domain, you can't put yourself in a position to do the genuinely original next thing. Why would you spend years laboriously becoming mediocre, as a necessary stage on the way to becoming good, if AI will do it for you, frictionlessly? I think widespread adoption of AI is likely to be a recipe for human deskilling, and therefore for cultural stagnation.