If you prefer to listen rather than read, this blog is available as a podcast here. Or if you want to listen to just this post:

Or download the MP3

A few weeks ago I came across a book review for The Wizard and the Prophet by Charles Mann. I haven’t had a chance to read the book, but it seemingly presents an interesting way of categorizing our two broad approaches to preparing for the future, and harnessing new technology.

According to the instructions on the tin, The Wizard and the Prophet is meant to outline the origin of two opposing attitudes toward the relationship between humans and nature through their genesis in the work and thought of two men: William Vogt, the “Prophet” polemicist who founded modern-day environmentalism, and Norman Borlaug, the “Wizard” agronomist who spearheaded the Green Revolution. Roughly speaking, Wizards want continual growth in human numbers and quality of life, and to use science and technology to get there: think Gene Roddenbury’s wildest dreams, full of replicators and quantum flux-harnessing doodads that untether us from our eons-long project of survival on limited resources and allow us to expand limitlessly. “Prophets” believe that we can’t keep growing our population or impact on the world without eventually destroying it, and ourselves along with it. Their ideal future is like one of those planets the Federation ships would Prime-Directive right over, where humankind scales back and lives in harmony with the land, taking just enough to sustain our (smaller) numbers and allowing the intricate web of human and non-human creatures to flourish.

This idea of dividing people into “Prophets” and “Wizards” intrigued me, particularly since it’s a distinction I’ve been making since my very first post in this space, though of course I didn’t use those terms. But I did point out that the modern world is racing towards one of two destinations, on the one hand, a technological singularity that changes everything for the better and, on the other hand, a catastrophe. Both are possible outcomes of our increasing mastery of technology. And one of the most important questions humanity faces is which destination will we arrive at first?

From the review it appears Mann approaches this question mostly from the perspective of the environment, with particular attention on carrying capacity, but I think the two concepts are useful enough that we should broaden things, using the label of Wizard for those who think the race will be won by a singularity, and the label of Prophet for those who think it will be won by catastrophe. Not only does broadening the terms make them more useful, but I also think it’s in keeping with the general theme of the book.

Of course, in that first post and in most of the posts following it, I have been on the side of the Prophets. The review takes the side of the Wizards. And indeed the Wizard side is pretty impressive. The quote mentioned the Green Revolution which probably saved the lives of a billion people. To this we could add the billion people saved by synthetic fertilizers, the billion people saved by blood transfusions, and the billion people saved by toilets. If we wanted to further run up the score we could add the millions saved by antibiotics, vaccines and water chlorination. With numbers like these, what possible reason could anyone have for not being on the side of the Wizards?

It gets even worse for the Prophets. I was recently listening to a podcast and the host was interviewing Niall Ferguson. Ferguson was on to promote his new book Doom: The Politics of Catastrophe. In the course of the interview he pointed out that when it comes to the most extreme claims of the Prophets, namely a total apocalypse, they have been wrong 100% of the time. That essentially in every age and among every people there have been predictions of apocalypse and armageddon, and no matter the time or the person they’ve all been wrong. So given all of the foregoing why on earth would I choose to defend the Prophets?

In order to answer that question we’re going to need to break things down a little bit. There’s a lot of things tied up in the labels “Wizard” and “Prophet”, and it’s easy to declare one the victor if if you only consider what has happened already and don’t consider what might happen, but once you start looking into the future (which is precisely what Prophets are doing) then the situation becomes far less clear. To illustrate, let me turn to another one of my past posts, and the metaphor of technological progress as an urn full of balls.

Imagine there’s an urn. Inside of the urn are balls of various shades. You can play a game by drawing these balls out of the urn. Drawing a white ball is tremendously beneficial. Off-white balls are almost as good but carry a few downsides as well. There are also some gray balls and the darker the gray the more downsides it carries. However, if you ever draw a pure black ball then the game is over, and you lose.

This is a metaphor for technological progress which was recently put forth in a paper titled, The Vulnerable World Hypothesis. The paper was written by Nick Bostrom, a futurist whose best known work is Superintelligence… [He also came up with the simulation hypothesis.]

In the paper, drawing a ball from the urn represents developing a new technology (using a very broad definition of the word). White balls represent technology which is unquestionably good. (Think of the smallpox vaccine.) Off-white balls may have some unfortunate side effects, but on net, they’re still very beneficial, and as the balls get more grey their benefits become more ambiguous and the harms increase. A pure black ball represents a technology which is so bad in one way or another that it would effectively mean the end of humanity. Draw a black ball and the game is over.

This metaphor allows us to more accurately define what distinguishes Wizards and Prophets. Wizards are those who are in favor of continuing to draw balls from the urn, confident that we will never draw a black ball. Prophets, on the other hand, are people who think that we will eventually draw a black ball, or that, on balance, the effect of continuing to draw balls from the urn is negative i.e. we will draw more dark gray balls than white balls. Viewed from this perspective whether you have any sympathy for Prophets depends in large part on whether you think the urn contains any black balls. Accordingly, stories about the amazing white balls which have been drawn, like the green revolution and vaccines and all the other stuff already mentioned, are something of a distraction because it doesn’t matter how many white balls you draw out of the urn, that can never be proof that there are no black balls. And of course Prophets are not opposed to white balls, they just know that if we ever draw a black ball the game is over.

To be fair there is one other possibility. More recently some of the Wizards have started to argue that it’s also possible for the urn to contain a ball of such surpassing whiteness that it also ends the game, but with a win, instead of a loss. That rather than permanently destroying us it permanently saves us. This permanent salvation would, by definition, be a singularity, though not all singularities ensure permanent salvation. But put in terms of the metaphor, my point from the very beginning is that we have been playing the ball drawing game for quite a while and eventually we’re probably going to draw one or the other, and I not only do I think drawing a pure black ball is more likely than drawing a pure white ball. I think that even a small chance of drawing a pure black outweighs even a large chance of drawing the pure white ball. To show why takes us into the realm of something else that’s been part of the blog from the beginning. The Ideas of Nassim Nicholas Taleb

Most of the balls we draw from the urn, particularly those that are very dark or very white, are black swans. I’ve already linked to the whole framework of Taleb’s philosophy but for those that don’t want to follow the link but still need a refresher: black swans are rare events with three qualities:

  1. They lie outside the realm of regular expectations
  2. They have an extreme impact
  3. People go to great lengths afterward to show how they should have been expected.

Technological progress allows us to draw more balls, which means there are more black swans. More things that “lie outside the realm of regular expectations”. The word “regular” is key here. Regular is the world as it was, the world we’re adapted for, the world we were built to survive in. This “regular” world also had positive and negative black swans and in fact may have had even more negative black swans, but since it didn’t involve the ball-drawing game, this regular world didn’t have to worry about black balls. We may not have been thriving, but there was no chance of us causing our own extinction either. Another way of saying this is that we already had the pure white ball. We had developed sufficient technology to assure our permanent salvation.

Part of the reason for this is that whatever the frequency of black swans, they were less extreme. The big thing capping this extremity is that they were localized. Until recently there was no way for there to be a global pandemic or a global war. This takes us to the second attribute of black swans: their extreme impact. Technology has served to increase the extremity of black swans. When the black swans are positive, this is a very good thing. No previous agricultural black swan ever came close to the green revolution, because a change of that magnitude was impossible without technology. It’s the same for all of the other Wizardly inventions. In their hands technology can do amazing things. But the magnitude of change possible with technology is not limited only to positive changes. Technology can make negative changes of extreme magnitude as well. In allowing us to draw all these fantastic white balls, it also introduced the possibility of the pure black ball. A negative black swan so bad we don’t survive it. A point we’ll return to in just a moment, but before we do that let’s finish out our discussion of black swans.

The third quality of a black swan is that in retrospect they seem obvious. When it comes to technology this quality is particularly pernicious. Our desire to explain the obviousness of past breakthroughs leads us to believe that future breakthroughs are equally obvious. That because there was one green revolution, and in retrospect its arrival seems obvious, that the arrival of future green revolutions whenever we need them are equally obvious. Somewhat related to this having demonstrated that we should have expected all previous advancements, because someone somewhere imagined they would come to pass, Wizards end up confusing correlation with causation and assume that anything we can imagine will come to pass. And in doing so they generally imagine that it will come to pass soon. You might be inclined to argue that I’m strawmanning Wizards, when in actuality I’m doing something different. I’m using this as part of my definition of what makes someone a Wizard as opposed to just, say, a futurist. They have a built in optimism and faith about technology.

A large part of the Wizard’s optimism derives from the terrible track record of the Prophets, which I already mentioned. Out of the thousands of times they’ve predicted the actual, literal end of the world, they’ve never been right. However when it comes to their record for predicting catastrophes short of the end of the world, they’ve done much better. Particularly if we’re more concerned with the how, than the when. Which is to say while it’s true that Prophets are often quite premature in their predictions of doom, they have a very good record of being right eventually.

This point about eventually is an important one because above and beyond all the other qualities possessed by black swans the biggest is that they’re rare. So the role of a Prophet is to keep you from forgetting about them, which because of their rarity is easy to do. And while most of the warnings issued by Prophets end up being meaningless, or even counterproductive, such is the extreme impact of black swans that these warnings end up being worth it on balance because the one time they do work it makes up for all the times they didn’t. I think I may have said it best in a post back in 2017:

Finally, because of the nature of black swans and negative events, if you’re prepared for a black swan it only has to happen once, but if you’re not prepared then it has to NEVER happen. For example, imagine if I predicted a nuclear war. And I had moved to a remote place and built a fallout shelter and stocked it with a bunch of food. Every year I predict a nuclear war and every year people point me out as someone who makes outlandish predictions [just] to get attention, because year after year I’m wrong. Until one year, I’m not. Just like with the financial crisis, it doesn’t matter how many times I was the crazy guy from Wyoming, and everyone else was the sane defender of the status quo, because from the perspective of consequences they got all the consequences of being wrong despite years and years of being right, and I got all the benefits of being right despite years and years of being wrong.

As I pointed out, technology has served to increase the extremity of black swans, and the mention of nuclear war in that quote is a good illustration of that. Which is to say the game continues to change. At the start of the scientific revolution we were only drawing a few balls, and most of them were white, and the effects of those that weren’t were often mitigated by balls which were drawn later. (Think heating your house with coal vs. heating it with natural gas.) But as time goes on we’re drawing more and more balls, which results in more extreme black swans both positive and negative.

You might say that the game is getting more difficult. If that’s the case how should we deal with this difficulty? What’s the best strategy for playing the game? It’s been my ongoing contention that the reason we have Prophets is that they were an important part of the strategy for playing the old game. They were terrible at predicting the literal end of the world but great at helping make sure people were prepared for the numerous disasters which were all too frequent. The question is, as the game becomes more difficult, does the role of Prophet continue to be useful? My argument is, if anything, the role of Prophet has become more important, because for the first time when a Prophet says the world is going to end, they might actually be right. 

One such prophet is Toby Ord whose book Precipice I reviewed almost exactly a year ago. I think what I said at the time has enormous relevance to the current discussion:

I’m sure that other people have said this elsewhere, but Oord’s biggest contribution to eschatology is his unambiguous assertion that we have much more to worry from risks we create for ourselves than any natural risks. Which is a point I’ve been making since my very first post and which bears repeating. The future either leads towards some form of singularity, some event that removes all risks brought about by progress and technology (examples might include a benevolent AI, brain uploading, massive interstellar colonization, a post-scarcity utopia, etc.) or it leads to catastrophe, there is no a third option. And we should be a lot more worried about this than we are.

In the past it didn’t really matter how bad a war or a revolution got, or how angry people were, there was a fundamental cap on the level of damage which humans could inflict on one another. However insane the French Revolution got, it was never going to kill every French citizen, or do much damage to nearby states, and it certainly was going to have next to no effect on China. But now any group with enough rage and a sufficient disregard for humanity could cripple the power grid, engineer a disease or figure out how to launch a nuke. For the first time in history technology has provided the means necessary for any madness you can imagine.

In the same vein, one of the inspirations for this post was the appearance in Foreign Affairs of Eliezer Yudkowsky’s “Moore’s Law for Mad Science”, which states that, “Every 18 months, the minimum IQ necessary to destroy the world drops by one point.” If you give any credence at all to either Yudkowsky, Ord, or myself, it would appear impossible to argue that we have passed beyond the need for Prophets, and beyond that hard to argue that the role of Prophet has not actually increased in importance. But that’s precisely what some Wizards have argued.

One of the most notable people making this argument is Steven Pinker, and it formed the basis for his books Better Angels of our Nature and Enlightenment Now. His arguments are backed by lots of evidence, evidence of all the things I’ve already mentioned, that over the last hundred some odd years while Prophets were busy being wrong, Wizards were busy saving billions of lives. But this is why I brought up the idea that the game has changed—growing more difficult. When you combine that with the time horizon we’re talking about—a century, give or take a few decades—it’s apparent that the Wizards are claiming to have mastered a game they’ve only barely started playing. A game which is just going to continue to get more difficult. 

Yes, we’ve drawn a lot of fantastic white balls, but what we should really be worried about are the black balls, and we don’t merely need to avoid drawing one for the next few years, we need to avoid drawing a one forever, or at least until we draw the mythical pure white ball that ensures our eternal salvation. And if I were to distill out my criticism of Wizards it would be that they somehow imagine drawing that pure white ball of guaranteed salvation will happen any day now, while refusing to even consider the existence of a pure black ball. 

If you’ve been following recent news you may have heard that there has been a shift in opinion on the origins of the pandemic. More and more people have started to seriously consider the idea that it was accidentally released from the Wuhan lab, and that it was created as part of the coronavirus gain-of-function research the lab was conducting. Research which was intentionally designed to make viruses more virulent. One might hope that this causes those of a wizardly bent to at least pause and consider the existence of harmful technology, and the care we need to exercise. But I worry that instead the pandemic created something of a “no true science fallacy”, akin to the “no true scotsman fallacy” where true science never has the potential to cause harm, but only to cure it. That the pandemic was caused by a failure of science rather than possibly being exactly what we might expect from the pursuit of science over a long enough time horizon. 

As I conclude I want to make it clear, Wizards have created some true miracles, and I’m grateful every day for the billions and billions of lives they’ve saved. And I have no doubt they will continue to create miracles, but every time they draw from the urn to create those miracles they risk drawing the black ball and ending the game. So what do we do about that? Well, could we start by not conducting gain-of-function research in labs operating at biosafety level 2 (out of 4), regardless of whether that oversight was involved in the origin of COVID-19? In fact could we ban gain-of-function research period? 

I am aware that once you’ve plucked the low hanging fruit, like the stuff I’ve just mentioned, this question becomes enormously more difficult. And while I don’t have the space to go into detail on any of these possible solutions, here are some things we should be considering:

  1. Talebian antifragility: In my opinion Taleb’s chief contribution is his method for dealing with black swans. This basically amounts to increasing your exposure to positive black swans while lessening your exposure to negative black swans. Easier said than done, I know, but it’s a way of maximizing the miracles of the Wizards while avoiding the catastrophes of the Prophets.
  2. Make better use of the miracles we do have: This is another way of getting the best of both worlds. While I have mostly emphasized the disdain Wizards have for Prophets it goes both ways, and many of the things Prophets are most worried about, like global warming, get blamed on the Wizards and as such people are reluctant to use Wizardly tools like nuclear power and geo-engineering to fix them. This is a mistake.
  3. Longer time horizons: Yes, maybe Wizards like Ray Kurzweil are correct and a salvific  singularity is just around the corner, but I doubt it. In fact I’m on record as saying that it won’t happen this century, which is to say it may never happen. Which means we’ve got a long time where black balls are a possibility, but white balls aren’t. Perhaps each year there’s only a 1% chance of drawing a black ball, but over the timespan of a century a 1% chance of something happening goes from “unthinkable” to “this will almost certainly happen”.

And finally, whatever other solutions we come up with, it’s clear that one of the most important is and will always be, give heed to the Prophets!

This post ended up being kind of a clip show. If it reminded you of past posts you enjoyed, and that lengthened your time horizon, consider donating. I’d like to keep doing this for a long time.