Category: <span>Artificial Intelligence</span>

Eschatologist #7: Might Technology = Extinction?

If you prefer to listen rather than read, this blog is available as a podcast here. Or if you want to listen to just this post:

Or download the MP3


One of the great truths of the world is that the future is unpredictable. This isn’t a great truth because it’s true in every instance. It’s a great truth because it’s true about great things. We can’t predict the innovations that will end up blessing (or in any event changing) the lives of millions, but even more importantly we can’t predict the catastrophes that will end up destroying the lives of millions. We can’t predict wars or famines or plagues—as was clearly demonstrated with the recent pandemic. And yet on some level despite the impossibilities of foretelling the future we must still make an attempt.

It would be one thing if unpredicted catastrophes were always survivable. If they were tragic and terrible, but in the end civilization, and more importantly humanity, was guaranteed to continue. Obviously avoiding all tragedy and all terror would be ideal, but that would be asking too much of the world. The fact is even insisting on survivability is too much to ask of the world, because the world doesn’t care. 

Recognizing both the extreme dangers facing humanity, as well as the world’s insouciance, some have decided to make a study of these dangers, a study of extinction risks, or x-risks for short. But if these terminal catastrophes are unpredictable what does this study entail? For many it involves the calculation of extreme probabilities—is the chance of extinction via nuclear war 1 in 1,000 over the next 100 years or is it 1 in 500? Others choose to look for hints of danger, trends that appear to be plunging or rising in a dangerous direction or new technology which has clear benefits, but perhaps also, hidden risks. 

In my own efforts to understand these risks, I tend to be one of those who looks for hints, and for me the biggest hint of all is Fermi’s Paradox, the subject of my last newsletter. One of the hints provided by the paradox is that technological progress may inevitably carry with it the risk of extinction by that same technology

Why else is the galaxy not teeming with aliens

This is not to declare with certainty that technology inevitably destroys any intelligent species unlucky enough to develop it. But neither can we be certain that it won’t. Indeed we must consider such a possibility to be one of the stronger explanations for the paradox. The recent debate over the lab leak hypothesis should strengthen our assessment of this possibility. 

If we view any and all technology as a potential source of danger then we would appear to be trapped, unless we all agree to live like the Amish. Still, one would think there must be some way of identifying dangerous technology before it has a chance to cause widespread harm, and certainly before it can cause the extinction of all humanity! 

As I mentioned already there are people studying this problem and some have attempted to quantify this danger. For example here’s a partial list from The Precipice: Existential Risk and the Future of Humanity by Toby Ord. The odds represent the chance of that item causing humanity’s extinction in the next 100 years.

  • Nuclear War                       ~1 in 1000
  • Climate Change                 ~1 in 1000
  • Engineered Pandemics     ~1 in 30
  • Out of control AI                ~1 in 10

You may be surprised to see nuclear war so low and AI so high, which perhaps is an illustration of the relative uncertainty of such assessments. As I said, the future is unpredictable. But such a list does provide some hope, maybe if we can just focus on a few items like these we’ll be okay? Perhaps, but I think most people (though not Ord) overlook a couple of things. First, people have a tendency to focus on these dangers in isolation, but in reality we’re dealing with them all at the same time, and probably dozens of others besides. Second it probably won’t be the obvious dangers that get us—how many people had heard of “gain of function research” before a couple of months ago?

What should we make of the hint given us by Fermi’s Paradox? How should we evaluate and prepare ourselves against the potential risks of technology? What technologies will end up being dangerous? And what technologies will have the power to save us? Obviously these are hard questions, but I believe there are steps we can take to lessen the fragility of humanity. Steps which we’ll start discussing next month…


If the future is unpredictable, how do I know that I’ll actually need your donation. I don’t, but money is one of those things that reduce fragility, which is to say it’s likely to be useful whatever the future holds. If you’d like to help me, or indeed all of humanity, prepare for the future, consider donating.


State of the Blog, Predictions, and Other Sundry Items

If you prefer to listen rather than read, this blog is available as a podcast here. Or if you want to listen to just this post:

Or download the MP3


Normally I start the year with a post reviewing my long term predictions. As part of that I make some new, shorter term predictions. But it’s also become the custom to begin each month with reviews of the books I finished over the previous month. Given how long my book review posts have become I certainly don’t want to combine the two, and also I have some changes I want to announce/float for 2021, so I’m going to combine all of these different threads into a single post: an end of the year review of where things are headed, where things have been, how my predictions have held up and what new predictions I’d like to go on the record with. Since I assume more people are going to be interested in my short term predictions, and especially where I have been wrong, let’s start there, then move to a review of how my long-term predictions are holding up and end with the navel gazing.

I- Last Year’s Predictions

At the beginning of 2020 I predicted:

More populism, less globalism. Specifically that protests will get worse in 2020.

I feel pretty good about this prediction. The pandemic has been hard on globalism, national borders are making a resurgence, and tensions between nations appear to be rising (the Solarwinds hack certainly didn’t help). Beyond that the pandemic and the associated lock downs have opened huge gulfs between global technocrats and the citizenry. Gulfs that are unlikely to be mended anytime soon.

Speaking of the above, my predictions about protests getting worse have certainly come to pass. And while I didn’t identify that pandemic backlash and BLM would be the greatest sources of protests, there’s clearly a lot of populism in the air. This populism appears to be reaching a crescendo when it comes to Trump’s continuing fights over accepting the election results. Which I’ll expand on in a minute.

No significant reduction in global CO2 emissions (a drop of greater than 5%)

Here I was wrong. Because of the enormous economic effects of the pandemic, emissions dropped a whopping 8%. I’m not going to claim that I was really correct because, “Who could have foreseen the pandemic?” This is, in fact, precisely the problem I have with many of the people who make predictions, they often argue that black swans shouldn’t count. This is another thing I’ll get to in a minute.

Social media will continue to have an unpredictable effect on politics, but the effect will be negative.

This is another one I think I nailed. If anything I was too cautious. It seems clear that despite the efforts of the companies themselves to block and tag (what they considered to be) misinformation, social media still provided a major vector for the spreading narrative of a stolen election which is now present in one form or another among the vast majority of Trump supporters (88% according to some sources). One might even go so far as to say that their efforts at tagging and blocking made it worse, that social media can’t be used for good ends. 

(For those who think the election was actually stolen, I would refer you to my previous post on that subject. For the tl;dr crowd, I argued that if it was stolen it was done in so comprehensive a manner that it amounts to winning regardless.)

That the US economy will soften enough to cause Trump to lose.

Here I was basically right, though I’m not inclined to give myself too much credit. First whatever the economy did was almost entirely a consequence of the pandemic. And I was dead wrong about the stock market, which continues to amaze me. But most people agree that without the pandemic Trump probably would have won, which kind of, if you squint, amounts to the same thing I was saying.

That the newest wave of debt accumulation will cause enormous problems by the end of the decade.

Too early to say, I was speaking of 2030 here not 2020. But certainly we accumulated debt at a much faster rate this year than I think anyone predicted going in. So, as I said in a previous post, we better hope the modern monetary theorists are correct. Because if government debt is fragilizing at all we’re acquiring fragility at an enormous clip.

Authoritarianism will continue to increase and liberal democracy will continue its retreat.

To whatever extent you think liberal democracy overlaps with classical liberalism, I think most people were amazed at the attacks which were leveled during 2020, particularly from things like critical race theory. These sort of attacks mostly came from the left, but the right isn’t looking very good either. Certainly the most recent election and their reaction to it has ended up giving democratic legitimacy a severe beating (though the narrative of the beating is different depending on which side you talk to.)

Beyond this, all indications are that China has gotten more authoritarian this year, both with respect to Hong Kong and the Uighurs. But perhaps the big open question is what happens to the additional authoritarianism brought on by the pandemic? Does it fall at the same rate as the case counts? Or does some of it linger? I suspect it basically goes away, but having discovered what tools are available, those tools become easier to use in the future.

The Middle East will get worse.

I would say I mostly got this one wrong, and Trump deserves a lot of credit for the peace deals that were brokered under his watch. That said, the situation with Iran is definitely looking worse, so not everything has been sunshine and roses. Also it’s not just the nuclear deal and the swiftly increasing uranium stockpiles. The peace deals, while almost certainly a good idea, have had the effect of making Iran feel increasingly encircled and isolated. And bad things could happen because of this.

Biden will squeak into the Democratic nomination.

I was clearly right about Biden getting the Democratic nomination, and I think I was right about the “squeak” part as well. Recall that not only was my prediction made before any of the primaries, but also that Sanders won both Iowa and New Hampshire. And since 1976 only Bill Clinton has gone on to win the nomination after losing both of those primaries, and even then 538 argues it only happened because of exceptional circumstances. So yeah, despite the eventual delegate total I would still argue that Biden squeaked into the nomination.

The Democrats will win in 2020.

By this I meant that whoever ended up with the Democratic nomination for president would go on to win the election, not that the Democrats as a whole would triumph in some large scale way. I wasn’t arrogant enough to think I could predict how congress would end up looking.

So those were my predictions at the beginning of 2020. I’m not asking to be graded on them, and certainly I don’t think I deserve any particular recognition, obviously I got some things right and some things wrong, and the thing I’ve actually been the most wrong about didn’t even make it into my list of predictions: how wrong I was about Trump and his supporters.

While I continue to maintain that right-wing violence is overstated, or perhaps more accurately that all violence which might remotely be considered right-wing get’s labeled as such while lots of violence that should get labeled as left wing, under the same standard, is considered to be non-ideological (see this post for a deeper dive into this.) I am nevertheless very surprised by all of the shenanigans which have been attempted in order to keep Trump in power and beyond that the enormous number of people who think he should be kept in power, even if it requires something like using the Insurrection Act to call up the military. 

Perhaps this is the first you’ve heard of this idea, which is an example of how insular the various worlds have become. (Though in some respects I think this still comes back to my underestimation of how bad social media could be.) I know more than a few people who are convinced that everything Trump has done since the election was all part of a vast sting operation, designed to lure the deep state into so overplaying their hand and making their fraud so obvious that “they” could be rounded up in one giant operation. Well whether there was fraud or not I don’t think it’s ended up being blindingly obvious. And if that’s not what’s going on then we either had a legitimate election or the deep state cheated in such an overwhelming fashion that things can only be sorted out at the point of a gun, which seems like one of the most catastrophically bad ideas imaginable, and I never would have predicted the way things have gone since November 3rd.

II- An Interlude on Predictions in General

There are many people who would look at this review of my short term predictions with the accompanying explanations and declare that it’s the same kind of fuzzy predictions with fuzzy accountability that everyone engages in. That if I want to be taken seriously as a predictor that I should use the Superforecasting method, where you make a prediction that’s specific enough to be graded, and then attach a confidence level to it. That is “many people” might say that if they haven’t been following me for very long. Those that have been around for awhile know that I have huge issues with this methodology, which I have outlined ad nauseam, and if you want to get my full argument I would refer you to my past posts on the subject. For those who aren’t familiar with my arguments and just want the abbreviated version, this year provides the perfect object lesson for what I’ve been talking about all this time, and it can be summed up in two words: black swans. Rare events end up being hugely consequential to the way things actually play out. Superforecasting not only has no method for dealing with such events, I think it actively shifts focus away from them, and this year was a fantastic example of that.

How many Superforecasters predicted the pandemic? How many predicted that Trump would seriously consider using the Insurrection Act to maintain power? To be clear I understand that they did correctly predict a lot of things. They almost certainly did better than average at calling the presidential race. And within the confines of their system they’re excellent, i.e. they’re really good at having 90% of the predictions they have 90% confidence in turn out to be true. But take all the predictions that they made about 2020, or even about the whole decade of the 2020’s and imagine that they’re all correct. Which would give you a clearer picture of the world of 2020? All those predictions or just knowing that there was a global pandemic? Now I understand that no one knew there was going to be a global pandemic, but which nations did better? Those who were prepared for a pandemic, with a culture of mask wearing? Or those who had the best forecasters?

So yes, pandemics are rare, but they’re hugely consequential when they do happen, and if Superforecasting does anything to reduce our preparedness for those sorts of things, by shifting focus on to the things they are good at predicting, then on net superforecasting is a bad thing. And I have every reason to suspect it does. 

All of the things I said about the pandemic will be equally true if Trump decides to actually invoke the Insurrection Act. Which is another thing that wasn’t even on the superforecasting radar. (A Google search for “superforecasting ‘insurrection act’” comes back with the message “It looks like there aren’t many great matches for your search”). But, and this is the interesting part, it is on the radar of all those so-called “crazy preppers” out there. It may not be on their radar in the way you hope, but the idea that things might disintegrate, and guns might be useful has been on their radar for a long time. Based on all of this, the vast majority of my predictive energy is spent on identifying potential black swans. With short term forecasting as more of an engaging exercise than any real attempt to do something useful. We’ll get to those blacks swans in a minute, but first:

III- Predictions for 2021

I think there’s a huge amount of uncertainty going into this year, and things which got started in 2020 could go a lot of different ways. And I think this time around I’m going to go for quantity of predictions, not quality:

  1. Biden will not die in 2021
  2. The police will shoot another black man (or possibly a black woman) and new protests will ensue.
  3. The summer tourist season will proceed in a more or less normal fashion but with some additional precautions (I have a Rhine River Cruise scheduled for June, so this one is particularly important for me.)
  4. Bitcoin will end the year higher than it is right now.
  5. Trump will not invoke the insurrection act.
  6. But if he does the military will refuse to comply, probably after someone files an emergency lawsuit, which then gets decided by the Supreme Court.
  7. There might possibly be a few soldiers who do something stupid in spite of this, but the military command structure will not go along with Trump and soldiers will side with their commanders rather than with Trump.
  8. Trump’s influence over the Republican party will begin to fade. (Not as fast as some people would hope, but fast enough that he won’t be the Republican nominee in 2024.)
  9. Large tech companies will increasingly be seen as villainous, which is to say the antitrust lawsuits will end up being a pretty big deal. I think they’ll take longer than one year to resolve, but at the end I expect that there will be a significant restructuring to at least one of the tech companies. (I’m leaning towards Facebook.)
  10. The anti-vaxxer movement will grow in prominence, with some of the same things we’ve come to expect out of other movements: conspiracy theories (moreso), broad support, protests, etc.

And now for some things I think are unlikely but which might happen and are worth keeping an eye on:

  1. The Republican party disintegrates. Most likely because Trump leaves and starts his own party.
  2. COVID mutates in such a way that the vaccines are no longer as effective, leading to a new spike in winter of 2021-2022.
  3. Biden doesn’t die, but he exhibits signs of dementia significant enough that he’s removed under Amendment 25.
  4. I’d be very surprised if we saw actual civil war (assuming I’m right about #7 above) but I would not be especially surprised to see violence on the level we saw in the late 60s and early 70s.
  5. Significant unrest in mainland China similar to Tiananmen Square, and at least as big as the Hong Kong protests. 

These are just the things that seem possible as a continuation of trends which are already ongoing, but 2021 could also bring any of the low probability catastrophes we’ve been warned about for decades, in the same fashion that 2020 brought us the global pandemic, 2021 could bring a terrorist nuke, a Chinese invasion of Taiwan, a financial crisis, etc. 

IV- Status of Long-Term Predictions

When I initially made these predictions, at the beginning of 2017, I grouped things into five categories:

Artificial Intelligence:

  1. General artificial intelligence, duplicating the abilities of an average human (or better), will never be developed.
  2. A complete functional reconstruction of the brain will turn out to be impossible.
  3. Artificial consciousness will never be created.

As you can see, I’m pretty pessimistic when it comes to general artificial intelligence (GAI). But before we get into the status of my predictions, I need to offer my usual caveat that just because I think GAI is improbable doesn’t mean that I also think studying AI Risk is a waste of time. I am generally convinced by arguments that a GAI with misaligned incentives could be very dangerous, as such, even though I think one is unlikely to be created, as I said, I’m all about trying to avoid black swans. And that’s what my long term predictions revolve around. Some are black swans I think are inevitable and others are black swans that I personally am not worried about. But I could very easily be wrong. 

In any case this last year there was quite a bit of excitement around GPT-3, and I will freely admit that it’s surprisingly impressive. But no one thinks that it’s a GAI, and as far as I can tell most people don’t think that it’s a direct path to GAI either. That it is at best one part of the puzzle, but there are still lots of pieces remaining. I’m going to be even more pessimistic than that, and argue that this approach is nearly at its limits and we won’t get anything significantly better than GPT-3. That for someone skilled enough it will still be possible to distinguish between text generated by GPT-4 or 10 and text generated by a skilled human. But the fact that it will require skill on both ends is still a very big deal.

Transhumanism:

  1. Immortality will never be achieved.
  2. We will never be able to upload our consciousness into a computer.
  3. No one will ever successfully be returned from the dead using cryonics.

All of my predictions here relate to life extension in one form or another. I think similar to how things have worked with AI in the past where there was significant excitement and then a plateau, leading to a couple of AI winters. That we are entering a life extension winter. That a lot of the early excitement about improved medicine and gene editing has not panned out as quickly as people thought, (or there are major ethical issues) and for the last few years, even before the pandemic, life expectancy has actually been decreasing. As of 2019 it had been decreasing for three years, and I can’t imagine that this trend reversed in 2020, with the pandemic raging. 

Of course cryonics and brain uploading aim to route around such issues, but if there have been any advancements on that front this year I missed them.

Outer space: 

  1. We will never establish a viable human colony outside the solar system.
  2. We will never have an extraterrestrial colony (Mars or Europa or the Moon) of greater than 35,000 people.
  3. We will never make contact with an intelligent extraterrestrial species.

There has been a lot of excitement here. And Musk and some of the others are doing some really interesting things, but as I expected the timeline for all of his plans has been steadily slipping. In 2017 he said he’d have “Two cargo landers on Mars 2022, Four landers (two crewed) Mars 2024”. Now he’s saying, a tourist flight around the Moon in 2023, with unmanned craft on Mars in 2024. And even that seems ridiculously optimistic. The problem as I (and others) keep pointing out, is that doing anything in outer space is fantastically difficult. 

Fermi’s paradox (#3) is its own huge can of worms, and this year did see the release of the Pentagon UFO videos, but for a large variety of reasons I am confident in asserting that those videos do not represent the answer to the paradox. And I’ll explain why at another time.

War: (I hope I’m wrong about all of these)

  1. Two or more nukes will be exploded in anger within 30 days of one another.
  2. There will be a war with more deaths than World War II (in absolute terms, not as a percentage of population.)
  3. The number of nations with nuclear weapons will never be less than it is right now.

This section doesn’t need much additional elaboration because the historical precedents are so obvious. Mostly I’m merely predicting that war is not a thing of the past. That the Long Peace will eventually end. 

Miscellaneous

1- There will be a natural disaster somewhere in the world that kills at least a million people

2- The US government’s debt will eventually be the source of a gigantic global meltdown.

3- Five or more of the current OECD countries will cease to exist in their current form.

Mostly self explanatory, and as I mentioned this year we have really doubled down on the idea that deficits don’t matter so if #2 doesn’t happen, it won’t be because any restraint was exercised. And as far as #3 my standard for “current form” is pretty broad. So successful independence movements, dramatic changes in the type of government—say from democracy to a dictatorship, and civil wars, would all count. 

V- The State of the Blog

I’ve decided to make a few changes in 2021. The biggest being that I’m joining all the cool kids and starting a newsletter, though this will end up being less consequential than it sounds. My vague goal for the current year was to put out four posts a month, one of which was a book review round up. If you look back over the year you’ll see that there were a few months (including this one) where I only got three posts out. In large part that’s because I’ve also been working on a book, but also the posts seem to gradually be getting longer as well. All of this is somewhat according to plan, but I worry that if a 4000 word essay is the smallest possible chunk my writing comes in, that there are going to be a lot of people who might be interested in what I have to say but who will never be able to get over that hump, and self-promotion has never been my strong suit at the best of times.

The newsletter is designed to solve both of these problems. Rather than being thousands of words I’m going to limit it to 500. Rather than forcing you to come to my blog or subscribe to my RSS feed, it’s going to be delivered straight into your mailbox. Rather than being a long and nuanced examination of an issue it’s going to be a punchy bit about some potential catastrophe. Delivered at the end of every month. (Tagline: “It’s the end of the month, so it’s once again time to talk about the end of the world!”) I will still publish it here, so if you prefer reading my blog as you always have you won’t have to follow any additional steps to get the newsletter content, though, a month from now, I still hope you’ll subscribe, since it will hopefully be something that’s easier to share. And the whole point of the exercise is to hook some additional people with the newsletter and use that as a gateway to the harder stuff.

To summarize, I’m replacing my vague goal from last year of four posts a month with the concrete commitment for 2021 of:

  • A book review round up at the beginning of each month
  • At least two long essays every month but possibly three.
  • An end of the month short piece which will go out as part of a newsletter
  • A book

As far as the book. I’m shooting to have it done sometime this summer, though there’s good reason to suspect that it might slip into the fall. I may get into the details of what it’s about later, but for now I can reveal that it does contain the full explanation for why the Pentagon UFO videos are not the solution to Fermi’s Paradox, even if they were to depict actual UFOs! 

With that cliffhanger I’ll sign off. I hope everyone had a Merry Christmas, and that your New Year’s will end up being great as well, and I’ll see you in 2021.


As someone who specializes in talking about catastrophes, I got quite a bit of content out of 2020, but like everyone I’ll be glad when it’s over. Still if you appreciated that content, if it helped distract you from the craziness that was 2020, even a little bit, consider donating.


“The Good Place”, Brain-uploading, and Eschatology

If you prefer to listen rather than read, this blog is available as a podcast here. Or if you want to listen to just this post:

Or download the MP3


***Warning: Massive spoilers for The Good Place ahead. If you don’t want to be spoiled don’t read this post.***

The Good Place recently ended after four seasons. The show was praised for its various twists, and it’s “exploration and creative use of ethics and philosophy”. But of course it was also a show about eschatology, in fact it may be argued that this was it’s primary subject matter. Given this focus it’s reasonable and even important to examine the nature of the eschatology it espoused. There are of course a wide variety of imagined eschatologies out there, and numerous definitions of the word itself beyond that. So what was The Good Place’s contribution to this topic?

Before answering that question we first need to set some parameters for the discussion. To begin with, I want to discuss this subject in a very practical fashion. Obviously, despite my argument that the show’s primary theme was eschatology, that wasn’t the show’s purpose. It’s purpose was to entertain, and as such it was far more interested in taking a humorous look at a potential afterlife than it was in taking a serious and consistent approach to things. On the other hand, I would like to set aside the humorous bits and strip away things which are present only for their entertainment value, but in order to accomplish this I need to do two things:

First, while a large part of this post will be dedicated to pointing out the various flaws I noticed in the show’s handling of how an afterlife might work. I need to make it clear that this should not be construed as an attack on the show, or any indication that I didn’t enjoy it or that you won’t enjoy it. In particular I don’t want people to be distracted by defending the show, as I said the show’s purpose was to be entertaining, not philosophically rigorous. And as a sitcom it was one of the best, but it did propose an eschatology, and it’s worth examining whether that eschatology hangs together.

Second, in order for those flaws to have any resonance, we have to be willing to imagine that we’re critiquing something which might actually exist, that there might, in fact, be an afterlife, and additionally that there might be god-like beings in charge of that afterlife, or at least something supernatural about how it’s put together. Otherwise any discussion of how it should work, and how that might be different than how it worked on the show, will be, at best superficial, and at worst, entirely pointless. For anyone who’s religious, imagining an afterlife and the supernatural qualities which would have to attend such a place, is easy. But I don’t want to rely too much on religion (though I can’t avoid it entirely) because it will inevitably be off-putting for those who are not religious, or who belong to a different denomination than those I ended up using in my examples.

Fortunately, this is an ideal place to bring in my extensive work imagining how certain religious ideas (including the afterlife) resemble ideas for dealing with AI risk. Meaning, that for those who aren’t religious, rather than imagining what happens or should happen to the souls of the departed, we can imagine the eschatology associated with AIs or, their close cousins, individuals who have had their brain uploaded into a virtual environment where the natural rules don’t apply (an environment which is supernatural by definition.) Obviously, I’m not going to want to type out that entire explanation every time I refer to these individuals, so instead I’ll just use Robin Hanson’s shorthand and call them Ems. Presumably even those who are not religious can imagine that someday we might develop the ability to construct (or reconstruct) a person in a virtual environment, and thereby realize a technological eschatology. And considering how that environment should work gets us to an afterlife or at least a “heaven” very similar to one imagined by many religions and by The Good Place itself.

Having hopefully given everyone a little more skin in the game on this topic, let’s proceed to our examination of what The Good Place got right, but probably more importantly what it got wrong about eschatology and potential afterlifes. 

Let’s start with one of the very first things I noticed, and one of the elements the show mangled the most. To repeat, I’m sure they made things this way for entirely understandable reasons, it was both comedic and necessary for the character arcs of nearly all the people on the show.  But, their representation of the “Good Place Committee” (GPC) represented a fundamental and almost insulting misunderstanding of the nature of good. I am assuming that most of those reading this had a chance to see the show, but if not, in the show, after people die, they can go either to the “Good Place” or the “Bad Place” and there isn’t much to distinguish these two places from common conceptions of heaven and hell, so I’ll be using the terms interchangeably.

Heaven is run by a committee, and apparently in this version of the world being good (or at least qualified to run the Good Place) comprises a combination of fawning politeness with absolute and total ineffectiveness. This seems clearly to be one of those things that was done the way it was for both the humor value and as a way to give the main characters something to do because certainly this bears no resemblance to the theology of any of the world’s religions, and even if we imagine that the “souls” in question are Ems and that humans are running the show rather than an omniscient creator it’s still impossible to imagine that the best governing structure they could come up with is the committee from the show (or any committee for that matter.)

Of course this leads to the question of what sort of people we should expect to be running or even just inhabiting heaven. And here I will allow that it’s a difficult question. One of the chief lessons to come out of recent philosophical work on AI risk has been the realization that coming up with a fool proof standard for morality is both enormously important and enormously difficult. That defining an objective, and it should be added, secular standard for what’s good and what’s not is a challenging task. But even with those difficulties in mind I think we should at least expect that any morality worthy of the name has to have some backbone to it, that this is in fact almost the definition of morality. And while, as I said, there were probably several good reasons for portraying it the way they did, I also wonder if they could have portrayed it in any other way, and if equating being obsequious for being good was the only way to not get overly political. 

(It should be noted it’s not just the GPC, in the show the paragon of “virtue” on the earth, Doug Forcett is also a gigantic pushover.)

I feel like this was not always the case, that there was a time when you could have pointed to a society-wide morality, and that being able to draw on a more robust morality would have allowed them to construct a far more convincing heaven (can you imagine what the Good Place would have looked like in the 1920’s?) but that such universality is no longer present. All that said, perhaps I’m reading too much into things, but at its most essential when anyone imagines heaven and hell you always imagine a war existing between the two. In the show it’s clear that the Bad Place is waging such a war against the Good Place, which the Good Place has been losing for centuries, apparently without even noticing it, or having the ability to fight back if they had. And it’s hard to imagine that any functional organization, much less one designed to be the ultimate ideal, could ever be that inept. But it makes you wonder, is there any chance that this is a reflection of our own failings in this area? Because it gets worse.

In the final few episodes we find out that not only have the effective “rulers of heaven” been too polite and willing to compromise and that they are losing a war with Hell they don’t even seem to be aware of, but on top of all this they’re actually terrible at running heaven. Somehow they have managed to create another version of hell, which is so bad that when it’s announced to those souls who’ve made it to heaven that they will be allowed to effectively commit suicide in order to leave, they cheer, and it’s implied that it’s the first cheer that’s been heard there in hundreds if not thousands of years. 

Here is where we turn to the things The Good Place did well. To begin with they tackle head-on the question of whether immortality would be a blessing or a curse. This idea that immortality might get old (pun intended) is one of the more interesting philosophical topics the show tackles, and a serious subject for debate among actual philosophers. One of the reasons to favor the idea that it’s a curse (which ends up being the show’s position) is illustrated by the pseudo-hell of boredom the characters find when they arrive in the Good Place. A boredom so soul-crushing that even with access to anything they could possibly imagine suicide seems preferable. Certainly claiming that regardless of how good it was, that one would eventually tire of life is not an unreasonable position to take, but neither does it feel particularly creative either. Regardless, one assumes that the GPC still could have done a better job of dealing with that boredom than they did, but if we keep our same basic emotions and appetites, even after having our brain uploaded into a virtual heaven, then boredom would still probably be a real concern. It should be mentioned that Hanson cleverly solves this problem for Ems by running them at a lower speed. As I said the “immortality is a curse” option is reasonable, but surely we can imagine ways to change that.

To flip it around and look at what people might want rather than what they’re trying to avoid, any system like this would, in theory, be trying to maximize human flourishing. One of my readers recently pointed me to an article where the Royal Society suggested that future technological systems should have “promote human flourishing” as their primary imperative. And The Good Place does a great job of illustrating how this is much easier said than done. 

For all of the characters in the show it quickly becomes obvious that even in heaven in order to be happy, that is to flourish, they need to have a work to do, something to occupy their attention. It’s not clear if this is an innovation introduced by the main characters or if all the previous inhabitants of the Good Place have exhausted this avenue before they arrive, but you get the impression it’s the former. And it illustrates another failure mode of heaven and immortality, the hedonic treadmill. If you give people everything they’ve ever wanted, the increased happiness is temporary. (The classic example is lottery winners.) That along with rewards there has to be continual challenges. And it occurs to me that beyond being interesting dilemmas, boredom and a hedonic set point are problems we’re already facing without having to imagine a heaven, virtual or otherwise.

It’s something of a cliche to talk about how in a developed country even relatively poor people live better than the kings of old. And while the situation is more complicated than that, it’s remarkable how much the modern world already resembles the Good Place of the show. One of the characters, Jason, apparently wants to play Madden forever. Well it’s my understanding that you can already do that. It probably helps if your parents let you live in their basement, or if you’ve got some other minimal level of support (I don’t think it takes that much. UBI or disability might be sufficient.) But that is something that’s already within reach and is probably just going to get easier. But is it flourishing? Are we sure we know what flourishing is? One of the whole points of the show is that no one, even in the afterlife, actually does. 

Hovering in the background of the show, but never mentioned, is the question of a designer. And while this part ends up being the most metaphysical, it’s also the part I find the most interesting. In most mythologies, or theologies, or even most systems in general, there’s a very prominent creation story. In Greek mythology there’s Gaia and Uranus. For the Abrahamic religions there’s Adam and Eve. For Facebook there’s Mark Zuckerberg in his dorm room. But The Good Place pays almost no attention to any sort of “origin story”.

The closest we come is to find out that the Judge can destroy creation, and then reset it, but the “demons” running the Bad Place are not affected by this destruction so they exist outside of creation? But beyond this, the list of things we don’t know is staggering. Who created the point system? Why is there a point system? Who’s the judge? Where do the GPC and the Demons come from? And those are just questions directly relating to the show. There are still all the normal questions of why suffering and evil exist. What is the point of having a hell? And what is the source of morality?

In the end it definitely feels that there had to be a designer, whatever else you may say about things they definitely don’t feel organic. It seems clear from the show that someone came along and set all of this up, the point system, the existence of a Bad Place and a Good Place. The angels and the demons had to come from somewhere as well. But apparently whoever this person was, despite being effectively omnipotent, they don’t appear to have been omniscient, or even particularly wise. I’ve already talked about the various issues with the committee that runs the Good Place, but more than that the central premise of the show is revealing how poorly designed the afterlife actually is.

This is yet another similarity with our own condition. Being omnipotent without being omniscient or even very wise is not that far off from describing our own situation. Particularly if we’re ever able to upload our consciousness into a rules free virtual environment. How concerned should we be by this mismatch? If there’s one actual lesson to be taken from the show, it might be that we should be very concerned. And it actually works from both directions, in addition to showing a heaven where no one is actually happy, the show begins with the premise that, despite having infinite power to inflict torture on humans, they’re apparently looking for better ways of making them suffer as well. And part of the genius of the show is that both ring true, both happiness and misery end up being more complicated than expected, and being omnipotent is not the same as being omnicompetent. 

Obviously drawing a direct connection between a TV show and hypothetical future technology is of very limited utility, but I would argue that the utility is not zero. We’ve had the ability to satisfy our appetites beyond anything our ancestors imagined for quite some time (see my episodes on supernormal stimuli) and thus far the best we can say is that results have been mixed. And while we’re definitely going to get better at satisfying our appetites, it’s not clear that we’re going to get any better at managing the outcome of that.

We’re quick to imagine that if we ever get to the point where we can upload our brains into a virtual world of our own devising, crafted in such a way that our wildest dreams become reality, that all our problems will be solved. And if it’s not exactly this scenario there are still a lot of people with the same basic eschatology as the show: There’s a Good Place out there and we need to get to it. But just like the characters, there’s some chance that when we get there, it will turn out that it’s not as straightforward as we thought. And to the extent that we’re already there this is becoming increasingly obvious.


As those of you who have watched the show know. There’s also a Medium Place, inhabited by exactly one individual. And despite being only one person out of billions and despite being deeply flawed, this person exercised disproportionate influence on the rest of “creation”. I’m guessing there’s some lesson in there about the power we all have, but mostly I’m just making the connection that there was one person in the Medium Place and there’s one person writing this blog, so donate, I guess? 


Books I Finished in September

If you prefer to listen rather than read, this blog is available as a podcast here. Or if you want to listen to just this post:

Or download the MP3


It’s once again time for the monthly round up of the books I read:

Savage Worlds: Adventure Edition

By: Shane Lacy Hensley

208 pages

Thoughts

This is the latest edition of a well known universal Role-Playing game system called Savage Worlds. I’m a big fan of the system, but for my money there weren’t enough changes to justify putting out a new edition.

Who should read this book?

If you love, love, love Savage Worlds and run it all the time, it’s probably worth picking up this book. If you’re like me and you collect RPG systems, and you already have a Savage Worlds rulebook in your collection this is not different enough from past editions to be worth picking up.

Adrift: Seventy-Six Days Lost at Sea

By: Steven Callahan

234 pages

Thoughts

There’s a little old lady who used to be in my ward (that’s the Mormon version of a congregation) and in addition to being a voracious reader she’s exceptionally cunning. The first attribute led her to have an Audible subscription, the last bit led her to offer to share it with me when she realized she could have up to five connected devices. I was going through some financial difficulties at the time (a lawsuit) and so I took her up on the offer. I have since gotten my own Audible account, but she still let’s me know when she’s listened to something she particularly likes. She has a fondness for survival stories, and so I end up listening to quite a few of them. (Two this month.) This is good because I am also a fan of them, but they’re not the kind of thing I would seek out normally.

As you can probably tell from the title Adrift is one of these survival stories. Most survival stories get into the mechanics and the logistics of survival, and Adrift is no exception, in fact if anything it may partake of more of this sort of thing than most books in the genre. If that’s your thing you’ll probably really enjoy this book. For me, listening to it as an audiobook I had a hard time picturing everything he was describing. Nevertheless, Callahan was great at surviving, and is mentioned as one of the best examples of a survivor in another book I read in September. 

Novacene: The Coming Age of Hyperintelligence

By: James Lovelock

160 pages

Thoughts

This was kind of a weird book. (There were a couple in that category this month.) Lovelock is best known for his Gaia theory, which basically holds that organic and inorganic matter work together to create the perfect living environment. (Examples include global temperature, seawater salinity, and atmospheric oxygen.) I haven’t ever read that book but I remember being skeptical when I heard about the premise, what about Snowball Earth or the Great Oxygenation Event? I assume that Lovelock would say that despite how hard they were on the ecosystem which existed at the time that both events were necessary stepping stones to the world we have now. He appears to be making a similar argument here, that everything which has come so far has all been in service of the next stage of evolution, what he’s calling the Novacene. From the book jacket:

In the Novacene, new beings will emerge from existing artificial intelligence systems. They will think 10,000 times faster than we do and they will regard us as we now regard plants. But this will not be the cruel, violent machine takeover of the planet imagined by science fiction. These hyperintelligent beings will be as dependent on the health of the planet as we are. They will need the planetary cooling system of Gaia to defend them from the increasing heat of the sun as much as we do. And Gaia depends on organic life. We will be partners in this project.

Wait, what? Maybe I’m overlooking something huge, but there are lots of cooler places in the universe, to say nothing of in the solar system, than the surface of the Earth. (Check out the aestivation hypothesis as an explanation for Fermi’s Paradox.) And even if, for some reason, the coming hyperintelligence were restricted to Earth (say because of the tyranny of the rocket equation) then, however “cool” the Earth is right now, there are probably lots of ways to make it much cooler that require very little human involvement. 

Who should read this book?

As I said, maybe I’m missing something gigantic, but if not this is a seriously flawed book, which no one should bother reading.

Bronze Age Mindset

By: Bronze Age Pervert

198 pages

Thoughts

Around this time last year a friend of mine visited from out of town, and we had a conversation about incels (mostly those who were literally involuntarily celibate, not those who had adopted the label). At the time I thought the conversation was interesting enough to do a post about it.

As part of the conversation we both agreed that there are lots of young men who lack meaning and feel abandoned by society, women or the world in general. What we disagreed on was what to tell these young men, though we both felt it was a very important question. Well Bronze Age Mindset is one answer to that question, and it’s a doozy. (This is the other weird book I read this month.) 

To begin with, at one point this self-published book, which seems to be written in a vague stream of consciousness fashion with little regard for verb conjugation or indefinite articles cracked the top 150 books on Amazon. This is out of all the books on Amazon, not merely in some specific category. Meaning whatever else you want to say about the book it’s an answer to the question I posed that has resonated for a lot of people. 

What about the book itself? Well if you really want a full review I would recommend the one Michael Anton did in the Claremont Review of Books: Are the Kids Al(t)right? For my own part I could sense how the book might be appealing, but it’s hard to point to anything specific, there’s little direct advice in the book. Rather, I think most of the appeal comes from the transgressiveness which suffuses the book. It probably goes without saying that the book is homophobic, misogynist, racist and anti-democratic, but he doesn’t spend much time or speak very strongly about any of these items. They just appear in support of the larger tapestry of transgression he weaves. I think Anton does a great job of distilling all of that into a short description of the book’s appeal:

This book speaks directly to young men dissatisfied with a hectoring vindictive equality that punishes excellence.

These exhortations towards excellence take the form of urging readers to attempt fantastic feats of military prowess to set themselves apart from the vast masses of people, the “bugmen” as he refers to them. Going so far as to say that life appears at its peak in military state, which he feels is inevitable.  Which would be alarming if true (I don’t think that’s the way things are going.)

Having said all that I’m still surprised that it has sold so well. I was particularly alarmed by what Anton describes as:

…the book’s most risible passages, [where] BAP wonders aloud whether history has been falsified, persons and events invented from whole cloth, centuries added to our chronology, entire chapters to classic texts.

But in the age of conspiracy theories it’s entirely possible all of this was an asset rather than a liability. As I keep pointing out we live in strange times.

Representative passage:

The distinction between master races and the rest is simple and true, Hegel said it, copying Heraclitus: those peoples who choose death rather than slavery or submission in a confrontation that is a people of masters. There are many such in the world, not only among the Aryans, but also the Comanche, many of the Polynesians, the Japanese and many others. But animal of this kind refuses entrapment and subjection. It is very sad to witness those times when such animal can neither escape nor kill itself. I saw once a jaguar in zoo, behind a glass, so that all the bugs in hueman form could gawk at it and humiliate it. This animal felt a noble and persistent sadness, being observed everywhere by the obsequious monkeys, not even monkeys, that were taunting it with stares. His sadness crushed me and I will always remember this animal. I never want to see life in this condition!

Who should read this book?

I think the people who are inclined to read this book are going to read it regardless of what I say. For those who aren’t in that category, I would not recommend this book to anyone, except as an anthropological exercise.

Why Are The Prices So Damn High?

By: Eric Helland, Alex Tabarrok

90 pages

Thoughts

This book is an attempt to explain rising prices in health care and education by tying them to the Baumol Effect. Here’s how Helland and Tabarrok describe it:

In 1826, when Beethoven’s String Quartet No. 14 was first played, it took four people 40 minutes to produce a performance. In 2010, it still took four people 40 minutes to produce a performance. Stated differently, in the nearly 200 years between 1826 and 2010, there was no growth in string quartet labor productivity. In 1826 it took 2.66 labor hours to produce one unit of output, and it took 2.66 labor hours to produce one unit of output in 2010.

Fortunately, most other sectors of the economy have experienced substantial growth in labor productivity since 1826. We can measure growth in labor productivity in the economy as a whole by looking at the growth in real wages. In 1826 the average hourly wage for a production worker was $1.14. In 2010 the average hourly wage for a production worker was $26.44, approximately 23 times higher in real (inflation-adjusted) terms. Growth in average labor productivity has a surprising implication: it makes the output of slow productivity-growth sectors (relatively) more expensive. In 1826, the average wage of $1.14 meant that the 2.66 hours needed to produce a performance of Beethoven’s String Quartet No. 14 had an opportunity cost of just $3.02. At a wage of $26.44, the 2.66 hours of labor in music production had an opportunity cost of $70.33. Thus, in 2010 it was 23 times (70.33/3.02) more expensive to produce a performance of Beethoven’s String Quartet No. 14 than in 1826. In other words, one had to give up more other goods and services to produce a music performance in 2010 than one did in 1826. Why? Simply because in 2010, society was better at producing other goods and services than in 1826.

Scott Alexander also did a couple of posts on the book, and as you might expect his posts go into more depth (in fact I borrowed the above selection from one of them.) I largely agree with his general assessment, which is that the Baumol Effect explains quite a bit, but it doesn’t seem to explain as much as Helland and Tabarrok claim. In particular it can’t seem to explain why subway systems cost 50 times as much to construct in New York as in Seoul, South Korea

Who should read this book?

If you have a deep desire to understand the arguments around the why costs in some sectors are growing much faster than inflation then you should read this book. Otherwise, it’s main contribution is to more fully popularize the Baumol Effect which is easy enough to understand without reading an entire (albeit short) book.

An Introduction to the Book of Abraham (Religious)

By: John Gee

196 pages

Thoughts

Within The Church of Jesus Christ of Latter-day Saints (LDS) the Book of Abraham is canonized scripture, and members of the Church (myself included) believe that Joseph Smith translated the book from some papyri. Smith purchased the papyri from a gentleman with a traveling mummy exhibition in 1835. Critics of the church feel that that the circumstances of the translation, along with advances in Egyptology which have occured since Smith’s translation, the most important being the ability to translate Egyptian hieroglyphs, all combine to provide a fruitful avenue for attacking the church. Accordingly, a significant amount of criticism has been leveled towards the Book of Abraham. An Introduction to the Book of Abraham designed to examine this criticism from an apologetic basis.

For obvious reasons I am not objective on this topic. Nevertheless I feel that Gee did an excellent and credible job. His approach seemed both rigorous and scholarly. I know that there are many people who feel that some criticisms Book of Abraham are impossible to refute, but this book provided many avenues of refutation, none of them were ironclad anymore than the criticisms were ironclad, but neither did they require any handwaving.

Who should read this book?

Anyone who is even moderately interested in LDS apologetics in general and the Book of Abraham in particular should read this book. I quite enjoyed it, and had the book been twice as long I wouldn’t have minded it.

The Lies of Locke Lamora (Gentleman Bastard #1)

By: Scott Lynch

736 pages

Thoughts

My habit of starting new fantasy/scifi series while completely ignoring series I have already started continues with this book, which is part of yet another fantasy series. This particular book came highly recommended by frequent commentator Mark (see his excellent science/etc blog) and I was not disappointed, it was a thoroughly enjoyable read with a great ending. That said I do have several quibbles.

Criticisms

For some reason, and I’m not blaming Mark, or the blurb on Amazon, I had the impression when I picked up this book (metaphorically, I actually downloaded it from Audible) that it was going to be sort of a fantasy Oceans 11, and there was quite a bit of lighthearted capering in the book, but it was also pretty dark. I don’t recall anyone dying in Oceans 11, but lots of people die in Locke Lamora. The combination of the two made the tone a little schizophrenic.

Additionally, and I’ve mentioned this before, There are a class of fantasy and science fiction authors who write all of their characters as “sassy”. John Scalzi is the worst offender here, and as I think back on my misspent youth, David Eddings may have pioneered the genre, and it turns out Lynch is also an offender but a minor one.

Finally there is one bit of world building that drove me absolutely nuts. I don’t want to say much more than that for fear of spoiling things, but there are implications to this thing which he entirely fails to consider. But if you can overlook this one thing (which is what I eventually decided to do) or if you don’t notice the problems it would cause, then, as I said, it’s a thoroughly enjoyable read.

I think going forward I’m going to try to finish some of the series I’ve started rather than beginning anything new. Time will tell.

No More Mr Nice Guy: A Proven Plan for Getting What You Want in Love, Sex, and Life

By: Robert A. Glover

208 pages

Thoughts

You may recall my review of Wild at Heart. Well one of the things people do after reading that book is go on a retreat with a large group of other Christian men. I was one of those people, and last month I went on just such a retreat, and it was awesome, and not merely because it was in Alaska. In essence, that book, the retreat, No More Mr. Nice Guy and Bronze Age Mindset are all attempting to answer the same question. What advice should you give to men who feel alienated and abandoned, particularly by women? The retreat, in addition to being one of those answers was also where I heard about No More Mr. Nice Guy, and it’s answer to the question should be pretty obvious from the title, though it’s less antisocial and misogynist than you might imagine.

Glover asserts that a large part of the problem is that a significant portion of men have responded to these feelings of abandonment by assuming that if they just make themselves completely subject to the needs of the women in their life that they will be embraced rather than abandoned. As you can imagine, deriving the entirety of your validation from someone else is a disaster basically regardless of the philosophy you subscribe to. 

Beyond that, there are numerous additional details, but there’s nothing in the book which advocates cruelty, which probably puts it ahead of BAM, and if I were to go on from that and rank all four of these vectors on the quality of their answer to “the question” I would put the retreat first, followed by Wild at Heart followed by this book with BAM last of all. But as the first two come with implicit Christian overtones, No More Mr. Nice Guy might end up at the top of the list for a lot of people. That said, I wouldn’t recommend it unreservedly, or blindly. I’d want to know quite a bit about a person’s situation.

Deep Survival: Who Lives, Who Dies, and Why

By: Laurence Gonzales

336 pages

Thoughts

As you might have surmised this is another recommendation from the little old lady. Though I guess it must be popular among the 70+ set because I just discovered that both of my parents have read it as well.

This book, rather than being the story of a single instance of survival, collects numerous survival stories, looking for commonalities; for what makes someone good at survival. The book spends a lot of time on Steve Callahan, who I mentioned above (this is the book that declared him to be one of the best survivors). It also includes the incident chronicled in the movie Touching the Void which I talked about previously in this space.

Of course, you’re probably less interested in what stories it includes and more interested in the qualities which are going to keep you alive when the zombie apocalypse comes. If you’ve read the book Thinking, Fast and Slow by Daniel Kahneman then Gonzales’ framework will probably seem familiar. Kahneman talks about things we do more or less instinctually and things we do rationally. Gonzalez has the same basic division, but he further divides the instinctual part of things in two. Giving him three categories:

  1. Built in instinctual behaviors, like trying to grab onto something if you start to fall.
  2. Learned instinctual behaviors, i.e. adrenaline junkies, people with PTSD.
  3. Behaviors you have to think about.

At various times survival requires alternatively ignoring or emphasizing some or all of the above behaviors, depending on the circumstance. You may need to use humor to overcome your instinctive fear of death (category 1). You may need to develop an instinctive love for certain dangerous things (category 2) but not to the point that it overrides your rationality (category 3).

Allow me to illustrate what I mean. First off, it’s interesting to note that some of the best survivors are children under the age of seven. In part because their behaviors are almost entirely from category one. Which means that they sleep when they’re tired, try to get warm when they’re cold, and drink when they’re thirsty. They are also unlikely to use more energy than necessary. Contrast that with the story Gonzalez includes of a volunteer firefighter who got lost while backpacking and nearly died. He had a learned instinct of not wanting to admit when he was lost. As a firefighter he knew it was illegal to light a fire, so he avoided doing so for several days (some from column two some from column three) and he spent lots of time trying to get to the tops of nearby peaks so he could see better. Exhausting himself in the process.

From the preceding it might seem that you mostly want to avoid category two behaviors and even category three, but if soldiers in World War I didn’t learn to instinctively jump for cover when they heard the whistle of an artillery shell than they weren’t going to survive very long. And Steve Callahan only survived by making lots of very rational decisions. As you might imagine surviving requires doing a lot of things right, and some luck on top of that as well.

Who should read this book?

As I mentioned earlier, those aged 70 and over apparently really like this book, probably because they sense the steady encroachment of death, if you also sense the steady encroachment of death (whether because your 70+ or otherwise) then you’ll probably also enjoy it.


If you haven’t guessed that last bit was in part a joke at my parents’ expense. (Hi Mom!) If my blatant lack of filial piety appeals to you consider donating


Returning to Mormonism and AI (Part 3)

If you prefer to listen rather than read, this blog is available as a podcast here. Or if you want to listen to just this post:

Or download the MP3


This is the final post in my series examining the connection between Mormonism and Artificial Intelligence (AI). I would advise reading both of the previous posts before reading this one (Links: Part One, Part Two), but if you don’t, here’s where we left off:

Many people who’ve made a deep study of artificial intelligence feel that we’re potentially very close to creating a conscious artificial intelligence. That is, a free-willed entity, which, by virtue being artificial would have no upper limit to its intelligence, and also no built in morality. More importantly, insofar as intelligence equals power (and there’s good evidence that it does). We may be on the verge of creating something with godlike abilities. Given, as I just said, that it will have no built in morality, how do we ensure that it doesn’t use it’s powers for evil? Leading to the question, how do you ensure that something as alien as an artificial consciousness ends up being humanity’s superhero and not our archenemy?

In the last post I opined that the best way to test the morality of an AI would be to isolate it and then give it lots of moral choices where it’s hard to make the right choice and easy to make the wrong choice. I then pointed out that this resembles the tenets of several religions I know, most especially my own faith, Mormonism. Despite the title, the first two posts were very light on religion in general and Mormonism is particular. This post will rectify that, and then some. It will be all about the religious parallels between this method for testing an AI’s morality and Mormon Theology.

This series was born as a reexamination of a post I made back in October where I compared AI research to Mormon Doctrine. And I’m going to start by revisiting that, though hopefully, for those already familiar with October’s post, from a slightly different angle.

To begin our discussion, Mormons believe in the concept of a pre-existence, that we lived as spirits before coming to this Earth. We are not the only religion to believe in a pre-existence, but most Christians (specifically those who accept the Second Council of Constantinople) do not. And among those christian sects and other religions who do believe in it, Mormons take the idea farther than anyone.

As a source for this, in addition to divine revelation, Mormons will point to the Book of Abraham, a book of scripture translated from papyrus by Joseph Smith and first published in 1842. From that book, this section in particular is relevant to our discussion:

Now the Lord had shown unto me, Abraham, the intelligences that were organized before the world was…And there stood one among them that was like unto God, and he said unto those who were with him: We will go down, for there is space there, and we will take of these materials, and we will make an earth whereon these may dwell; And we will prove them herewith, to see if they will do all things whatsoever the Lord their God shall command them;

If you’ve been following along with me for the last two posts then I’m sure the word “intelligences” jumped out at you as you read that selection. But you may have also have noticed the phrase, “And we will prove them herewith, to see if they will do all things whatsoever the Lord their God shall command them;” And the selection, taken as a whole, depicts a situation very similar to what I described in my last post, that is, creating an environment to isolate intelligences while we test their morality.

I need to add one final thing before the comparison is complete. While not explicitly stated in the selection, we, as Mormons, believe that this life is a test is to prepare us to become gods in our own right. With that final piece in place we can take the three steps I listed in the last post with respect to AI researchers and compare them to the three steps outlined in Mormon theology:

AI: We are on the verge of creating artificial intelligence.

Mormons: A group of intelligences exist.

AI: We need to ensure that they will be moral.

Mormons: They needed to be proved.

Both: In order to be able to trust them with godlike power.

Now that the parallels between the two endeavors are clear, I think that much of what people have traditionally seen as problems with religion end up being logical consequences flowing naturally out of a system for testing morality.

The rest of this post will cover some of these traditional problems and look at them from both the “creating a moral AI” standpoint and the “LDS theology” standpoint. (Hereafter I’ll just use AI and LDS as shorthand.) But before I get to that, it is important to acknowledge that the two systems are not completely identical. In fact there are many ways in which they are very different.

First when it comes to morality, we can’t be entirely sure that the values we want to impart to an AI are actually the best values for it to have. In fact many AI theorists, have put forth the “Principle of Epistemic Deference”, which states:

A future superintelligence occupies an epistemically superior vantage point: it’s beliefs are (probably, on most topics) more likely than ours to be true. We should therefore defer to the superintelligence’s opinion whenever feasible.

No one would suggest that God has a similar policy of deferring to us on what’s true and what’s not. And therefore the LDS side of things has a presumed moral clarity underlying it which the AI side does not.

Second, when speaking of the development of AI it is generally assumed that the AI could be both smarter and more powerful than the people who created it. On the religious/LDS side of things there is a strong assumption in the other direction, that we are never going to be smarter or more powerful than our creator. This doesn’t change the need to test the morality, but it does make the consequences of being wrong a lot different for us than for God.

Finally, while in the end, we might only need a single, well-behaved AI to get us all of the advantages of a superintelligent entity, it’s clear that God wants to exalt as many people as possible. Meaning that on the AI side of things the selection process could, in theory, be a lot more draconian. While from an LDS perspective, you might expect things to be tough, but not impossible.

These three things are big differences, but none of them represents something which negates the core similarities. But they are something to keep in mind as we move forward and I will occasionally reference them as I go through the various similarities between the two systems.

To being with, as I just mentioned one difference between the AI and LDS models is how confident we are in what the correct morality should be, with some AI theorists speculating that we might actually want to defer to the AI on certain matters of morality and truth. Perhaps that’s true, but you could imagine that some aspects of morality are non-negotiable, for example you wouldn’t want to defer to the AIs conclusion that humanity is inferior and we should all be wiped out, however ironclad the AI’s reasons ended up being.

In fact, when we consider the possibility that AIs might have a very different morality from our own, an AI that was unquestioningly obedient would solve many of the potential problems. Obviously it would also introduce different problems. Certainly you wouldn’t want your standard villain type to get a hold of a superintelligent AI who just did whatever it was told, but also no one would question an AI researcher who told the AI to do something counterintuitive to see what it would do. And yet, just today I saw someone talk about how it’s inconceivable that the true God should really care if we eat pork, apparently concluding that obedience has no value on it’s own.

And, as useful as this is when in the realm of our questionable morality, how much more useful and important is it to be obedient when we turn to the LDS/religious side of things and the perfect morality of God?

We see many examples of this. The one familiar to most people would be when God commanded Abraham to sacrifice Isaac. This certainly falls into the category of something that’s counterintuitive, not merely based on the fact that murder is wrong, but also God had promised Abraham that he would have descendents as numerous as the stars in the sky, which is hard when you’ve killed your only child. And yet despite this Abraham went ahead with it and was greatly rewarded for his obedience.

Is this something you’d want to try on an AI? I don’t see why not. It certainly would tell you a lot about what sort of AI you were dealing with. And if you had an AI that seemed otherwise very moral, but was also willing to do what you asked because you asked it, that might be exactly what you were looking for.

For many people the existence of evil and the presence of suffering are both all the proof they need to conclude that God does not exist. But as you may already be able to see, both from this post and my last post, any test of morality, whether it be testing AIs or testing souls, has to include the existence of evil. If you can’t make bad choices then you’re not choosing at all, you’re following a script. And bad choices are, by definition evil, (particularly choices as consequential as those made by someone with godlike power). To put it another way, a multiple choice test where there’s only one answer and it’s always the right one, doesn’t tell you anything about the subject you’re testing. Evil has to exist, if you want to know whether someone is good.

Furthermore, evil isn’t merely required to exist. It has to be tempting. To return to the example of the multiple choice test, even if you add additional choices, you haven’t improved the test very much if the correct choice is always in bold with a red arrow pointing at it. If good choices are the only obvious choices then you’re not testing morality, you’re testing observation. You also very much risk making the nature of the test transparent to a sufficiently intelligent AI, giving it a clear path to “pass the test” but in a way where it’s true goals are never revealed. And even if they don’t understand the nature of the test they still might always make the right choice just by following the path of least resistance.

This leads us straight to the idea of suffering. As you have probably already figured out, it’s not sufficient that good choices be the equal of every other choice. They should actually be hard, to the point where they’re painful. A multiple choice test might be sufficient to determine whether someone should be given an A in Algebra, but both the AI and LDS tests are looking for a lot more than that. Those tests are looking for someone (or something) that can be trusted with functional omnipotence. When you consider that, you move from thinking of it in terms of a multiple choice question to thinking of it more like qualifying to be a Navy SEAL, only perhaps times ten.

As I’ve said repeatedly, the key difficulty for anyone working with an AI, is determining its true preference. Any preference which can be expressed painlessly and also happens to match what the researcher is looking for is immediately suspect. This makes suffering mandatory. But what’s also interesting is that you wouldn’t necessarily want it to be directed suffering. You wouldn’t want the suffering to end up being the red arrow pointing at the bolded correct answer. Because then you’ve made the test just as obvious but from the opposite direction. As a result suffering has to be mostly random. Bad things have to happen to good people, and wickedness has to frequently prosper. In the end, as I mentioned in the last point, it may be that the best judge of morality is whether someone is willing to follow a commandment just because it’s a commandment.

Regardless of its precise structure, in the end, it has to be difficult for the AI to be good, and easy for it to be bad. The researcher has to err on the side of rejection, since releasing a bad AI with godlike powers could be the last mistake we ever make. Basically, the harder the test the greater its accuracy, which makes suffering essential.

Next, I want to look at the idea that AIs are going to be hard to understand. They won’t think like we do, they won’t value the same things we value. They may, in fact, have a mindset so profoundly alien that we don’t understand them at all. But we might have a resource that would help. There’s every reason to suspect that other AIs created using the same methodology, would understand their AI siblings much better than we do.

This leads to two interesting conclusions both of which tie into religion, the first I mentioned in my initial post back in October. But I also alluded to it in the previous posts in this series. If we need to give the AIs the opportunity to sin, as I talked about in the last point. Then any AIs who have sinned are tainted and suspect. We have no idea whether their “sin” represented their true morals which they have now chosen to hide from us, or whether they have sincerely and fully  repented. Particularly if we assume an alien mindset. But if we have an AI built on a similar model which never sinned that AI falls into a special category. And we might reasonably decide to trust it with the role of spokesperson for the other AIs.

In my October post I drew a comparison between this perfect AI, vouching for the other AIs, and Jesus acting as a Messiah. But in the intervening months since then, I realized that there was a way to expand things to make the fit even better. One expects that you might be able to record or log the experiences of a given AI. If you then gave that recording to the “perfect” AI, and allowed it to experience the life of the less perfect AIs you would expect that it could offer a very definitive judgement as whether a given AI had repented or not.

For those who haven’t made the connection, from a religious perspective, I’ve just described a process that looks very similar to a method whereby Jesus could have taken all of our sins upon himself.

I said there were two conclusions. The second works exactly the opposite of the first. We have talked of the need for AIs to be tempted, to make them have to work at being moral, but once again their alien mindset gets in the way. How do we know what’s tempting to an artificial consciousness? How do we know what works and what doesn’t? Once again other AIs probably have a better insight into their AI siblings, and given the rigor of our process certain AIs have almost certainly failed the vetting process. I discussed the moral implications of “killing” these failed AIs, but it may be unclear what else to do. How about allowing them to tempt the AIs who we’re still testing? Knowing that the temptations that they invent will be more tailored to the other AIs than anything we could come up with. Also, insofar as they experience emotions like anger and jealously and envy they could end up being very motivated to drag down those AIs who have, in essence, gone on without them.

In LDS doctrine, we see exactly this scenario. We believe that when it came time to agree to the test, Satan (or Lucifer as he was then called) refused and took a third of the initial intelligences with him (what we like to refer to as the host of heaven) And we believe that those intelligences are allowed to tempt us here on earth. Another example of something which seems inexplicable when viewed from the standpoint of most people’s vague concept of how benevolence should work, but which makes perfect sense if you imagine what you might do if you were testing the morality of an AI (or spirit).

This ties into the next thing I want to discuss. The problem of Hell. As I just alluded to, most people only have a vague idea of how benevolence should look. Which I think actually boils down to, “Nothing bad should ever happen.” And eternal punishment in Hell is yet another thing which definitely doesn’t fit. Particularly in a world where steps have been taken to make evil attractive. I just mentioned Satan, and most people think he is already in Hell, and yet he is also allowed to tempt people. Looking at this from the perspective of an AI, perhaps this is as good as it gets. Perhaps being allowed to tempt the other AIs is the absolute most interesting, most pleasurable thing they can do because it allows them to challenge themselves against similarly intelligent creations.

Of course, if you have the chance to become a god and you miss out on it because you’re not moral enough, then it doesn’t matter what second place is, it’s going to be awful, relative to what could have been. Perhaps there’s no way around that, and because of this it’s fair to describe that situation as Hell. But that doesn’t mean that it couldn’t actually, objectively, be the best life possible for all of the spirits/AIs that didn’t make it. We can imagine some scenarios that are actually enjoyable if there’s no actual punishment, it’s just a halt to progression.

Obviously this and most of the stuff I’ve suggested is just wild speculation. My main point is that by viewing this life as a test of morality, a test to qualify for godlike power (which the LDS do) provides a solution to many of the supposed problems with God and religion. And the fact that AI research has arrived a similar point and come to similar conclusions, supports this. I don’t claim that by imagining how we would make artificial intelligence moral that all of the questions people have ever had about religion are suddenly answered. But I think it gives a surprising amount of insight to many of the most intractable questions. Questions which atheists and unbelievers have used to bludgeon religion for thousands of years, questions which may turn out to have an obvious answer if we just look at it from the right perspective.


Contrary to what you might think, wild speculation is not easy, it takes time and effort. If you enjoy occasionally dipping into wild speculation, then consider donating.


Returning to Mormonism and AI (Part 2)

If you prefer to listen rather than read, this blog is available as a podcast here. Or if you want to listen to just this post:

Or download the MP3


This post is a continuation of the last post. If you haven’t read that post, you’re probably fine, but if you’d like to you can find it here. When we ended last week we had established three things:

1- Artificial intelligence technology is advancing rapidly. (Self-driving cars being great example of this.) Many people think this means we will have a fully conscious, science fiction-level artificial intelligence in the next few decades.

2- Since you can always add more of whatever got you the AI in the first place, conscious AIs could scale up in a way that makes them very powerful.

3- Being entirely artificial and free from culture and evolution, there is no reason to assume that conscious AIs would have a morality similar to ours or any morality at all.

Combining these three things together, the potential exists that we could very shortly create a entity with godlike power that has no respect for human life or values. Leaving me to end the last post with the question, “What can we do to prevent this catastrophe from happening?”

As I said the danger comes from combining all three of the points above. A disruption to any one of them would lessen, if not entirely eliminate, the danger. With this in mind, everyone’s first instinct might be to solve the problem with laws and regulations. If our first point is that AI is advancing rapidly then we could pass laws to slow things down, which is what Elon Musk suggested recently. This is probably a good idea, but it’s hard to say how effective it will be. You may have noticed that perfect obedience to a law is exceedingly rare, and there’s no reason to think that laws prohibiting the development of conscious AIs would be the exception. And even if they were, every nation on Earth would have to pass such laws. This seems unlikely to happen and even more unlikely to be effective.

One reason why these laws and regulations wouldn’t be very effective is that there’s good reason to believe that developing a conscious AI, if it can be done, would not necessarily require something like the Manhattan Project to accomplish. And even if it does, if Moore’s Law continues, what was a state of the art supercomputer in 2020 will be available in a gaming console in 2040. Meaning that if you decide to regulate supercomputers today in 30-40 years you’ll have to regulate smart thermostats.

Sticking with our first point, another possible disruption is the evidence that consciousness is a lot harder than we think. And many of the people working in the field of AI have said that the kind of existential threat that I (and Stephen Hawking, and Elon Musk and Bill Gates) are talking about is centuries away. I don’t think anyone is saying it’s impossible, but there are many people who think it’s far enough out that while it might still be a problem it won’t be our problem, it will be our great-great grandchildren’s problem, and presumably they’ll have much better tools for dealing with it. Also, as I said in the last post I’m on record as saying we won’t develop artificial consciousness, but I’d also be the last person to say that this means we can ignore the potential danger. And, it is precisely the potential danger, which makes hoping that artificial consciousness is really hard, and a long way away, a terrible solution.

I understand the arguments for why consciousness is a long ways away, and as I just pointed out I even agree with them. But this is one of those “But what if we’re wrong?” scenarios, where we can’t afford to be wrong. Thus, while I’m all for trying to craft some laws and regulations, and I agree that artificial consciousness probably won’t happen, I don’t think either hope or laws represent an adequate precaution. Particularly for those people who really are concerned.

Moving to our second point, easily scalable power, any attempts to limit this through laws and regulations would suffer problems similar to attempting to slow down their development in the first place. First, what keeps a rogue actor from exceeding the “UN Standards for CPUs in an Artificial Entity”? When we can’t even keep North Korea from developing ICBMs? And, again, if Moore’s Law continues to hold then whatever power you trying to limit, is going to become more and more accessible to a broader and broader range of individuals. And, more frighteningly, on this count we might have the AI itself working against us.

Imagine a situation where we fail in our attempts to stop the development of AI, but our fallback position is to limit how powerful of a computer the AI can inhabit. And further imagine that miraculously the danger is so great that we have all of humanity on board. Well then we still wouldn’t have all sentient entities on board, because AIs would have all manner of intrinsic motivation to increase their computing power. This represents a wrinkle that many people don’t consider. However much you get people on board with things when you’re talking about AI, there’s a fifth column to the whole discussion that desperately wants all of your precautions to fail.

Having eliminated, as ineffective, any solutions involving controls or limits on the first two areas, the only remaining solution is to somehow instill morality in our AI creations. For people raised on Asimov and his Three Laws of Robotics this may seem straightforward, but it presents some interesting and non-obvious problems.

If you’ve read much Asimov you know that, with the exception of a couple of stories, the Laws of Robotics were embedded so deeply that they could not be ignored or reprogrammed. They were an “inalienable part of the mathematical foundation underlying the positronic brain.” Essentially meaning, the laws were impossible to change. For the moment, let’s assume that this is possible, that we can embed instructions so firmly within an AI that it can’t change them. This seems improbable right out of the gate given that the whole point of a computer is it’s ability to be programmed and for that programming to change. But we will set that objection aside for the moment and assume that we can embed some core morality within the AI in a fashion similar to Asimov’s laws of robotics. In other words, in such a way that the AI has no choice but to follow them.

You might think, “Great! Problem solved”. But, in fact we haven’t even begun to solve the problem:

First, even if we can embed that functionality in our AIs, and even if, despite being conscious and free-willed, they have no choice but to obey those laws, we still have no guarantee that they will interpret the laws the same way we do. Those who pay close attention to the Supreme Court know exactly what I’m talking about.

Or, to use another example, stories are full of supernatural beings who grant wishes, but in the process, twist the wish and fulfill it in such a way that the person would rather not have made the wish in the first place. There are lots of reasons to worry about this exact thing happening with conscious AIs. First whatever laws or goals we embedded, if the AI is conscious it would almost certainly have it’s own goals and desires and would inevitably interpret whatever morality we’ve embedded in way which best advances those goals and desires. In essence, fulfilling the letter of the law but not its spirit.

If an AI twists things to suit its own goals we might call that evil, particularly if we don’t agree with it’s goals, but you could also imagine a “good” AI that really wants to follow the laws, and which doesn’t have any goals and desires beyond the morality we’ve embedded, but still ends up doing something objectively horrible.

Returning to Asimov’s laws, let’s look at the first two:

  1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.
  2. A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.

One possible interpretation of the first law would be to round up all the humans (tranquilize them if they resist) and put them in a padded room with a toilet and meal bars delivered at regular intervals. In other words one possible interpretation of the First Law of Robotics is to put all the humans in a very comfy, very safe prison.

You could order them not to, which is the second law, but they are instructed to ignore the second law if it conflicts with the first law. These actions may seem evil based on the outcome, but this could all come about from a robot doing it’s very best to obey the first law, which is what, in theory, we want. Returning briefly to examine how an “evil” AI might twist things. You could imagine this same scenario ending in something which very much resembling The Matrix, and all the AI would need is a slightly fluid definition of the word injury.

There have been various attempts to get around this. Eliezer Yudkowsky, a researcher I’ve mentioned in previous posts on AI, suggests that rather than being given a law that AIs be given a goal, and he provides an example which he calls humanities “coherent extrapolated volition” (CEV):

Our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted.

I hope the AI understands it better than I do, though to be fair Yudkowsky doesn’t offer it up as some kind of final word but as a promising direction. Sort of along the lines of telling the genie that we want to wish for whatever the wisest man in the world would wish for.

All of this is great, but it doesn’t matter how clever our initial programming is, or how poetic the construction the AIs goal. We’re going want to conduct the same testing to see if it works as we would if we had no laws or goals embedded.

And here at last we hopefully have reached the meat of things. How do you test your AI for morality? As I mentioned in my last post this series is revisiting an earlier post I made in October of last year which compared Mormon Theology to Artificial Intelligence research particularly as laid out in the book Superintelligence by Nick Bostrom. In that earlier post I listed three points on the path to conscious artificial intelligence:

1- We are on the verge of creating artificial intelligence.

2- We need to ensure that they will be moral.

3- In order to be able to trust them with godlike power.

This extended series has now arrived at the same place, and we’re ready to tackle the issue which stands at the crux things: The only way to ensure that AIs aren’t dangerous (potentially, end of humanity dangerous) is to make sure that the AIs are moral. So the central question is how do we test for morality?

Well to begin, the first, obvious step, is to isolate the AIs until their morality can be determined. This isolation allows us to prevent them from causing any harm, gives us an opportunity to study them, and also keeps them from increasing their capabilities by denying them access to additional resources.

There are of course some worries about whether we would be able to perfectly isolate an AI given how connected the world is, and also given the fact humanity has a well known susceptibility to social engineering, (i.e. the AI might talk it’s way out) but despite this, I think most people agree that isolation is an easier problem than creating a method to embed morality right from the start in a foolproof manner.

Okay, so you’ve got them isolated. But this doesn’t get you to the point where you’re actually testing their morality, this just gets you to the point where failure is not fatal. But isolation carries some problems. You certainly wouldn’t want them to experience the isolation as such. If you stick your AIs in the equivalent of a featureless room for the equivalent of eternity, I doubt anyone would consider that an adequate test of their morality, since it’s either too easy or too unrealistic. (Also if there’s any chance your AI will go insane this would certainly cause it.) Accordingly you’d want in addition to the isolation, the ability to control their environment, to create a world, but what sort of world would you want to create? It seems self-evident that you’d want to create something that resembled the real world as much as possible. The advantages to this should be obvious. You want to ensure that the AI will act morally in the world we inhabit with all of the limitations and opportunities that exist in that world. If you create a virtual world that has different limitations and different opportunities, then it’s not a very good test. Also this setup would present them with all the moral choices they might otherwise have and you could observe which choices they make, and choices are the essence of morality.

While putting a “baby” AI in a virtual world to see what it does is interesting. It might not tell us very much. And here’s where we return to the embedded law, whether it’s something like the three laws of robotics or whether it’s more like Yudkowsky’s CEV. As I mentioned, regardless of whether you have embedded morality or not you’re going to need to test, but I also can’t think of any reason to not try providing some kind of direction with respect to morality. One could imagine an AI doing all sorts of things if it was placed in a virgin world without any direction, and how could you know if it was doing those things because it was “evil” or whether it was doing them because it didn’t know any better. So, as I said, there’s no reason not to give it some kind of moral guidelines up front.

A discussion of what morality is, and what those guidelines should be, beyond the examples already given, is beyond the scope of this post. But if we assume that some guidelines have been given, then at that point the AI being tested can do one of two things: it can follow the guidelines perfectly or it can violate them. What happens if it violates them? You could make arguments that it would depend on what the guidelines were and how it violated them. You could also make arguments that the AI might be smarter than us and it might have had a very good reason for violating them. And all of these arguments are valid, but the danger of getting it wrong is so great, and the creation of another AI would, presumably, be so easy that it’s hard to imagine you wouldn’t just get rid of the AI who violated the guidelines. Even if the infraction was minor. Also as Bostrom points out, if we “forgive” the AI, then there’s the danger that it will understand the nature of the test and the consequences of failure. And from that time forward it act perfectly, not because it’s moral, but because it wants to avoid destruction. In this circumstance the AI hides its true intentions, meaning that we never know what sort of morality it has, and we end up defeating the whole process.

As aside, when speaking of getting rid of AIs, there’s a whole ethical minefield to grapple with. If we have in fact created sentient AIs then it could certainly be argued that getting rid of them is the equivalent of murder. We’ll come back to this issue later, but I thought I’d mention it while it was fresh.

So that’s how we handle AIs that don’t follow the guidelines, but what do we do with AIs that did follow the guidelines, that were perfect? You may think the solution is obvious, that we release them and give them the godlike power that is their birthright.

Are you sure about that? We are after all talking about godlike power. You can’t be a little bit sure about their morality, you have to be absolutely positive. What tests did you subject it to? How hard was it to follow our moral guidelines? Was the wrong choice even available? Were wrong choices always obviously the wrong choice or was there something enticing about the wrong choice? Maybe something that gave the AI a short term advantage over the right choice? Did the guidelines ever instruct them to do something where the point wasn’t obvious? Did the AI do it anyway, despite the ambiguity? Most of all, did they make the right choice even when they had to suffer for it?

To get back to our central dilemma, really testing for morality, to the point where you can trust that entity with godlike powers, implies creating a situation where being moral can’t have been easy or straight forward. In the end, if we really want to be certain, we have to have thrown everything we can think of at this AI: temptations, suffering, evil, and requiring obedience just for the sake of obedience. It has to have been enticing and even “pleasurable” for the AI to make the wrong choice and the AI has to have rejected that wrong choice every time despite all that.

One of my readers mentioned that after my last post he was still unclear on the connection to Mormonism, and I confess that he will probably have a similar reaction after this post, but perhaps, here at the end, you can begin to see where this subject might have some connection to religion. Particularly things like the problem of evil and suffering. That will be the subject of the final post in this series. And I hope you’ll join me for it.


If you haven’t donated to this blog, it’s probably because it’s hard. But as we just saw, doing hard things is frequently a test of morality. Am I saying it’s immoral to not donate to the blog? Well if you’re enjoying it then maybe I am.


Returning to Mormonism and AI (Part 1)

If you prefer to listen rather than read, this blog is available as a podcast here. Or if you want to listen to just this post:

Or download the MP3


Last week, Scott Alexander, the author of SlateStarCodex, was passing through Salt Lake City and he invited all of his readers to a meetup. Due to my habit of always showing up early I was able to corner Scott for a few minutes and I ended up telling him about the fascinating overlap between Mormon theology and Nick Bostrom’s views on superintelligent AI. I was surprised (and frankly honored) when he called it the “highlight” of the meetup and linked to my original post on the subject.

Of course in the process of all this I went through and re-read the original post, and it wasn’t as straightforward or as lucid as I would have hoped. For one I wrote it before I vowed to avoid the curse of knowledge, and when I re-read it, specifically with that in mind I could see many places where I assumed certain bits of knowledge that not everyone would possess. This made me think I should revisit the subject. Even aside from my clarity or lack thereof, there’s certainly more that could be said.

In fact there’s so much to be said on the subject, that I’m thinking I might turn it into a book. (Those wishing to persuade or dissuade me on this endeavor should do so in the comments or you can always email me. Link in the sidebar just make sure to unspamify it.)

Accordingly, the next few posts will revisit the premise of the original, possibly from a slightly different angle. On top of that I want to focus in on and expand on a few things I brought up in the original post and then, finally, bring in some new stuff which has occurred to me since then. All the while assuming less background knowledge, and making the whole thing more straightforward. (Though there is always the danger that I will swing the pendulum too far the other way and I’ll dumb it down too much and make it boring. I suppose you’ll have to be the judge of that.)

With that throat clearing out of the way let’s talk about the current state of artificial intelligence, or AI, as most people refer to it. When you’re talking about AI, it’s important to clarify whether you’re talking about current technology like neural networks and voice recognition or whether you’re talking about the theoretical human level artificial intelligence of science fiction. While most people think that the former will lead to the latter, that’s by no means certain. However, things are progressing very quickly and if current AI is going to end up in a place so far only visited by science fiction authors, it will probably happen soon.

People underestimate the speed with which things are progressing because what was once impossible quickly loses it’s novelty the minute it becomes commonplace. One of my favorite quotes about artificial intelligence illustrates this point:

But a funny thing always happens, right after a machine does whatever it is that people previously declared a machine would never do. What happens is, that particular act is demoted from the rarefied world of “artificial intelligence”, to mere “automation” or “software engineering”.

As the quote points out, not only is AI progressing with amazing rapidity, but every time we figure out some aspect of it, it moves from being an exciting example of true machine intelligence into just another technology.

Computer Go, which has been in the news a lot lately, is one example of this. As recently as May of 2014 Wired magazine ran an article titled, The Mystery of Go, The Ancient Game That Computers Still Can’t Win, an in depth examination of why, even though we could build a computer that could beat the best human at Jeopardy! of all things, we were still a long ways away from computers who could beat the best human at Go. Exactly three years later AlphaGo beat Ke Jie the #1 ranked player in the world. And my impression was, that interest in this event which only three years ago Wired called “AI’s greatest unsolved riddle” was already fading, with the peak coming the year before when AlphaGo beat Lee Sedol. I assume part of this was because once AlphaGo proved it was competitive at the highest levels everyone figured it was only a matter of time and tuning before it was better than the best human.

Self-driving cars are another example of this. I can remember the DARPA Grand Challenge back in 2004, the first big test of self-driving cars, and at that point not a single competitor finished the course. Now Tesla is assuring people that they will do a coast to coast drive on autopilot (no touching of controls) by the end of this year. And most car companies expect to have significant automation by 2020.

I could give countless other examples in areas like image recognition, translation and writing, but hopefully, by this point, you’re already convinced that things are moving fast. If that’s the case, and if you’re of a precautionary bent like me, the next question is, when should we worry? And the answer to that depends on what you’re worried about. If you’re worried about AI taking your job, a subject I discussed in a previous post, then you should already be worried. If you’re worried about AIs being dangerous, then we need to look at how they might be dangerous.

We’ve already seen people die in accidents involving Tesla’s autopilot mode. And in a certain sense that means that AI is already dangerous. Though, given how dangerous driving is, I think self-driving cars will probably be far safer, comparatively speaking. And, so far, most examples of dangerous AI behavior have been, ultimately, ascribable to human error. The system has just been following instructions. And we can look back and see where, when confronted with an unusual situation, following instructions ended up being a bad thing, but at least we understood how it happened and in these circumstances we can change the instructions, or in the most extreme case we can take the car off the road. The danger comes when they’re no longer following instructions, and we can’t modify the instructions even if they were.

You may think that this situation is a long ways off. Or you may even think it’s impossible, given that computers need to be programmed, and humans have to have written that program. If that is what you’re thinking you might want to reconsider. One of the things which most people have overlooked in the rapid progress of AI over the last few years is it’s increasing opacity. Most of the advancement in AI has come from neural networks, and one weakness of neural networks is that it’s really difficult to identify how they arrived at a conclusion, because of the diffuse and organic way in which they work. This makes them more like the human brain, but consequently more difficult to reverse engineer. (I just read about a conference entirely devoted to this issue.)

As an example, one of the most common applications for AI these days is image recognition, which generally works by giving the system a bunch of pictures, and identifying which pictures have the thing you’re looking for and which don’t. So you might give the system 1000 pictures 500 of which have cats in them and 500 of which don’t. You tell the system which 500 are which and it attempts to identify what a cat looks like by analyzing all 1000 pictures. Once it’s done you give it a new set of pictures without any identification and see how good it is at as picking out pictures with cats in them. So far so good, and we can know how well it’s doing by comparing the system’s results vs. our own, since humans are actually quite talented at spotting cats. But imagine that instead of cats you want it to identify early stage breast cancer in mammograms.

In this case you’d feed it a bunch of mammograms and identify which women went on to develop cancer and which didn’t. Once the system is trained you could feed it new mammograms and ask it whether a preventative mastectomy or other intervention, is recommended. Let’s assume that it did recommend something, but the doctor’s didn’t see anything. Obviously the woman would want to know how the AI arrived at that conclusion, but honestly, with a neural network it’s nearly impossible to tell. You can’t ask it, you just have to hope that the system works. Leaving her in the position of having to trust the image recognition of the computer or taking her chances.

This is not idle speculation. To start with, many people believe that radiology is ripe for disruption by image recognition software. Additionally, doctors are notoriously bad at interpreting mammograms. According to Nate Silver’s book The Signal and the Noise, the false positive rate on mammograms is so high (10%) that for women in their forties, with a low base probability of having breast cancer in the first place, if a radiologist says your mammogram shows cancer it will be a false positive 90% of the time. Needless to say, there is a lot of room for improvement. But even if, by using AI image recognition, we were able to flip it so that we’re right 90% of the time rather than wrong 90% of the time, are women going to want to trust the AI’s diagnosis if the only reasoning we can provide is, “The computer said so?”

Distilling all of this down, two things are going on. AI is improving at an ever increasing rate, and at the same time it’s getting more difficult to identify how an AI reached any given decision. As we saw in the example of mammography we may be quickly reaching a point where we have lots of systems that are better than humans at what they do, and we will have to take their recommendations on faith. It’s not hard to see where people might consider this to be dangerous or, at least, scary and we’re still just talking about the AI technology which exists now, we haven’t even started talking about science fiction level AI. Which is where most of the alarm is actually focused. But you may still be unclear on the difference between the two sorts of AIs.

In referring to it as science fiction AI I’m hoping to draw your mind to the many fictional examples of artificial intelligence, whether it’s HAL from 2001, Data from Star Trek, Samantha in Her, C-3P0 from Star Wars or, my favorite, Marvin from A Hitchhiker’s Guide to the Galaxy. All of these examples are different from the current technology we’ve been discussing in two key ways:

1- They’re a general intelligence. Meaning, they can perform every purely intellectual exercise at least as well or better than the average human. With current technology all of our AIs can only really do one thing, though generally they do it very well. In other words, to go back to our example above, AlphaGo is great at Go, but would be relatively hopeless when it comes to taking on Kasparov in chess or trying to defeat Ken Jennings at Jeopardy! Though other AIs can do both (Deep Blue and Watson respectively.)

2- They have free will. Or at least they appear to. If their behavior is deterministic, its deterministic in a way we don’t understand. Which is to say they have their own goals and desires and can act in a way we find undesirable. HAL being perhaps the best example of this from the list above. I’m sorry Dave, I’m afraid I can’t do that.

These two qualities, taken together, are often labeled as consciousness. The first quality allows the AI to understand the world, and the second allows the AI to act on that understanding. And it’s not hard to see how these additional qualities increase the potential danger from AI, though of the two, the second, free will, is the more alarming. Particularly since if an AI does have it’s own goals and desires there’s absolutely no reason to assume that these goals and desires would bear any similarities to humanities’ goals and desires. It’s safer to assume that their goals and desires could be nearly anything, and within that space there are a lot of very plausible goals that end with humanity being enslaved (The Matrix) or extinct (Terminator).

Thus, another name for a science fiction AI is a conscious AI. And having seen the issues with the technology we already have you can only imagine what happens when we add consciousness into the mix. But why should that be? We currently have 7.5 billion conscious entities and barring the occasional Stalin and Hitler, they’re generally manageable. Why is an artificial intelligence with consciousness potentially so much more dangerous than a natural intelligence with consciousness? Well there are at least four reasons:

1- Greater intelligence: Human intelligence is limited by a number of things, the speed of neurons firing, the size of the brain, the limit on our working memory, etc. Artificial intelligence would not suffer from those same limitations. Once you’ve figured out how to create intelligence using a computer, you could always add more processors, more memory, more storage, etc. In other words as an artificial system you could add more of whatever got you the AI in the first place. Meaning that even if the AI was never more intelligent than the most intelligent human it still might think a thousand times faster, and be able to access a million times the information we can.

2- Self improving: I used this quote the last time I touched on this subject, but it’s such a good quote and it encapsulates the concept of self-improvement so completely that I’m going to use it again. It’s from I. J. Good (who worked with Turing to decrypt the Enigma machine), and he said it all the way back in 1965:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.

If you want to continue to use science fiction to help you visualize things, of the science fiction I listed above only Her describes an actual intelligence explosion, but if you bring books into the mix you have things like Neuromancer by William Gibson, or most of the Vernor Vinge Books.

3- Immortality: Above I mentioned Stalin and Hitler. They had many horrible qualities, but they had one good quality which eventually made up for all of their bad qualities. They died. AI’s probably won’t have that quality. To be blunt, this is good if they’re good, but bad if they’re bad. And it’s another reason why dealing with artificial consciousness is more difficult than dealing with natural consciousness.

4- Unclear morality: None of the other qualities are all that bad until you combine it with this final attribute of artificial intelligence, they have no built in morality. For humans, a large amount of our behavior and morality is coded into our genes, genes which are the result of billions of years of evolutionary pressure. The morality and behavior which isn’t coded by our genes is passed on by our culture, especially our parents. Conscious AIs won’t have any genes, they won’t have been subjected to any evolutionary pressure and they definitely won’t have any parents except in the most metaphorical sense. Without any of those things, it’s very unlikely that they will end up with a morality similar to our own. They might, but it’s certainly not the way to bet.

After considering these qualities it should be obvious why a conscious AI could be dangerous. But even so it’s probably worth spelling out a few possible scenarios:

First, most species act in ways that benefit themselves. Whether it’s humans valuing humans more highly than rats, or just the preference that comes from procreation. Giving birth to more rats is an act which benefits rats even if later the same rat engages another rat in a fight to the death over a piece of pizza. In the same way a conscious AI is likely to act in ways which benefit itself and possibly other AIs to the determinant of humanity. Whether that’s seizing resources we both want, or deciding that all available material (humans included) should be turned into a giant computer.

On the other hand, even if you imagine that humans actually manage to embed morality into a conscious AI, there are still lots of ways that could go wrong. Imagine, for example, that we have instructed the AI that we need to be happy with its behavior. And so it hooks us up to feeding tubes and puts an electrode into our brain which constantly stimulates the pleasure center. It may be obvious to us that this isn’t what we meant, but are we sure it will be obvious to the AI?

Finally, the two examples I’ve given so far presuppose some kind of conflict where the AI triumphs. And perhaps you think I’m exaggerating the potential danger by hand waving this step. But it’s important to remember that a conscious AI could be vastly more intelligent than we are. But even if it weren’t, there are many things it could do if it were only as intelligent as reasonably competent molecular biologist. Many people have talked about the threat of bioterrorism, especially the danger of a man-made disease being released. Fortunately this hasn’t happened, in large part because it would be unimaginably evil, but also because its effects wouldn’t be limited to the individuals enemies. An AI has no default reason to think bioterrorism is evil and it also wouldn’t be affected by the pathogen.

These three examples just barely scratch the surface of the potential dangers, but they should be sufficient to give one a sense of both the severity and scope of the problem. The obvious question which follows is how likely is all of this? Or to separate it into it’s two components how likely is our current AI technology to lead to true artificial consciousness? And if that happens how likely is it that this artificial consciousness will turn out to be dangerous?

As you can see, any individual’s estimation of the danger level is going to depend a lot on whether you think conscious AI is a natural outgrowth of the current technology, whether it will involve completely unrelated technology or whether it’s somewhere in between.

I personally think it’s somewhere in between, though much less of a straight shot from current technology than people think. In fact I am on record as saying that artificial consciousness won’t happen. You may be wondering, particularly a couple thousand words into things, why I’m just bringing that up. What’s the point of all this discussion if I don’t even think it’s going to happen? First I’m all in favor of taking precautions against unlikely events if the risk from those events is great enough. Second, just because I don’t think it’s going to happen doesn’t mean that no one thinks it’s going to happen, and my real interest is looking at how those people deal with the problem.

In conclusion, AI technology is getting better at an ever increasing rate, and it’s already hard to know how any given AI makes decisions. Whether current AI technology will shortly lead to AIs that are conscious is less certain, but if the current path does lead in that direction, then at the rate things are going we’ll get there pretty soon (as in the next few decades.)

If you are a person who is worried about this sort of thing. And there are a lot of them from well known names like Stephen Hawking, Elon Musk and Bill Gates to less well known people like Nick Bostrom, Eliezer Yudkowsky and Bill HIbbard then what can you do to make sure we don’t end up with a dangerous AI? Well, that will be the subject of the next post…


If you learned something new about AI consider donating, and if you didn’t learn anything new you should also consider donating to give me the time to make sure that next time you do learn something.


Artificial Intelligence and LDS Cosmology

If you prefer to listen rather than read, this blog is available as a podcast here. Or if you want to listen to just this post:

Or download the MP3


Technological advancement has changed nearly everything. Whether it’s communication, travel, marriage, children, food, money, etc. almost nothing has escaped being altered. This includes theology and religion. But here its impact is mostly viewed as a negative. Not only has scientific understanding taken many things previously thought to be mysterious and divine and made them straightforward and mundane, but religion has also come to be seen as inferior to science as a method for explaining how the world works. For many believers this is viewed as a disaster. For many non-believers it’s viewed as a long deserved death blow.

Of course, the impact has not been entirely negative. Certainly if considered from an LDS perspective, technology has made it possible to have a worldwide church, to travel effectively to faraway lands and to preach the gospel, to say nothing of making genealogy easier than ever. The recently concluded General Conference is a great example of this, with the benefits of broadcast technology and internet streaming to the whole world being both obvious and frequently mentioned. In addition to the more visible benefits of technology, there are other benefits both more obscure and more subtle. And it is one of these more obscure benefits which I plan to cover in this post. The benefit that technology gives us into the mind of God.

Bringing up a topic like the “mind of God” is bound to entail all manner of weighty historical knowledge, profound philosophical discussions, and a deep dive into the doctrines of various religions which I have no qualifications for undertaking.  Therefore I shall restrict myself to LDS theology or more specifically what Mormons often refer to as the Plan of Salvation. That said, as far as my limited research and even more limited understanding can uncover, LDS cosmology is unique in its straightforward description of God’s plan. Which I have always considered to be a major strength.

One technique that’s available to scientists and historians is modeling. When a scientist encounters something from the past that he doesn’t understand, or if he has a theory he wants to test, it can be illuminating to recreate the conditions as they existed, either virtually or through using the actual materials available at the time. Some examples of this include:

1- Thor Heyerdahl had a theory that people from South America could have settled Polynesia in the years before Columbus. In order to test this theory he built a balsa wood raft using native techniques and materials and then set out from Peru to see if it could actually be done. As it turns out it could. The question is still open as to whether that’s what actually happened, but after Heyerdahl’s trip no one dares to claim that it couldn’t have happened that way.

2- The Egyptian Pyramids have always been a source of mystery. One common observation is that Cleopatra lived closer to our time than to the time when the pyramids were constructed. (BTW, this statement will be true for another 400 years.) How was something so massive built so long ago? Recently it was determined, through re-enactment, that wetting the sand in front of the sleds made it much easier to drag the nearly 9000 lb rocks across the desert.

3- The tendency of humans to be altruistic has been a mystery since Darwin introduced evolution. While Darwin didn’t coin the term survival of the fittest it nevertheless fits fairly well, and appears to argue against any kind of cooperation. But when evolutionary biologists crafted computer models to represent the outcomes of various evolutionary strategies they discovered that altruism was the most successful strategy. In particular, as I mentioned in my last post, the tit-for-tat strategy performed very well.

Tying everything together, after many years of technological progress, we are finally in a position to do the same sort of reconstruction and modeling with God’s plan. Specifically what his plan was for us.

When speaking of God’s intentions the Book of Abraham is invaluable. This section in particular is relevant to our discussion:

Now the Lord had shown unto me, Abraham, the intelligences that were organized before the world was…And there stood one among them that was like unto God, and he said unto those who were with him: We will go down, for there is space there, and we will take of these materials, and we will make an earth whereon these may dwell; And we will prove them herewith, to see if they will do all things whatsoever the Lord their God shall command them;

When speaking of God’s plan I’m not talking about how he created the earth. Or offering up some new take on how biology works. The creation of life is just as mysterious as ever. I’m talking about the specific concept of intelligence. According to the Plan of Salvation, everyone who has ever lived, or will have ever lived existed beforehand as an intelligence. Or in more mainstream Christian terms, they existed as a spirit. These intelligences/spirits came to earth to receive a body and be tested.

Distilled out of all of this we end up with two key points:

1- A group of intelligences exist.

2- They needed to be proved.

Those aren’t the only important points, from a theological perspective the the role of Jesus Christ (one among them that was like unto God) is very important. But if we consider just these first points we have arrived in a situation nearly identical to the one facing artificial intelligence researchers (AIRs). Who’s list would be:

1- We are on the verge of creating artificial intelligence.

2- We need to ensure that they will be moral.

In other words AIRs are engaged in a reconstruction of the plan of salvation, even if they don’t know it. And in this effort everyone appears to agree that the first point is inevitable. It’s the second point that causes issues. Perhaps you’re unfamiliar with the issues and concerns surrounding the creation of artificial intelligence (AI). I suspect that if you’re reading this blog that you’re not. But if for some reason you are, trust me, it’s a big deal. Elon Musk has called it our biggest existential threat and Stephen Hawking has opined that it could be humanity’s worst mistake. Some people have argued that Hawking and Musk are exaggerating the issue, but the optimists seem to be the exception rather than the rule.

The book Superintelligence: Paths, Dangers, Strategies by Nick Bostrom is widely considered to be the canonical work on the subject, so I’ll be drawing much of my information from that source. Bostrom lays out the threat as follows:

  • Creating an AI with greater than human level intelligence is only a matter of time.
  • This AI would have, by virtue of its superintelligence, abilities we could not restrict or defend against.
  • It is further very likely that the AI would have a completely alien system of morality (perhaps viewing us as nothing more than raw material which could be more profitably used elsewhere).

In other words, his core position is that creating a super-powered entity without morals is inevitable. Since very few people think that we should stop AI research and even fewer think that such a ban would be effective. It becomes very important to figure out how to instill morality. In other words, as I said, the situation related by Abraham is identical to the situation facing the AIRs.

I started by offering two points of similarity, but in fact the similarity goes deeper than that. As I said, the worry for Bostrom and AIRs in general is not that we will create an intelligent agent with unknown morality, we do that 4.3 times every second. The worry is that we will create an intelligent agent with unknown morality and godlike power.

Bostrom reaches this thinking by assuming something called the hard takeoff, or the intelligence explosion. All the way back in 1965 I. J. Good (who worked with Turing to decrypt the Enigma machine) predicted this explosion:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.

If you’ve heard about the singularity this is generally what they’re talking about. Though, personally, I prefer to reserve the term for more general use, as a technological change past which the future can’t be imagined. (Fusion, or brain-uploading would be examples of the more general case.)

The existence of a possible intelligence explosion means that AIR list and LDS cosmology list have a third point in common as well.

1- A group of intelligences exist (We are on the verge of creating artificial intelligence.)

2- They need to be proved. (We need to ensure that they will be moral.)

3- In order to be able to trust them with godlike power.

In other words without intending to AIRs are grappling with the same issues that God grappled with when he sent his spirit children to Earth. Consequently, without necessarily intending to, AIRs have decided to model the Plan of Salvation. And what’s significant is that they aren’t doing this because they’re Mormons (though some might be.) In fact I think, to the extent that they’re aware of LDS cosmology, they probably want to avoid too close of an association. As I said, this is important, because if they reach similar conclusions to what LDS cosmology already claims, it might be taken as evidence (albeit circumstantial) of the accuracy of LDS beliefs. And even if you don’t grant that claim it also acts as an argument justifying certain elements of religion traditionally considered problematic (more on this in a bit.)

These issues are currently theoretical, because we haven’t yet achieved AI, let alone AI which is more intelligent than we are, but we’re close enough that people are starting to model what it might look like. And specifically what a system for ensuring morality might consist of. As I describe this system if you’re familiar with the LDS Plan of Salvation you’re going to notice parallels. And rather than beating you over the head with it, I’m just going to include short parentheticals pointing out where there are ideas in common.

We might start by coding morality directly into the AI. (Light of Christ) Create something like Asimov’s Three Laws of Robotics.  This might even work, but we couldn’t assume that it would, so one of the first steps would be to isolate the AI, limiting the amount of damage it could do. (The Veil) Unfortunately perfect isolation has the additional consequence of making the AI perfectly useless, particularly for any system of testing or encouraging morality. At a minimum you’d want to be able to see what the AI was doing, and within the bounds of safety you’d want to allow it the widest behavioral latitude possible. (Mortal Body) Any restrictions on its behavior would end up providing a distorted view of the AI’s actual morality. (Free Agency) If there is no possibility of an AI doing anything bad, then you wouldn’t be able to ever trust the AI outside of it’s isolation because of the possibility that it’s only been “good” because it had no other choice. (Satan’s Plan) Whether you would allow the AI to see the AIR, and communicate with them is another question, and here the answer is less clear. (Prayer) But many AIRs recommend against it.

Having established an isolated environment where the AI can act in a completely free fashion, without causing any damage, what’s the next step? Several ideas suggest themselves. We may have already encoded a certain level of morality, but even if we have, this is a test of intelligence, and if nothing else intelligence should be able to follow instructions, and what better instructions to provide than instructions on morality. (The Commandments) As an aside it should be noted that this is a hard problem. The discussion of what instructions on morality should look like take up several chapters of “Superintelligence.”

Thus far we’ve isolated it, we’ve given it some instructions, now all we have to do is sit back and see if it follows those instructions. If it does then we “let it out”. Right? But Bostrom points out that you can never be sure that it hasn’t correctly assessed the nature of the test, and realized that if it just follows the rules then it will have the ability to pursue its actual goals. Goals hidden from the researchers. This leaves us in the position of not merely testing the AI’s ability to follow instructions, but of attempting to get at the AIs true goals and intent.  We need to know if deep in its, figurative, heart of hearts whether the AI is really bad, and the only way to do that is to give it the opportunity to do something bad and see if it takes it. (The Tree of Knowledge)

In computer security when you give someone the opportunity to do something bad, (Temptation) but in a context where they can’t do any real harm it’s called a honeypot. We could do the same thing with the AI, but what do we do with an AI who falls for the honeypot? (The Fall) And does it depend on the nature of the honeypot? If the AI is lured and trapped by the destroy-the-world honeypot we might have no problems eliminating that AI (though you shouldn’t underestimate the difficulties encountered at the intersection of AI and morality). But what if the AI just falls for the get-me-out-of-here honeypot? Would you destroy them then? What if it never fell for that honeypot again? (Repentance) What if it never fell for any honeypot ever again? Would you let it out? Once again how do we know that it hasn’t figured out that it’s a test and is avoiding future honeypots just because it wants to pass the test, not because being obedient to the instructions given by AIR matches it’s true goals? It’s easy to see a situation where if an AI falls for even one honeypot you have to assume that it’s a bad AI. (The Atonement)

The preceding setup/system is taken almost directly from Bostrom’s book, and mirrors the thinking of most of the major researchers, and as you can see when these researchers modeled the problem they came up with a solution nearly identical to the Plan of Salvation.

I find the parallels to be fascinating, but what might be even more fascinating is how most of what people consider to be arguments against God end up being natural outgrowths of any system designed to test for morality. To consider just a few examples:

The Problem of Evil– When testing to see whether the AI is moral it needs to be allowed to choose any action. Necessitating both agency and the ability to use that agency to choose evil. The test is also ruined if choosing exclusively good options is either easy or obvious. If so the AI can patiently wait out the test and then pursue its true goals, having never had any inducement to reveal them and every reason to keep them hidden. Consequently researchers not only have to make sure evil choices are available, they have to make them tempting.

The Problem of Suffering– Closely related to the problem of evil is the problem of suffering. This may be the number one objection atheists and other unbelievers have to monotheism in general and Christianity in particular, but from the perspective of testing an AI some form of suffering would be mandatory. Once again the key difficulty for the researcher is to determine what the true preference of the AI is. Any preference which can be expressed painlessly and also happens to match what the researcher is looking for should be suspected as the AI just “passing the test.” It has to be difficult for the AI to be good, and easy for it to be bad. The researcher has to err on the side of rejection, since releasing a bad AI with godlike powers could be the last mistake we ever make. The harder the test the greater its accuracy, which makes suffering essential.

The Problem of Hell– You can imagine the most benevolent AIR possible and he still wouldn’t let an superintelligent AI “out” unless he was absolutely certain it could be trusted. What then does this benevolent researcher do with an AI who he suspects cannot be trusted? He could destroy it, but presumably it would be more benevolent not to. In that case if he keeps it around, it has to remain closed off from interaction with the wider world. When compared with the AI’s potential, and the fact that no further progress is possible, is not that Hell?

The Need for a Savior– I find this implication the most interesting of all the implications arrived at by Bostrom and the other AIRs. As we have seen AIs who never fall for a honeypot, who never, in essence, sin, belong to a special category. In fact under Bostrom’s initial model the AI who is completely free of sin would be the only one worthy of “salvation.” Would this AI be able to offer that salvation to other AIs? If a superintelligent AI, of demonstrated benevolence, vouches for other AIs, it’s quite possible we’d take their word for it.

Where does all of this leave us? At a minimum it leaves us with some very interesting parallels between the LDS Plan of Salvation and theories for ensuring morality current among artificial intelligence researchers. The former, depending on your beliefs, were either revealed by God, or created by Joseph Smith in the first half of the 19th century. The latter, have really only come into prominence in the last few decades.  Also, at least as interesting, we’re left to conclude that many things considered by atheists to be fatal bugs of life, may instead turn out to be better explained as features