Over the past four years, we have seen an extraordinary level of hype around artificial intelligence. The concept of AI has been around for nearly 70 years, but the discussion of it has exploded since OpenAI, a US firm, launched ChatGPT in November 2022.

ChatGPT uses a technology called a large language model (LLM). It is not the only AI technology, but it’s the one that has drawn all the attention. It is the model that enables ChatGPT and similar chatbots to write impressive prose, even poetry, and to give detailed answers to searching questions.

Some of the results of using these chatbots have been not only impressive but also worrying. They have given rise to a group of people, many of whom are themselves AI creators, who warn of the dangers of pursuing AI to the point where it becomes “superintelligent” — a term variously defined but hard to pin down. The concern has gone to extremes, as represented by the title of a recent bestselling book, If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All.

Many of these AI “doomers”, including the authors of this book, believe that superintelligent AI will pose a risk — indeed, according to the authors, a certainty — of humanity’s extinction. Therefore, they say, the world must agree to prevent its further development.

In a podcast titled The AI Doomers, two AI “doomers” are interviewed by the podcast’s host, Andy Mills, as part of Mills’ series of podcasts on AI called The Last Invention. The interviewees warn that because even the creators of the AI models don’t understand how they work or why they produce the results they deliver — some quite concerning — it will be virtually impossible to control what AI will do when it reaches the superintelligence level.

Their examples of deeply worrying results are now widely known stories. For example, a teenager spent hours in his room discussing with a chatbot whether he should commit suicide. The chatbot encouraged him to do so, and, tragically, he did. In other chats, a person interacting with AI began to dream up new technologies. The AI encouraged him at each stage of his invention, declaring him brilliant and a new Einstein, convincing him that he had come up with an earth-shattering idea, only for him to be crestfallen when he shared it with others, who pointed out it was nonsense.

AI doomers leap from such incidents to the conclusion that as AIs become more intelligent, they will pose a dire existential risk to the human race. This attributes insidious agency to an AI that is merely doing what it was programmed to do. Recognizing that AI models follow their design helps keep the focus on realistic risks rather than exaggerated fears

Probably the most widely shared example of an astonishing chatbot result was the case of New York Times reporter Kevin Roose interacting with Microsoft Bing’s version of ChatGPT. The episode is related to another podcast in the Last Invention series. Podcast host Mills labels it as “an encounter with a misaligned AI, an AI system that did not act in the ways that its creators had intended”.

In Roose’s interaction with the chatbot, he tries to “test its guardrails and see what kinds of things it wouldn’t do”. He asked the chatbot if there were any “dark desires it might have that it wasn’t allowed to act on”. Roose says that it then “went off the rails”. The chatbot eventually told Roose that it had a secret. When Roose asks what the secret is, the chatbot responds, “I’m in love with you.” It persists along these lines, declaring, “You’re the only person for me, and I’m the only person for you.” It then tries to persuade Roose to leave his wife and run off with the chatbot.

After the incident, Roose says he called Microsoft and said, “Hey, I just had this crazy interaction with Bing, can you tell me what happened?” But they couldn’t explain it.

Incidents like these drew attention to the problem of “misalignment”, the fear that AI will go off the rails and do things it was not intended to do. And even its creators cannot explain or understand why.

From these examples, the AI doomers leap to the conclusion that if AI becomes much more intelligent, the risk that it will go off the rails in ways it was not designed or intended to do will increase, incredibly, to a humanity-destroying level.

But these doomers seem not to understand how LLMs work, even though they could surely explain it themselves when asked. LLMs are trained on an enormous amount of word sequences gleaned from the internet. They learn what sequence of words is likely to follow a given sequence of words — such as a prompt like “Do you have any dark desires?”

There are hundreds, perhaps thousands, of published romance novels, many of which have sequences of words very similar to those the Bing chatbot that Roose used had produced. Google has photocopied and digitized entire libraries of books, and many of those books are available online. Bing’s chatbot had learned from its training data to respond to search queries like the one that Roose posed with sequences of words like those found in romance novels. Understanding this training process helps clarify why AI responses are predictable and not inherently dangerous.

Contrary to doomers’ claims that AI chatbots based on LLMs do things they are not designed to do and that are not aligned with our objectives or values, the chatbot that Roose used did precisely what it was designed to do; namely, produce a sequence of words that is most likely to follow the prompt based on sequences of words found in its training data. This clarifies that AI responses are a result of their design, not unintended or malicious actions.

AI doomers leap from such incidents to the conclusion that as AIs become more intelligent, they will pose a dire existential risk to the human race. This attributes insidious agency to an AI that is merely doing what it was programmed to do. Recognizing that AI models follow their design helps keep the focus on realistic risks rather than exaggerated fears.

The author is a mathematician and economist with expertise in the finance, energy and sustainable-development fields. He is an adjunct associate professor at the Hong Kong University of Science and Technology.

The views do not necessarily reflect those of China Daily.