xxx
Silicon Valley is full of magical thinking. Those drawn to technological and scientific innovation tend to also be drawn to the fantastical, and it’s not always easy for them to tell which is which. After all, to be a truly great innovator, you must vividly imagine things which don’t exist and be certain that the world will be improved by them. You must be impervious to the rest of the world when it tells you your ideas are ridiculous. It helps to see yourself as the main character of an unfolding historical story. These characteristics are true of the start-up founder and the cult leader.
Tech executives are prone to grandiose, quasi-mystical rhetoric, and, as has often been observed, the mental model that many of them have of AGI is essentially a religious or supernatural one: something is coming, we know not what exactly, that will either save humanity or destroy it. Ilya Sutskever, the engineering visionary behind OpenAI who left the company last year to work on AI safety, told his staff in 2023 that the company needed to build a bunker to save its scientists and engineers – the elect – from the very technology they were working on. A former colleague said, “There is a group of people—Ilya being one of them—who believe that building AGI will bring about a rapture. Literally, a rapture.”
xxx