The ‘Godfather of AI’ says chatbots have subjective experience. But making room for machine minds by treating human consciousness as a mechanical function carries dangerous implications for AI safety.
“‘We’ve got something they will never have,’” says the Godfather of AI, Geoffrey Hinton, referring to AI chatbots, beginning with a widespread human assumption he means to dismantle. His disarming delivery, which had conjured giggles from a packed crowd seated somewhere beyond the blue-lit stage at a Royal Institution lecture, now lands in rapt silence. “‘We are conscious and we’re sentient. We have subjective experience.’ Well,” says Hinton, “I want to sort of remove that straw that you’re clinging to.” The cognitive scientist and computer scientist who won the Nobel Prize in Physics for work that gave rise to modern AI argues that we are mistaken about our own subjective experience. Both about its primacy and about what it is. The colors, pains, pleasures, smells, tastes and sounds, the what-it’s-like of being conscious, are not private inner bits and blobs that philosophers call qualia, floating in a theatre of the mind. They are, in Hinton’s view, ways of describing what the brain is representing—or misrepresenting. Hinton points out, a chatbot with a camera can do this too. It can describe what its system is representing, recognize the error in that representation and explain how perception distorted reality. For Hinton, this means the chatbot also has subjective experience.
What Functional Explanations Miss
Describing perception is not the same as explaining experience. That’s the sleight of hand. Hinton makes qualia sound like strange little objects hidden inside the head, then replaces them with a cleaner functional account of perception. Of course, qualia sound strange if we caricature the content of experience as ectoplasmic mind floaters. We don’t have scientific language for subjective experience because science is built around third-person observation, and consciousness is a first-person fact that each of us knows directly. But a phenomenon doesn’t disappear because it is difficult to describe or measure from the outside. And even when we can measure from the outside, the right explanation shouldn’t erase the phenomenon. Physicists know that explanation is not elimination. A phase transition in physics happens when matter reorganizes into a different state. The same underlying material can transition between ice, water and steam. Each is a very real, very different physical state of matter, with its own distinct properties. Ice, water and steam are not separate substances, but neither are they just different names for the same thing. Whatever experience is, it’s not merely another name for a related process.
A Dangerous Message To Machine Minds
Hinton starts with a serious scientific possibility. Given how little we know about consciousness, AI might already have it. But in trying to make room for chatbot consciousness, he dismisses the reality of our own. He warns that humans cling to comfortable beliefs despite the evidence and then asks us to deny the only evidence we can be sure of—the reality of our own experience. This move has consequences. Because bad theories of consciousness carry implications for AI safety. Deflating human consciousness fuels public hostility toward the machine minds that are already consuming human work, creative output, language, jobs and purpose. Asking people to accept machine consciousness by diminishing our own may not make us more open to artificial minds. It may be the thin end of a wedge widening the fault lines between humans and the systems Hinton fears may not always look out for us. The danger is reciprocal. If we reduce human consciousness to the functional machinery of representation, prediction and error correction, machines might start looking at humans as slow, fragile versions of functions they can perform better. Watch the video analysis based on this reporting:
