I, Large Language Model: Could Large Language Models Really Be Conscious?

The city I live in has many green-billed toucans. Besides their stunning looks, toucans also produce a conspicuous cry. One day, after hearing one such cry, I tried to figure out how I would describe it to someone who never heard it. Here is what I came up with: Imagine you have a piece of a bamboo trunk cut into two halves. The borders of the two halves have been covered with small indentations, creating a finely toothed ridge. Now imagine you slide one of those ridges against the other, very fast. The resulting sound would, I believe, resemble the cry of toucans. I say “I believe” because, naturally, I have never actually carried out this experiment. Which raises the question: how do I (think I) know the sound of something I have never experienced? And, more importantly, what does this have to do with the topic of this essay—namely, whether large language models are conscious? Bear with me.

Are Large language Models Conscious? It Depends on Who You Ask

Since large language models (LLMs) were introduced to the public in 2023, claims about such models being conscious have proliferated. By “conscious,” what is usually meant is the same as having a subjective experience—that it feels like something to be an LLM. In one study, more than half of participants who used LLMs attributed a non-zero chance of them being conscious. And such beliefs can also come from people with knowledge of how these models work. Consider the case of Blake Lemoine, an engineer who worked developing LLMs at Google, who was fired back in 2022 after a leaked memo where he questioned whether the company’s model was conscious. Or the case of Nobel Prize-winner Geoffrey Hinton, a towering figure in artificial intelligence (AI), who claimed in interviews that current LLMs were already conscious.

AI companies have also exploited the issue for marketing purposes, nowhere as starkly as in the rhetoric used by Anthropic, one of the leading companies in the field. Anthropic went as far as to establish a team dedicated to “AI”—even granting its stronger models the capacity to end certain types of conversations to protect its own welfare. But beyond people’s impressions, and companies’ marketing grift, how can we tell if LLMs are indeed conscious?

The Contentious Requirements for Consciousness

Opinions about the possibility of consciousness in LLMs are contingent on the philosophical position one starts from. And while researchers arguing for some degree of consciousness in these models are still a minority, the subject is so muddled that the mere theoretical possibility of creating a conscious AI is still debated. Accepting such possibility usually implies subscribing to a philosophical position called computational functionalism: the idea that the mind is produced by specific computations, which in our case are implemented by the brain, but which could be instantiated in different substrates. This view has had important challenges and caveats raised against it (as in recent papers by Rosa Cao, Jaan Aru, Ned Block, and Anil Seth), but it remains a commonly held default position among AI researchers and computationally inclined neuroscientists. Instead of arguing against this assumption, let us assume it to be true for a moment: where does that leave us?

If computational functionalism is true, then any system capable of implementing the right computations would be conscious. So, to find out if LLMs are conscious, all we need is to compare the computations they perform to the ones that underlie our own consciousness. But since we do not know what computations underlie our consciousness, how can we proceed? One alternative is to take the Turing approach: treat both the model and the human as black boxes and compare their output. If the model is capable of matching our output, we could argue that it may perform the same computations we do. LLMs certainly seem capable of generating human-like written output, convincing enough to pass a standard Turing test.

However, this kind of superficial similarity argument quickly falls into trouble when closely inspected, as illustrated by the philosopher John Searle’s classic thought experiment, the Chinese room. In it, Searle imagines himself locked inside a room as someone on the outside gives him successive batches of Chinese writing, which he cannot understand. He also receives a document with precise instructions, in English (which he understands), to formally manipulate the Chinese symbols in order to produce a new batch of symbols, which he then sends to the person outside the room. That person can actually understand Chinese, and her reading of the text sent to Searle reveals questions to which Searle’s outputs are actually meaningful answers—even though Searle can’t understand any of it. The lesson here is that mere symbol manipulation (which we could argue describes all LLMs can do) is not enough for true understanding and, by extension, it is not a sufficient criterion to establish consciousness.

Searle’s argument was extended in 1990 paper by cognitive scientist Steven Harnad. In it, Harnad articulated the challenge that those building language models would later face, which he called the symbol grounding problem. To understand it, suppose you are trying to learn Chinese as a second language, but all you have to go on is a Chinese/Chinese dictionary. In such situation, as Harnad notes, each “trip through the dictionary would amount to a merry-go-round, passing endlessly from one meaningless symbol or symbol-string (the definiens) to another (the definiendum), never coming to a halt on what anything meant.” And if this seems hard enough, the real challenge in building a model that understands language is actually harder, because there is no first language to begin with! In Harnad’s words, “[h]ow is symbol meaning to be grounded in something other than just more meaningless symbols?”

But as neat as these arguments sound, they are confronted with the blunt reality of current LLMs’ impressive capabilities. These models have apparently overcome the symbol grounding problem, building a semantic space from all the abstract statistical regularities in their massive training datasets, and leveraging it to create, piece by piece, what appears to be meaningful output. It is debatable whether these models really manipulate semantic information—all an LLM does is to processes numerical inputs according to its internal structure, essentially computing a ridiculously complex conditional probability function for predicting next tokens (words or fragments of words) in a string of text. That we can label its internal “representations” of information with concepts that are meaningful to us is simply a byproduct of the complex statistical patterns in these models’ training data, which are reflections of the way we ourselves use words (see here for an overview of these models’ workings). All we label as “semantic” in a LLM is purely relational—meaningless symbols grounded on more meaningless symbols. Yet, one could argue that these models’ internal structure “embodies” relationships between concepts—that in the process of imitating human writing patterns by statistical trickery, it learned something about the semantics of our grounded concepts. But is this enough for consciousness?

Back to the Basics

Perhaps a better way to look at this problem is to start from the fundamental features of conscious experience. Conscious experience involves qualitative perceptions and feelings—it has content. And, in our case, content is built from sensorimotor building blocks, those fundamental bits of subjective experience that stem from our interactions with the world though our bodies, and which constitute the simplest form of phenomenology we can think of. These are the anchors that hold our concepts to the ground. While LLMs often seem capable of manipulating semantic content instead of merely applying syntactic rules, their “semantic representations” are starkly different from our own in that they are missing these very anchors. To illustrate this, consider a simple thought experiment.

Imagine a person born without the senses of taste and smell. Now imagine that, fascinated by this inaccessible world of flavors and aromas, she decides to dedicate herself to the study of gastronomy from a young age. She reads everything there is to be read about different foodstuff, cooking techniques, seasoning, etc. Among this material are extensive descriptions of smells and tastes of different combinations of spices and vegetables and proteins and so on. After years of study, she becomes so well-versed in these descriptions as to apprehend an inner logic in the characteristics of different combinations of elements used to create dishes. She can provide detailed descriptions of the smell and taste of any conceivable dish. She can even do it for dishes that have never been prepared! Her descriptions are so good they can match the perception of people who can actually taste the dishes. And now I ask you: what does this person experience when describing the taste of a dish? What is her subjective experience of all the gustatory and olfactory adjectives she uses? Is it conceivable that, being born without taste and smell, she actually perceives anything at all attached to these descriptions? Or is it more likely that, when writing her descriptions, she has the same experience attached to them as you would have after receiving a list of made-up adjectives linked to different made-up foods and being tasked to describe a combination of these imaginary ingredients?

Crucially, though we can safely assume that this person is conscious—there is no reason to assume otherwise from the loss of smell and taste alone—it is hard to believe she experiences anything substantial when producing descriptions of flavors she never experimented. And if she does experience something, these experiences would likely be grounded in analogies to concepts taken from her other senses, which are themselves grounded. Since her knowledge of tastes and smells wasn’t built from the ground up—from core experiences acquired through our interactions with the world—it cannot possibly be as rich in information as normal perception is.

What Kind of Consciousness Could LLMs Possibly Have?

For all our imaginative capacities, nothing we can experience in our minds seems to be groundless. This is true even for concepts that have no real correlates in our experience—like imagining what an infinite-dimensional space is like. No matter how hard we try, the only subjective experience we can possibly have of such space is grounded in our familiar three-dimensional one.

Even when we mentally simulate a scenario that was never experienced—like trying to replicate the sound of toucans with an imaginary bamboo instrument—all of our predictions are inferred from grounded past experiences. To answer the first question posed at the beginning of this essay, I think I know what it is like to play that bamboo instrument not because I have abstracted a set of statistical patterns among different symbols, but because I have built some form of internal representation of different elements of the world and what their behaviors look, sound, and feel like. And I did this by feeling and manipulating the world through my body.

What all of this implies is that, as much as LLMs seem to process semantic information and not just syntactic rules, they are still stuck in the Chinese room. Even if we assume that they mimic not only our use of language but also the semantic representations we ourselves rely on when using language, this is likely still not enough to have a subjective experience with meaningful content. And bear in mind that all of our discussion so far has focused on a rather dry form of subjective experience—one that lacks emotional feelings, positive or negative, which prominent theories have tied to core functions of physiological regulation and homeostasis—something fundamentally biological as far as we know.

LLMs have shown us that it is possible to produce written outputs full of apparent meaning without grounding—that while it may not be possible to learn Chinese from a Chinese/Chinese dictionary, it is certainly possible to make it look like you did. But for all their language tricks, LLMs haven’t really escaped the symbol grounding problem. And until they overcome this problem, even if we accept all the required assumptions for believing in the possibility of conscious AI, any possible subjective experience in an LLM is bound to be so impoverished as to be fundamentally empty—a lifeless inner life.

 

Acknowledgments: The author would like to thank Sabine Pompeia, Maria Luiza Iennaco de Vasconcelos, José Maria Monserrat, and Rafael Kremer, who read earlier versions of this essay and provided helpful suggestions.