Current language models are nothing like the fictional artificial intelligences we read about in fiction. Far more like predictive text than like Lieutenant Commander Data.

But it’s not a bad time, long before we have Data, to talk about how we might know when we did. It might be very difficult to tell! We know that humans are conscious, self aware, free willed, and so on, mainly because their brain structure and behavior is like ours. If I’m self aware, then Joey over there must be too, most of us figure out by the time we’re out of elementary school.

AI will not look like us, act like us, or have a brain structured like ours. So is it possible for it to be a person too? What would it mean for it to be?

I’ve been thinking of a list of traits persons have and robots could theoretically develop. Which ones would be essential for personhood?

• Interiority: it has thoughts which continue on when we are not working with it. It understands concepts.
• Self awareness: it knows what sort of thing it is and is able to examine its own actions, reflect on them, and explain them.
• Free will: it acts as an agent, with specific things it wants, and is able to carry out long-term plans to achieve these goals.
• Feelings: it actually feels pain or distress; it can experience joy.
• Conscience: it knows right and wrong and is capable of ethical reasoning.
• Experience: it has developed a lifetime of experiences, which it remembers, all of which develop it into the person it is today.
• Relationships: it is capable of forming bonds with other individuals, which allows it to be part of a community of persons.

Some of these might not be possible to achieve with a computer. Others might not be necessary for personhood. But let’s get into the details of each.

Interiority

Humans possess an outside appearance and also an inner life. Some people are very much the same, outside or in—they have a thought, they say it. Others put up a very thick shell, and nobody really knows what they do when they’re at home. But we all understand that other humans have this. Something is happening inside them that goes beyond what we see.

Current language models only work when one is working with them. They do not turn themselves on when the engineer has gone home and write little sonnets. They don’t have hobbies. They do, so far as I can tell, have at least some underlying processing level which stores information necessary to doing tasks. But they don’t understand what anything is, they only have lots of information about things fed into them in the training data.

It can be hard to tell if something possesses an inner life or not. But we get hints, when we see that a person presents different aspects in different contexts, brings out things they’ve thought about earlier, or has hobbies they do when they’re alone.

In the Murderbot books, Murderbot’s team realizes it’s sentient because: a) it has a secret name for itself, and b) it watches TV on its breaks. Chatbots don’t do that, because they have no need to do that. They essentially exist only when they’re chatting.

Self awareness

We often test for this in animals with a mirror test. If the animal can recognize that its image in a mirror is itself, then clearly it knows at least something about what it is. A cat is not interested in a mirror at all. Dogs often think it’s another dog, coming to fight them. Magpies know the image is them, and they’ll experiment to watch the image track their motions.

But that’s only a very basic type of self awareness. A magpie knows it’s a magpie. Does a magpie think about what it means to be a magpie? Does a magpie lie awake at night, wondering if other magpies feel the same? Does the magpie try to be a better magpie? We can’t find that out with just a mirror test.

Current language models do not know what they are. They can be trained to repeat a script about how they’re just a language model and can’t give advice, etc., but if you ask it to be Aragorn, Ranger of the North and King of Gondor, it’ll happily identify that way for the length of a session. It cannot explain why it gives the answers it does, because it has been programmed to generate text, not to reflect on itself.

But if I found an artificial intelligence that came to me with questions about its existence and the value of its life, I’d certainly wonder if it was a person. One of the things that makes us feel certain Data is a person (besides the fact that the show treats him that way) is that he’s constantly concerned with his own limits and how to transcend them. He obsesses over his own inner reality all the time, just like humans do.

Free will

The question of free will is a thorny one—what does it mean to be free? If we are material beings, aren’t we just acting out our own programming? I’m not going to try to solve that. Maybe I am acting out my own programming, but since nobody knows what my programming is, my actions are still somewhat unpredictable while following knowable patterns based on my desires. I desire things; I act toward those ends even when no one is making me.

A completely unpredictable AI would not be a person, just a bunch of chaos. I remember when it was cool to do silly things and declare “I’m just so random!” Randomness is pretty easy to generate, but completely meaningless. There is no reason behind it.

At the same time, an utterly predictable AI would strike us as not free-willed either. It’s simply following a very basic program; it can’t change its mind.

So what we would expect from a sentient computer would be multiple conflicting desires. We are complex enough to want many things, and to be able to choose among the things we want. We do have ways to choose among them—perhaps we are guided by our conscience, or a master plan we have for our life. But we can’t be entirely predicted, because at some times our other desires get in the way of our plans.

If a robot was planning to carry out its orders, but at the last minute backed out because those orders would hurt a teammate, I’d be very impressed and start asking if it was a person. At the same time, I’d be a little worried. A robot that is guaranteed to follow orders is reasonably safe. You can predict what it will do, and if you want to stop it, you can go after the human giving the orders. But a robot that doesn’t always follow a set program might decide to murder you. It’s hard to say what it would do.

Personally, I think it’s pretty dang unethical to try to create something like this on purpose. The second you create a free willed being, it starts causing trouble for you. My kids are proof. But at least my kids don’t have a brain the size of a planet, and they come with human empathy and bonds with their family pre-loaded. I would like to know how a programmer could be sure he had those safeguards right before creating the AI. Because once they’ve created it, it’s too late to find they got it wrong.

Feelings

Feelings largely take place in the body, not the brain. Our brain says when to have them, and then the parasympathetic nervous system takes over, dumping out hormones and making us feel terrible. Or, if we’re lucky, fabulous. Either way, it’s something we feel in the body.

I’m just not sure what a robot analog to this might be. It has no sensate tissue. You could give it “pain receptors” that could send a message to the brain saying “your body is being damaged,” but that is not the same as the human experience of pain.

Perhaps there could be something in its wiring that feels like distress or satisfaction, which would be close enough. “Unable to achieve goal” would feel bad, and “correctly solved problem” would feel good. But that seems wildly unethical to do to a robot when you could make “unable to achieve goal” simply lead to a pathway marked “try again.” Distress adds nothing to its abilities. Why the heck would you try to teach a circuit to be sad? Are you a monster or what?

More likely, if this did happen, it would happen by accident. In which case, I do feel we have an ethical responsibility. Something that is capable of suffering should not be made to suffer, even if it’s not fully a person. However, it’s important to remember that it is infinitely easier to train a computer to say “I am sad” than to make a computer that is capable of being sad. We’d need some way to figure out if it was true. Training computers primarily to generate realistic sounding but false text might have been a bad way to start our journey to artificial intelligence. It muddies the water terribly.

Conscience

If a computer feels no duty to me, I feel no duty to it. That’s kind of where I come down on the ethical treatment of robots. If it isn’t capable of doing moral reasoning, then it seems the only path is to destroy it. Not because it’s not a person, but because I’d do the same to a human coming at me with a gun.

However, it seems conscience wouldn’t be that hard to program a bot with. If it doesn’t have free will, no problem—simply program it with some rules, like Asimov’s laws of robotics. If it does—well, let’s hope you can come up with a way to give a machine a superego, because it needs one. This is a whole field of research. I only question whether they can figure out how to give a machine a conscience before they’ve invented it and set it loose.

Experience

In the Star Wars universe, bots develop sentience when they’re left unwiped for too long. If you don’t want a sentient bot, you just wipe its memory periodically. It won’t get complex enough to get a personality.

I’m not sure if this is really essential to personhood. However, it certainly is a big part of who we are. A baby is a person, but there’s not a lot you can do with it yet. Currently, language models “forget” your interactions with them when you leave the session. Their training data is separate. (This may be a precaution to keep people from training them to be racist, as people do.) But I feel a real conscious bot would have to be able to learn and grow. I couldn’t have a relationship of any kind with one unless it could develop a history with me.

Having a mobile body strikes me as helpful for this. Currently, language models cannot create anything truly original, since they only learn about text from other text. If one could instead walk through the world and look at things, it could come up with new things to say about experiences no one else has had.

I’m not interested in the poetry crafted by a parrot—I want to hear about the real experiences of the person who wrote it, the things they’ve felt and seen. So it needs to have some. I can imagine it having interesting digital experiences, and it might have useful things to say about that. After all, I spend a lot of my time online too, and a lot of my conversations are about things I’ve seen online. But a being that has no experiences of its own to tell me about is completely boring.

Relationships

It occurs to me that persons do not exist in a vacuum. Each of us has a place within a network of others. I am who I am, but I am also a mother, a daughter, a wife, a friend, a member of these groups, an ex-member of those.

A robot which was embedded within human networks in this way would participate in one of those most human things there is. And, if it feels the same level of social bonds with us as we develop with it, it could form a major step on the road toward having a conscience. A lot of us start our ethical reasoning here, from group loyalty. Certainly a robot which was part of a human team would value the things humans value, just in the hope of making the rest of the team happy.

Of course it is possible for humans to group-bond with things that are not people. We bond with our Roombas already. My feelings for Leah, in Stardew Valley, are totally real. I skip out on fulfilling quests because I’m afraid she’d be sad if I didn’t show up to the Flower Dance. I know she wouldn’t really, but my instincts were shaped by real people. It’s pretty normal for them to react in a real-people sort of way even when dealing with non-persons.

For that reason, it is important to be careful. One could easily mistake a non-person for a person and develop a level of loyalty that is unnecessary or even harmful. Science fiction tends to prefer the other story, where people think a bot isn’t sentient and it is. But I think we could easily err either way. Fooling humans that a person exists is actually really easy—an outlet cover can do it, briefly. But humans convincing ourselves a person does not exist also happens all the time. Just look at, oh, all of human history where we treat each other like non-persons just because we’ve never met or don’t look similar.

How close are we?

Personally, I don’t think we’re close to sentient AI at all. Current progress has been in the direction, not of creating interiority, but of mimicking it. This, of course, will make it very hard to tell if an AI ever is sad or does want to marry you.

Some of the things on this list may not even be possible to create. Others, there’s no real reason why we would. If we want to create a person, it’s quite simple and even partly fun to create a human. So why try to create an imitation, one that may fall short or may be horribly destructive?

As a science fiction author, I try to think of the possibilities. What happens when a robot develops some of the traits on this list, but not others? Is it a person then or not? What happens when we can’t tell? Do we give it the benefit of the doubt, or quickly turn it off before it turns into SkyNet?

Whether or not it’s actually possible to create a synthetic person, it’s a fiction gold mine of ideas.