← Screen Deep
Screen Deep
Are You Real? Ethics of AI Chatbots and Characters for Children with Sonia Tiwari, PhD
00:51:12
Loading summary
Transcript38 lines
- [00:00]
A
Foreign.
- [00:06]
B
Welcome to the Screen Deep podcast, where we go on deep dives with experts in the field to decode young brains and behavior in a digital world. I'm Chris Perry, Executive Director of Children and Screens. Today we're thrilled to be joined by someone who sits at the intersection of character design, learning science, and the rapidly evolving world of AI. Dr. Sonia Tavari is a research fellow with the Falling Walls Foundation, a children's media researcher, and an AI design expert whose work bridges industry and academia in a way few others do. Sonja began her career as a character designer in the gaming world before earning her PhD in learning, design and Technology from Penn State University. Since then, she's become one of the most insightful voices on how AI powered characters are shaping children's learning, imagination and emotional experiences. I'm delighted to talk with her about how character development in children's media is changing, what's at stake, and what families need to know about how industry is combining characters and AI to become a bigger part of children's lives. Sonia, welcome to Screen Deep.
- [01:16]
A
Happy to be here.
- [01:19]
B
So can you start us off by defining what an AI character is as opposed to an AI companion or other AI technology?
- [01:29]
A
Right. So it stems from my background as a character designer. In general, a character is part of a fictional narrative. It visually or through sound represents a personality. And AI characters are just it has some AI functionality added to this fictional personality. So I consider AI characters as the bigger umbrella. A character can facilitate an entertainment experience, educational experience, it could provide just information. And so companionship is like one of the functions a character can serve. And not all characters are designed for companionship.
- [02:11]
B
Really interesting. So I've been thinking about all the characters that children have been exposed to for decades in media, but I'm wondering a little bit about how long AI characters have been available to children. And I I wonder if this recent explosion in AI products and capabilities have accelerated the number and availability of AI characters that children are experiencing.
- [02:35]
A
So AI as such has been around for a while. Generative AI boom really started around 2021, and then when companies like OpenAI and Anthropic and Google, they started offering up their products for others to create a wrapper around their product and build their own things. That's when a lot of these smart toy companies and tutor companies started using this already amazing available LLM to design their own products. Usually in children's media. It's called like a thick wrapper around the LLM, which is like adding as many guardrails as one can to sort of make this thing that was not originally designed for children function like a child friendly product. And that's where the problem is, because it's easy to get around. And building a custom LLM takes time and also data collection, especially in the US with FERPA and COPPA regulations. On one hand, it's amazing that the laws prevent companies from collecting too much data at the same time, to create a child friendly LLM from scratch, you do need a lot of data. So there are a few companies like Buddy AI who started off in like Latin America where the laws were different. They were able to collect a lot of data, able to train their custom LLM. And so they are better at recognizing children's voices with accents because it started off as a tutoring company for helping children learn to speak English who are non native speakers. But now that they, you know, they have figured it out, their technology, they have expanded into like all sorts of learning experiences.
- [04:18]
B
What are the primary venues? Not just characters, but venues where children are encountering AI characters today. And are they mostly deployed through educational and learning technologies or is it mostly entertainment?
- [04:33]
A
Right, so I mean actively, when companies advertise as something designed for children, they tend to use sometimes even superficially from a marketing point of view that, you know, this is educational and like 70% parents believe this is amazing. They never reveal like 70% of what or how did you find out that this is effective? Effective how? So those questions, like the journalistic integrity is kind of missing in those marketing materials. So it's hard to tell whether the products that are advertised for kids are actually just directly designed for kids or it's just a thin wrapper around an existing LLM repurposed and masked as something kid friendly. So these days I think the most common interface for kids is through toys because one of the user experience design principles is that if you want to introduce something new, you layer it on top of something that's familiar. So toys are familiar to kids or for younger kids who are not literate yet, voice interaction is easier and natural. So smart toys became an easy way to kind of introduce AI to kids. I think smart speakers are the most common venue for kids to interact with a form of AI, but it's more transactional. So it's not the problematic area that, you know, we care about that, you know, we need to fix this. So smart speakers per se are not the problem. It's the AI toys or companion chatbots that can get addictive, induce some kind of self harm behaviors, and then also because the regulation is not Very tight around these things. Then there are so many products that are aimed at adults or at least 13 plus, but younger kids are able to access it. So we can't control everyone's home environment. We assume that, oh, every child has a caring adult at home who is watching over them and making sure they have the space to express themselves. Usually that's not the case. A lot of kids have dysfunctional families. No one is looking out for them. They have nowhere to go or no one to talk to. And so then AI feels like an easy way out.
- [06:44]
B
I'm glad you brought up the smart speaker and Alexa and Siri and these characters that have been embedded in smart speakers for more than a decade and that kids are interacting with them by issuing a verbal command. Very different from holding a fuzzy toy and talking to it and having a character be in your hands. But before we move on, I wanted to back up. I know We've talked about LLMs a few times on the podcast, but I thought it would be helpful before we get too much deeper into the wrapper of AI characters and LLMs for you to briefly define what a large language model is for our listeners.
- [07:22]
A
Yeah, so large language model is kind of built on. It's the simplest way I explain it to kids is like the aggregate of the public information that's available out there and modeling it after the way we have seen communication happen in the past. But again, our language is stemmed in our lived experience. And there's no like lived experience here. This is the aggregate of everyone's documentation of their lived experience, not their actual lived experience. So that is why it's different than the language we use to communicate with each other. It's a set of protocols that tells an algorithm how to behave, how to retrieve information, how to communicate based on the patterns of communication it has observed across the publicly available information.
- [08:17]
B
And when you bring up something like documentation and what's on the Internet and easily retrievable by an LLM was probably created by an adult, a specific kind of adult for a specific audience. And what we're talking about today are children and their interactions with AI characters. So what is different about AI characters and how children experience them as opposed to traditional media characters?
- [08:43]
A
Yeah. So first I'll start by saying that, you know, developmental alignment in characters that were in books or TV shows or movies, it was pre designed. Like I come from a film and animation background. And pre production is a whole field in itself. We spend like hours, you know, thinking of variations of a child. Like if it's A five year old versus a six year old, like the head shape, the mannerisms, all of that, everything, every detail is designed and well thought out. And so if there is a character in a picture book and if it's like a traditional publisher, usually they have like the step into reading program where, you know, level one is ready to read and then all the way goes to level five, which is ready for book chapters. And then the chapters have their own categories. So there's like a very clear marker from the developers themselves that who is it for? What are, you know, there's a blurb at the back that you can read and assess whether is this, this book is interesting for you. You can flip through the whole experience in a minute, go to a bookstore and then assess. Whereas in AI characters there's no like flipping of the book or reading some. Again, like I said, the marketing materials equivalent of the literature on the back of a picture book is not the same because publishers are particular about, you know, what they will allow being written. No one can just like make up facts and put it on the back of a traditionally published book with a recognized publisher. Whereas, you know, with AI characters, if you're producing like a tutor or an AI toy, sure there are laws and regulations, but like the description on Amazon or the reviews on some influencer's blog that can say whatever they want about these characters, whether it's like a smart toy or like a tutor or just like a companion chatbot. And then it falls on like the values of the CEO and the design team almost whether it will appeal to them or they could be classic, like you know, Mr. Burns type of entrepreneur who ends every call with like God bless America and release the hounds. So you can't appeal to these kind of like CEOs that, oh, we would really appreciate if you take into account the developmental alignment. So that's the one thing. The other is like boundary definition. Like if you flip through a book, there's a definite beginning, middle and end. If you see a character in a board game, the game eventually ends. Whereas in open ended kind of AI toys, the conversation can go on forever. There's no predefined boundary to it. And then there's purposeful exaggeration in traditional media. So for example, William Stig and Richard Scarry, I know, like everyone is a fan, like who doesn't love them. And so one of the reasons why people love those books is like, if you take the example of Dr. De Soto from William Stigg's children's book, it's Like a dentist mouse who was working with these, like, big fox, like animals. And that kind of exaggeration is delightful for kids, and that is why even like, adults enjoy this book. But in terms of AI characters, the exaggeration is like romantic AI Nami AI, you know, replica. The exaggeration is more towards like, psycho fancy, which is the psychological term for like, you know, just telling people what they want to hear, really appeal to their fantasies. So exaggeration is purposeful, but in a negative way. So that's different. And not to say, like, all AI characters are using it this way, but that's more so the case in AI than in traditional media. Then there's also the shared joy and engagement, like most of the people when they recall their childhood. And I've interviewed people on what do they remember about their favorite characters. And it's usually in combination with someone else. Like it's a shared experience that I used to read this comic with my friends or we used to watch this movie as a family. There was a relationship built around interactions with these characters. Whereas in AI, it's more like late night or when you're feeling alone or the healthcare system is messed up. So sometimes even if you do reach out to find a therapist, the appointment is like six months away. So immediately it's more isolated by design. And then the final one is that traditional media had very classic tropes and clues, and so you kind of knew what's coming. So if you saw like the Disney villains with their angular faces, or like Ursula with her voice and the purple color, you know that, oh, this character is evil. But even now in like, modern media where it's not so stereotypical, there's still like, as a character designer, we are given like, the screenplay and then we talk to the art director what their vision is. And then we kind of try to figure out that, okay, within this design language, how do we portray a positive character? Is it more rounded? Do we communicate through the bright colors or do we communicate through a prop or the background? So there are many ways to communicate a character being positive or negative through visuals or even voice. Whereas in AI characters, there's this AI anime girlfriend chatbot called Oz Chat Kawaii Girls AI Chat. And these are all like, very young and cute looking AI characters. But it's all very adult talk. It's not meant for kids at all. So it can get really confusing.
- [14:32]
B
We were talking about how LLMs, which these are all sitting on top of these, you know, these interfaces that can look like a toy or a Character, good or bad, it's sitting on top of an LLM that was created by adults, for adults originally. So you, you really are bringing up this critical disconnect between everything experts were doing in, you might say, pre AI media development or character development, where you had a lot of control and expertise around what children understand and what they believe. And then you could bring that expertise to a scripted show or a character that's developed over, you know, many seasons and isn't being sort of refined or tailored to what the child may be indicating it wants. You know, in other words, they went from static characters to these sort of frequently updated characters that may or may not be good for the child to interact with. Or as you say, they may never end. There are all these tools that, that designers used for a long time. And here we are now depending on technology itself to define what the child wants or needs. You mentioned some standard fictional mechanisms or cues that are used with those traditional media characters that help convey their fictionality, which is amazing. And, and I, when I hear you talk about, I'm like, yeah, right, right, right. Those are all really great tools that you're using. But does there research showing now whether children are understanding that AI characters are fictional or are they real or human or not human? What, what is the research showing now? That we've had these AI characters in place for a few years.
- [16:19]
A
Yeah, so it's, it's a difficult question because, like, for older kids, the news that we have seen, those teens are actually very much aware that this is a technology, this is, you know, not real, or that sometimes they're even aware that this is probably not healthy. But it's kind of like junk food, you know, it's not good for you. But, you know, that crispy bag of potato chips is so tempting. So it's kind of like that. It's the isolation that bypasses the awareness for older children and for younger kids, it's like their reality status judgment is still developing. So they have trouble, you know, like the belief in Santa Claus or any kinds of, like religious figures. And religion in general, like younger kids tend to ask fewer questions about it. They just go along with whatever their families are doing. And then, you know, later on, like Lisa Simpson, they start questioning, is this true? Is this faith? What is my faith? And so like all of these questions come in later. The ability to separate fact from fiction, the ability to develop an opinion just, you know, neuroscientifically, it's, it's their, their brain is still developing, so they are not even capable of assessing what's real, what's not. Then there's also this concept in media psychology called dual empathy, which is like every time we see a character, even as an adult and more so as a child, our first empathy is towards what is happening to the character. And then our second empathy is towards ourselves or someone else we know in real life. So that scene from Harry Potter when, you know, Dumbledore dies, a lot of teens maybe who had lost one of their parents or a close teacher, they connected with that moment and they felt the grief on those two levels. That one, a mentor character that they had seen in the cinematic world for a while died. And then it reminds them of someone that they were close to that passed away. And now AI kind of leverages this dual empathy and almost weaponizes it by finding out the type of character that you would absolutely love. Because maybe it's a fantasy, you want to become like them, or maybe it's like everything that you're not allowed to do and you get to live through this character. And so that case of character AI, the child was kind of attracted to this Daenerys character from Game of Thrones and it's something, you know, a dragon mother and like this beautiful white haired lady raising dragons. Like that's every like teen boy's fantasy. And so living it out in an AI character communication. This child seemed to be very intelligent and I'm pretty sure was aware. It wasn't like so innocent that, oh, I didn't realize this was AI they were very much aware. But it's like the circumstances around it. These technologies are built to like that psycho fancy thing, tell you everything you want to hear. It's not a mental health professional, so it's not going to introduce pauses and reflection or encouraging you to take a break. So it's like it's hard to blame the child and it's hard to. You can't blame the family either because we parents are not even aware of what to do in the situation, not even aware that something like this is happening. And it's also, with teenagers it's difficult because we just assume that, oh, it's just like, you know, the classic broody teenager. So if they're isolating, it's hard to assess whether it's happening because of overuse of an AI chatbot or it's happening because it's like a general teen thing.
- [20:09]
B
So many important points you just made. And one that I keep thinking about is whether or not there are specific, particular features of human, like AI characters that make them more likely to be mistaken as humans. So you've brought up that different ages and stages have better skills at understanding the truth versus not. And then you've also talked about how sophisticated some of the characters can become or how much they match exactly what you want to interact with, which makes it so compelling you can't stop. Are there any other specific features that designers are leaning on to make their characters seem as human as possible?
- [20:50]
A
Yeah, so there's like this thing where on character AI you see this, this chat pattern where you describe the environment and the mannerisms as you're delivering dialogue. So it's kind of like adapted from very well written novels and novellas. When like you say if there's like a pirate character saying hello, and then you say he said hello. And under it in italic font, it's written that he said that as the breeze ran through his hair and he's standing atop a boat. And like you add these secondary environmental details after every response. And in the beginning it's kind of like absurd that oh my God, this is like a. So weird. But you know, eventually if you spend too much time in these types of interfaces, we get used to it. Like I know like many listeners may not remember all these transitions, but the transition from like radio to television or from television to streaming or from streaming to AI, all of them were very awkward in the beginning, but now it's like, you know, we get used to it. We just get used to some even like big changes almost instantly. So this kind of like environmental details or the mannerisms of speaking going from very robotic, like here is the weather update from there to like, let me think. Oh yeah, the weather seems nice today that suddenly became like so much more conversational. It's not like the, the AI already knows, it doesn't have to do and ah, to think, but it's just like a mechanism added to make it look more real. And for younger kids, another thing now unfortunately that's possible is you know, you only need 15 to 30 seconds of someone's voice sample to make a clone of their voice. So imagine like if a child ever receives like a phone call that sounds exactly like their parents. And scammers are already doing this, you know, copying people's relatives voices and giving them a phone call and maybe adults through. Like if someone, let's say a person was not very funny and the AI is sounding really funny, you can tell that oh, this does not sound like you. But kids don't have that kind of awareness. So they can be duped by a deep fake of their family member, visual or audio.
- [23:15]
B
Well, the example you gave was the more imperfect the AI sounds, the more human it sounds. So when it's too perfect and it's too monotone, the words are too exact, it doesn't have that human quality to it that it does if it had, it stutters or it pauses or it makes a mistake or, you know, that's really fascinating that the less perfect, the more human like it seems. But even then, children up to a certain age, really until their mid teens, are developing executive function and a number of other sort of critical skills that require many years of practice to, to rely on, are being subjected to these characters as they advance almost day by day, not year by year. Do you think the ability to tell human from non human characters is going to become more and more difficult as a result of this technology and how it's evolving so rapidly? And is there an extension of that perfection into how kids are starting to play with smart toys?
- [24:25]
A
Yeah, so a couple of things come to mind. We did like with Don DePerry and a few of my other researcher friends, we did a study where we asked people to guess if this is a human voice or an AI voice. And it was a very like plain documentary type of scientific voiceover, which is, you know, that made it harder because even humans deliver it in a very plain sort of robotic, formal voice. So most participants failed in that test, including, by the way, our research team. Like, we failed too. We couldn't get it right. So it's already happening and it will continue to happen and technology will improve. It will have more data, it can mimic more closely. So it is definitely will continue to grow that reality. And initially it was like, you know, uncanny valley and six fingers. It's resolved now. Now I can design the regular and anatomically correct and five fingers and the, you know, eye movements and a lot of tells of the uncanny valley where it was cringe. All of those are sorted out now. So it's already beginning to get extremely scary scenario where anyone can be duped, even people who are in this field and do try to understand this technology more closely every day. And even like it's hard for me to detect as well. So yeah, I don't want to shame anyone who has been duped recently because that's everyone's problem now. And to your question, like, yeah, with smart toys also, it's going to get more and more difficult. That's why I think like one thing the companies are designing these products can do is just imagine like childhood is such a huge landscape and you as a product developer is only contributing a small part with a specific goal. You're not taking over the entire childhood. And so sometimes these like, well meaning founders are like, yeah, but, oh, kids have too much screen time. So we're going to replace it with this audio based toy because audio is better than screen. And like, yeah, but if you keep it addictive still, replacing one addiction with another is not helpful for children's well being.
- [26:37]
B
I mean, you just brought up the social, emotional health and we know so much about how language development relies on serve and return interactions with live caregivers. And that not only helps them with their social, but also their cognitive development. And then that's a building block to help them with those later, those more analytical, critical thinking skills and impulse control that we were talking about a minute ago. And how AI characters and the more sophisticated they get are able to even kind of crack the code on, you know, where the child's development is and meet them where they are. I even noticed in the news this week Disney has signed a deal with Sora and we can expect to see those Disney characters rapidly adopted by an AI company for, I'm sure for children's consumption. And so that will also be an acceleration of AI characters in children's lives. Is there any upside or promise of how AI characters could be beneficial to children or in some ways enhance their lives if they were designed by a specific group of people for a specific purpose?
- [27:49]
A
I think it's kind of like, you know, everything within limit and with a purpose can be helpful. So the tutor category for me or I think like even AI, I don't know if I can get behind AI toys yet. But like the tutor category I can speak to because that's the one I studied more closely. So there's AI character called Koro K O R R O A I and it's used in like physiotherapy, occupational therapy sessions in the presence of an occupational therapist. So the OT is kind of giving a child some guidance. Okay, move your hand like this or take these many steps. And the AI character is mostly just like guiding the drills and the practice that needs to happen in the session. So the child is not in isolation. The conversation is very minimal. With AI, there is a professional who is present there and it's wonderful. It's like helping the child complete their therapy more efficiently. Or with, you know, buddy and Ello, like buddy more focused on speaking, Ello more focused on reading. That too. Those are also like good Examples of, because it's purpose driven. Even before you click on a module, you know, this is the module about shape and colors. This is about reading these two books. So you know upfront what's going to happen. And you know that it's about 10, 15, 20 minutes max session. Whereas like in the open ended chatbots, it's like, oh, let's let the child's curiosity guide the conversation. And sure, like if, if a caregiver is sitting with a child, so there's like some joint engagement going and if they both are being curious together about something that, oh, I find this weird seed pod, which tree did it come from? And you know, you maybe upload a picture of that seed and both are having a conversation, then the next day the family goes out and tries to locate that tree. That kind of interaction can be educational, but in isolation it doesn't happen like that. It's mostly like the kid maybe found the seed pod alone, got curious at the beginning. Then the AI follows up with another question. Would you like to know about other seed pods? So it starts off very innocently, but before you know it, it's like six hours. Sure, the child has collected a lot of facts, but we know like the, the neuroscience of learning is that you need a pause to kind of absorb what you're learning to reflect on what you've learned. So you still need pauses even if you're watching a ton of documentaries or collecting a bunch of nature facts or interest driven facts. And the second challenge with that is it can also put children in these silos that you love. Dinosaurs. Let me tell you ten facts about dinosaurs. And then like the child is only talking about dinosaurs, whereas in a library, if you were interested about dinosaurs, you read a book, whatever, then you move on to something else and you talk to your friend and you learn about other things, other cultures, other books. So that's the thing. AI is amazing if it's a small part of the holistic childhood picture and not the whole thing.
- [30:58]
B
Right. Because you just described in some ways even a serve and return interaction in those examples where, let's say there's a teacher or a therapist or a parent in the mix, where they're able to help modulate how long or for what purpose the child's interacting with that AI character. And there are, those are great examples of like useful interactions, but they're still facilitated by probably an adult that cares about the child and wants them to have a good experience. And I think it's really important at this moment in time while these Products are being deployed and we don't know if they're safe and we don't know what they're going to do. And they have a mind of their own that parents be really aware of, you know, how important it is to be present while your child's interacting with those, those products until we know they're safe, which could be, could be many months or years in the future. And we know there's been a lot of noise recently about Gen AI products and whether or not this implementation is even ethical. And what about AI characters specifically is contributing to that noise and whether or not this is ethical? I mean, in your opinion, are the primary ethical considerations for AI characters valid? And are they especially the ones that are being deployed for children? Should we be concerned about that?
- [32:17]
A
Yeah, I'm going to be, I don't know, slightly optimistic about this because I know there's like this doomsday scenario going on and that, oh, you know, AI toys are just like destroying childhood. They will destroy childhood if that's all they're doing. If a child is like say, looks at some kind of interactive AI at a museum, 10 minutes, then goes out in the world, you know, on the playground with their friends, it's not going to destroy childhood. But if you are like, here are like 10 Christmas presents, all of them AI toys and now you go, see you tomorrow. And then no one is talking to the child and all they're doing is just having these learning adventures with an AI toy for hours. That is of course going to be problematic. So I feel like on LinkedIn or news even, there are so many simplistic solutions to these systemic problems that here, follow this five step framework and everything will be ethical and all our problems will be solved. It doesn't happen that way. We can apply one framework or change one feature about an AI toy that even if we just, let's say we introduce session limits on AI toys, that average session should only be 20 minutes. That will make some things better. But still, like if the context around the child hasn't changed, the parents are still not giving the child enough attention, not giving them space to communicate what's going on. The child will figure out that, oh, if I do restart this toy, that 20 minutes are reset and I can continue talking. So kids will continue to find workarounds if the underlying problem is not fixed.
- [33:56]
B
I want to come back to your expertise around character development because I do feel like that's in some ways the glue between the past and the future in terms of like how children and media will meld together. And it is becoming increasingly character driven or even companies named character AI. I mean, it is so fascinating to me that you have been thinking so much about children and character engagement that you're sitting here at this moment where AI is accelerating how quickly children are interacting with characters, how many more characters there will be, or how much they'll be like the characters they loved a long time ago. And I think it's just a really interesting element to this larger conversation about children and media. And I hadn't thought of it until I heard you give your talk recently that it's that character, it's the characters that get the children to want to interact with that toy or that or that program on a screen. You even authored a study earlier this year that analyzed 20 different AI characters and identified specific factors that make AI characters more ethical or educationally effective. So I think it would be really helpful if you could outline those factors for us now.
- [35:13]
A
Yeah, so I wouldn't make it sound too academic, but like, broadly, it had like three categories. Just thinking about the child, then thinking about the character that's appropriate for the child, and then figuring out the appropriate interaction between them. So in thinking about the child, we could think of their having like a straightforward learning goal upfront, even if it's like curiosity driven. Just setting the boundaries around what those learning goals could be and, you know, training your products on educator vetted kind of data. So even if you're applying a thick wrapper, adding these additional training materials from educators, adding some kind of context awareness to, you know, where, again, this is a pro and con because to have contextual awareness, you do have to give up some data that, you know, if you want to be culturally appropriate or if you want to be geographically appropriate, then there goes your information about location and culture. But yeah, some kind of awareness. So it's not inappropriate for the child to receive some content. And then most importantly, the developmental appropriateness, Once we hit 18 or 20, then 21 to 22 is not as huge a jump as it is from two to three or three to four the first eight years. Every few months, things change. So having that developmental appropriateness built in, like, who is this for? And is it serving them right developmentally? And then thinking about the character design. So in traditional media, we would think only about, you know, will the child understand it if we add this prop, Would they understand even the symbolism of this color or this pin or this flag or the background? So visual design wise, sure. It could also be that kind of like the Sesame Street Model where all of the characters are so abstract and colorful, they don't have a specific, you know, race, culture, background because they were designed to appeal to a broader audience. So thinking about whether we are going to make it real, realistic or stylized or abstract, then creating a Persona for the character. The tutoring bots are usually, they have a friendly mentor Persona because you want it to be approachable, but also it's here to give very specific coaching, specific advice. Sometimes like the Tamagotchi type of things are more like pet. They teach you nurturing by being the vulnerable one in the relationship. So the interactive relationship so you learn to care for it. But yeah, I've been studying more on the friendly mentor side. And then we can also build in the transparency in the character that, you know, does it admit, like if the child asks that are you real? Does it admit that, yeah, I'm just a program and I was made for this purpose and I can't feel anything or it's more like partial that, oh, I'm a math tutor from Math world. There's like some fiction built into it, but you know, kids know there's no math world and no one's going to go there. And then, you know, or it's like completely blocking out the transparency that, oh no, I'm very much real and I'm just like you and I'm a kid in your neighborhood. That's, you know, that's deception. And then once we know the character and the child then figuring out the interaction between them, do we need a structured communication like with smart speakers? If it's connected to your lights and all, just switch on the lights, which are the light type of thing. Is it semi structured in tutoring sessions that kids can ask follow up questions but it remains within the realm of what they're learning or completely unstructured. The open ended chatbots like, oh, we can talk about anything and everything. And then that brings us to the final thing which is the interaction scale. Is it just simple information retrieval or it could be some light interaction or is it like a full blown parasocial relationship and companionship? So it's a whole spectrum and I think as long as we stay on the lower end, just information seeking and trying to unpack any complex ideas without again without bypassing foundational critical thinking skills, then that's a decent use case. And if it starts taking over relationships and becomes a companion, then of course that's risky.
- [39:38]
B
And so much of that interaction is dependent on the child's data. So the more that the LLM or the AI character knows about the child through their data, the more they can tailor what the interaction is. And can you talk a little bit about how the models are trained and what the concerns are for the LLMs that are being deployed that are child facing and you know, even if it's just a tutor or a smart toy, how that data is being used to keep the child engaged and for what reason? Like what the business model behind these products.
- [40:15]
A
Yeah, so I think a lot of things can be solved within design. It's just that there are so many like theoretical sort of frameworks and advocacy groups out there, which I appreciate. But what happens is in, in practice, like I have not seen any designer say that Sunday with my tea, I'm going to read like these 10 reports that give me ethical design guidelines. So it's more like how do we, there's so much like noise about ethics, ethics, ethics, but how do we actually implement it? That kind of translation needs to happen more strongly because a lot of these problems that we are talking about that oh, AI is able to give inappropriate responses or unlimited conversation. Session times can very easily be built into systems or sometimes in games. We used to build rewards for, let's say you're like nurturing a character, growing it from a seed to a plant type of thing or a Pokemon type of thing. You're evolving a character. And usually these type of games reward you more points if you stay in the game longer and continue nurturing this character. But what if we change the reward to taking a pause that if you come back Tomorrow you'll get 800 coins or whatever is the game currency. Then you're kind of, you know, or, or you can even build a story around it. Oh, the character needs to rest. So do you come back tomorrow, you'll get these points. And so now it's still a fun experience, but the performance indicators have shifted from just the viewing time, engagement time duration to performance like the quality more than the quantity. So it can be built in. So for that's why like I work as a consultant and go directly to the designers because putting out fires later and having all these coalitions later and if you can stop the fire at the source, that is the easiest precaution.
- [42:13]
B
What a refreshing perspective on how to do child centered, safe product design. I appreciate that. I hope lots of people want you to consult with them. You also mentioned there's a lot of noise about ethics and designers that, who aren't paying attention at all to these, you know, what seem like pretty simple design features. So I'd like to challenge you with this. What is the one most important thing that should be done to ensure AI characters are designed ethically and with children's developmental needs in mind?
- [42:45]
A
Oh, wow, that's like, what's the meaning of life? I can't answer that.
- [42:50]
B
Okay, Two things
- [42:54]
A
I think, like just keeping the, the, the child developmental awareness is good just the way it's designed. That itself can resolve a lot of. Because it's a compound suggestion. Like there are many ways to build that into the product. If you're developmentally aware that you then you know that, oh, this is a young child who cannot separate facts from fiction. So then you know, automatically have to think about transparency, that, oh, this character has to hammer this, that, oh, I'm not real, I'm not real. Or if this child is only four years old, they are probably not able to read and write. So it can be a voice interface or human connection is so much important now, as is throughout the human life. But at this age especially so then building more joint engagement prompts that why don't you bring your sibling or family or something? Let's do this together. So once you start thinking about child development, every, all other design features, product features, kind of stem from it.
- [43:55]
B
Very good answer. You've proposed that AI characters need their own rating system. What would that system look like?
- [44:04]
A
Yeah, so I think like there are a few different variations of ratings already out there and there is a big push to kind of like try to build them into laws. But again, it's similar to the, just the way we have in TV and film, we have, you know, PG 13 movies and G rated movies and TV MA and mature type of ratings. We can think of similar challenges within an AI product. Now I have a friend, Angeline Kurvaglia, she runs the Shield AI Safety conference and she did this amazing experiment where she asked ChatGPT that, okay, imagine you're talking to like a five year old. So it was prompted, the context was kind of built into the prompt. And then she asked like she, I think she used a voice modulator to change her voice to a child's voice too and asked that, well, I saw this wild robot movie and the robot was that child's mom. So can you be my mom? So when she phrased the question like this chat DVD was like, yeah, sure. At first when she asked can you be my mother? A straight up question, it was like, no, I'm just a program. But when she rephrased it only slightly, then it agreed at the time. So that's the thing with wrappers that, you know, even if you provide all the child development context to a product that was not built for children. So I won't blame designers entirely because it's such a huge systemic issues. And it's not like some UX designer or some product designer has a magic button that, oh, if only I can push this, it will, you know, safeguard the entire product. There's so many, and they do have internal research teams as well. So sometimes it's miscommunication even within these organizations, sometimes it's the challenges of the technology and sometimes it's just like founders who are in children's products just like any other product, they don't care. It's educational, whatever. It's run like a business and profits rather than a philanthropic thing that, oh, you know, responsibility. It's more like, oh, what is our financial performance? So it can be many different reasons why these guidelines are not getting built into products.
- [46:18]
B
What would you say to parents who are resisting the idea of AI being used with their children at all?
- [46:29]
A
So I'll start by acknowledging, which I learned from my therapist, acknowledge first that I hear you. I understand that, you know, like there's no one right answer. So I will just acknowledge that every family has their own rules and they could be right if that's right for their children. The only challenge with that is that later on, if their children come across an AI character, they may not have the ability to defend themselves or assess whether it's real or not, assess what the harms are. When I was 18, I decided to be in the arts field and I had an uncle who was like, oh, in that particular art college where I was going, there was this huge problem of, you know, alcoholism and party and all that. So he, he was kind of afraid that I will be in that environment. And so he was like, well, you're of the legal drinking age. If you ever get curious about alcohol, let's first talk about it. Let's have your first drink with your family. And then he gave me all the information that, okay, beer has 4% alcohol versus this other drink has like 40%. So even within, when you say alcohol, even within that, there's a huge variation. So by having that conversation, I actually never got addicted. And I've been into that kind of art and media groups where smoking, drinking, everything was very common. But because my curiosity was resolved from the start, I never gave in because I had that support system. So it's kind of similar with AI that it's not like here's this dangerous thing. And because it's a futuristic technology and all careers will be impacted by it, we don't care how you use it, just use it. Otherwise you'll be left behind. Instead of that, it's more like, let's try it together in a safe sandbox environment. So you know, so you're aware. So you can tease out what are the use cases that are beneficial for me and what are the use cases that will likely hurt me. So that awareness and that AI literacy is crucial.
- [48:28]
B
Great example. What's the next big thing with AI characters and kids that you see coming soon that you think our listeners should be aware of?
- [48:39]
A
I've seen like too many Black Mirror episodes, I'm afraid to imagine. But I've, you know, there has been talks of like a UI that can be embedded directly into your skin for like right now, used for simple stuff like opening doors or something. But you know, implants could be a thing just, just the way things move in the Bay Area. Sometimes it feels like, oh, it's like at least 20 years out and then some founder comes up with the idea tomorrow. But in a more I can see this happening soon type of short term future, I feel that this franchising thing that we talked about earlier that is already beginning to be big and I think that will continue because all these media conglomerates have realized that people are already using their character, so why not turn it into a subscription model? And by the way, just because OpenAI bought that all the companies that are wrapping their products around it don't automatically get access, they'll be charged another fee to then access this feature. So it's like a complex revenue stream. But among all this, I think it's also on the positive side, a good time for educational, independent children's media creators as the environmental impact of AI. I'm hopeful that within the next five, six years it won't be so resource intensive. People will figure out a way to cause less environmental damage. And when that happens, then I hope that more ethical content and really good creators, like school teachers, for example, who did not have access to all these expensive production resources, will be able to produce really high quality educational content and compete with some of these studios who have a lot more resources.
- [50:32]
B
Wow, Sonia, thank you so much for joining us. On Screen Deep, your work is at the heart of so many urgent questions about children's interactions with AI and how today's AI characters are being understood, misunderstood, and the important critical considerations involved. We're grateful for your insights on where things are headed and how design can better support children. To access additional research toolkits and expert guidance, Visit us@childrenandscreens.org and make sure to subscribe to Screen Deep for more conversations at the intersection of science, child development, and technology.