Loading summary
Transcript44 lines
- [00:04]
A
Hey, Bankless Nation. We got some extra bonus content for you here on the podcast feed. This is an episode from Limitless, our sister podcast that focuses on AI and frontier tech. If you remember, earlier this year we were just doing a ton of AI content on Bankless, just because AI is supremely interesting and worth paying attention to. But we were making so much AI content that it just started to crowd out all of the crypto content that we also wanted to keep doing. So we made a second podcast called Limitless and you should go subscribe to it. The episode from Limitless that you're about to hear right now is with Limitless hosts Jaz and I as we interview two OpenAI team members who are in charge of memory and personalization of ChatGPT. They're the ones who helped ChatGPT evolve from just a dumb LLM chatbot to your personal assistant who can remember everything about you and make guesses about your interests and just overall be helpful and perhaps be a friend rather than just an inert chatbot. It's not often that OpenAI team members go on podcast, so we are very privileged to be able to host Samir and Christina to get all of our questions answered about our favorite little chat assistant and what it's going to be able to do for us in the future. So I hope you enjoy this episode from Limitless. If you do enjoy it, make sure to click the link in the show notes to go. Subscribe to the Limitless podcast so you can stay up to date with the rapidly evolving world of AI. So let's go ahead and get right into this episode of Limitless.
- [01:26]
B
I'm so excited about today's episode. We are joined by Christina Kaplan and Samir Ahmed, who both lead memory and personalization at OpenAI. Samir Kristina, welcome.
- [01:37]
C
Thank you. So excited to be here today.
- [01:40]
B
Okay, you are both solely responsible for creating the most valuable moat that's ever been discovered in AI memory and personalization. In fact, I remember the very moment you guys released the memory feature because five seconds before that I was ranting to my friend about how annoying it was that ChatGPT had no recollection of who I am when I opened a new chat and it was like this light bulb moment went off where I stopped viewing ChatGPT as this tool and more of a friend. And that's an incredibly sticky product experience. But I want to hear about it from your side. What was it like to develop ChatGPT's memory feature? Give us the inside scoop. What was it like to go about Creating this vision around memory, I think.
- [02:24]
D
Thanks for the. The generous introduction there, Jaz. I think we probably would classify it slightly different. And, you know, to be totally candid, memory and chatgpt predates both Christina and me joining the team. I can, if we rewind all the way back to 2022, ChatGPT comes out. You know, I'll give you a little bit of a analogy here that I think is like, truly opt way of how we think about it. And 2022, sort of like imagine you had an assistant in a room. You walked in, you asked a question, and it answered natural language. And this is the first time you seen a computer like answer in natural language and gives you the answer. You leave the room, stops working, your assistant stops working. You walk back in and you say, hey, how's it going? Ask another question. Assistant does not remember who you are. No idea. It's a paradigm shift. But really is not, you know, this the type of experience that you would normally have in a, you know, situation? We're working with an assistant. So fast forward maybe to 2024. It's almost like we gave ChatGPT a notebook and it could write certain things down as you were talking to it and you would say something. ChatGPT write it down. It would try its best to know what's going on. You leave the room, you come back, you'd ask another question, and it would like, scour through the notebook to help answer the question it had. And it's a little bit like, you know, maybe the movie Memento where you got tattoos on you and you have, like, some clues, but it's not a perfect picture. So, you know, Christina and I have been thinking, like, how can we make this better? There's a lot of people involved here. But that's when, you know, the April memory update, that's. That's around the time we came up with our latest variation of memory.
- [04:01]
C
Yeah. So the latest variation of memory, like, we had a assistant, a person that had a notebook with information about you, but still very limited, like, might understand your name but doesn't really understand you or who you are or really can't carry a conversation from new conversation to new conversation. So the update to memory that we launched in April was really trying to bring a more natural memory to ChatGPT. So taking this, like, person with, you know, memento style tattoos to be like a real person that remembers you, that can, like, pick up a conversation from where you left off. So that's really what inspired this launch in April is turning Chatgpt's memory into something that's more natural, that is the beginning of what you might expect from like a real assistant that is like the same person in the conversation time after time that you come back and talk to them.
- [04:49]
D
Yeah. And that's when I think, you know, that released early April. And that's funnily enough, how, you know, we've. We. We got in touch because we'd listened to, I think, a podcast you'd done on this.
- [05:02]
A
Yeah, yeah. Talk about that actually. What was it like? So we were talking about memory. I think Mia, Jaws, Josh were all very inspired just by what could happen next, what doors this unlocked. You know, in hindsight now as a ChatGPT user, I'm like, oh, yeah, this seems like a very obvious, like, feature to unlock. But before it got delivered, I don't think anyone really knew where the arc of this product was going to again. Now, now it seems obvious. Maybe you could put us in your shoes as you were, as you were listening to us, like chit chat about our experiences with memory. What. What was going through your guys heads?
- [05:34]
D
Yeah, actually, like, Christina had just gone on vacation and I was just, you know, scrolling through everything on Twitter or looking for different coverage of it. And, you know, we. The. The change was not like a visual feature that you could see, and so not everyone picks up on it right away. And I remember someone forwarded me a link to the podcast and I was a little skeptical, and I was like, okay, let me. Let me read.
- [06:00]
A
Let me listen for Limitless has forked off from Bankless. So it's a crypto podcast.
- [06:06]
D
I texted Christina, I said, you got to listen to these crypto Bros talk about ChatGPT.
- [06:14]
C
I was skeptical. And then Samir basically, like, bullied me over the course of multiple days to listen to this podcast. He was like, this is like the thing you have to do. It was really interesting to hear you both about this launch and, and to hear, I think, like, David, you talk about it's a small step, but maybe it's a small step. That's really a big change. And I mean, Ajaz, we. We like, DM'd you on Twitter immediately to talk because we were just so curious. Like, your reaction was like, much stronger. But I'm. I'm very grateful to Samir for. For influencing me listen to this on vacation. Yeah.
- [06:45]
D
Also, funny story, Ajaz, you made us do like a three factor auth. We DM'd you.
- [06:50]
B
Yep.
- [06:51]
D
And you were like, can you prove this?
- [06:53]
A
Yeah, yeah.
- [06:55]
C
From our OpenAI accounts and then I got a mutual friend three factor authentication before AJAWS would talk to us.
- [07:02]
B
But, yeah, I. I didn't believe OpenAI was listening to the Limitless podcast back then, but turns out. Turns out we say some interesting things. Yeah.
- [07:10]
A
The context for that is that, you know, in crypto there's a lot of scams, and so when somebody from OpenAI texts you, messages you, on a very public social media platform, you're like, what's going on here? Can we color in a little bit about what was the bigger, the broader reception of memory when it was introduced? Because I think we're just to kind of jog our listeners memories, because that was, you know, so many months ago, half a year ago, and time flies. I was skeptical. I honestly can't necessarily remember what I was skeptical about. I know I remember Josh being very excited. But, like, give us the range of, like, feedback and reception that ChatGPT users had to the. To the memory feature.
- [07:45]
C
This was a change that was mostly under the hood, except we did release a notification to share with people that this change was happening. And it kicked people into a conversation that was something like, tell me based on everything you know about me, about who I am, and, like, make it catchy. And I think the reception was twofold. There was, like, really strong reception, actually stronger than I think we had originally anticipated to that prompt. Like, people were really excited to see how ChatGPT had been understanding them and, like, really learning about themselves from hearing about ChatGPT's perception of them. But I think that we were surprised by the reception. Like, it Jaws is honestly like, hey, this really changes the experience because other than that one notification, there really wasn't much to it visually. And we sort of hypothesized that this natural evolution of memory would be really meaningful. But it was a surprise to us how many people actually, like, felt that in their experience.
- [08:44]
D
And a lot of people will, like, extrapolate, come to, you know, jump to different ideas from that point in time as soon as they see it. And so it was really, really interesting to just see different takes on this, but, you know, to ground us in, like, how we think about it. A lot of it has to do with this, just a universality of memory and how we can. We can sort of meet people where they are. And there's a little element to this around, like, everyone has a different experience with ChatGPT. If you're, you know, living in San Francisco or if you're living in Indonesia, if you're in Brazil, everyone has a way that they communicate with other humans. And memory helps sort of like make the playing field universal so that people all over the world are able to like communicate with ChatGPT at the level that they understand. And it's not like mapping it to, you know, how people communicate, interact with computers. Humans have been talking to humans for tens of thousands of years. Everyone sort of understands that humans have been talking to computers or interacting computers for 30 years. Button clicks like widgets. And so that's maybe not as natural and not the best way to communicate and tackle higher order concept discussions.
- [09:54]
C
Hey Jaz, I'm curious to replay a bit of like, your reaction because it was very strong and also like how you think about it now a few months later.
- [10:03]
B
Okay, so how I thought about ChatGPT pre memory was it was like talking to a friend that had amnesia. You'd have a conversation, you'd get into a topic about something, and then a couple hours later it'd be like, hey, who are you? Tell me about yourself. And that kind of broke the illusion. Maybe illusion is not the right word, but it kind of broke the experience a little bit. And so when I saw that memory update on my app, what you described as a kind of small, internal, under the hood update meant a lot more to me because I knew that over the next few days I'd be having less of those, hey, this is me conversations and more of more of just conversations that floated into something much more bigger.
- [10:44]
D
That compounded when you guys think about.
- [10:47]
A
ChatGPT and the evolution of the product. I'm a psych major and I find it very easy to put into metaphors a lot of how ChatGPT works, or at least how I understand it to work. So like, short term, long term memory, like the short term memory is literally what is the last four prompts that the user and I have just been discussing. So let's retain that information first, but also in my context, in my response. And let me also understand a little bit about the deep truth that I know about this user and their interests and their hobbies. And I will frame my response based on like the long term memory that I, that I understand the user to be. When you guys are thinking about ChatGPT, do you think of it in a cognitive psychology lens?
- [11:32]
D
Yeah, we look at it, I think, you know, we try our best to look at some of the prior art here. And in that scenario it's really helpful to understand like how humans interact with humans and some of the advantages and pitfalls there. And chatgpt memory is Today, nowhere as good as humans. Memory and sort of understanding the gist of people interactions and the fidelity and the triggers. So there's a lot that we can draw on from the sort of existing cognitive precedence.
- [12:01]
C
It's like a much more natural way for people to interact with something as well. Like, on the other end of this is a person. So a person. Like I am interacting with ChatGPT, I interact with, like many other people, it's much more natural to interact with something that you expect to be an assistant in a way that you would interact with another person. So I think a lot of the, like, at the end of the day, this really comes back to like, not chatgpt, but the user. Like, how do people expect something to interact with them? And like, how do we meet everyone, where they are and like, what they expect as the user?
- [12:32]
A
I can't exactly remember what I was so skeptical about. Clearly that skepticism has dissolved because I remember just maybe one, two weeks after this feature got released that I realized, like, oh, this product gets better the more I tell it about myself. And a lot of, A lot of.
- [12:49]
D
People had reached out to us further down the line, you know, weeks, months, and then mention that this is one of their favorite features. And, you know, I. I would always ask, is it a single moment, a magic, or is it a slow burn? And, you know, different people give different responses, but for some people, it's a slow burn. Like, they just realize that things have shifted and like, ChatGPT is behaving in a way where it's. Clearly, it understands them. And I think it has to do with maybe how you're using it. Everyone uses a different fashion.
- [13:17]
A
Yeah. And this feature, the memory feature, unlocked, I think what is many people's not favorite prompts, but I think a prompt that many people have asked ChatGPT, which is based on everything that you know about me. Who do you think I am? Or. Or what do you think my personality is? Or some like, hold up a mirror to me and tell me what you think I am. And so it kind of turns chatgpt into a diary in a sense. It's like I have like, queries and prompts, like, help me plan a trip that I want to make, but also the. The trip that I want to make tells ChatGPT a little bit about myself. And so every single prompt, it. It gives, it's like a little bit of just like, here's one more shade of, of who I am. And it really encouraged me to like, you know, give ChatGPT all my data Tell me all my interests. Tell me the secrets that I've never told anyone before. Yeah, talk to us about the arc of how user behavior changed with the introduction of the memory feature.
- [14:14]
C
It's a good question. I actually would say user. My user behavior has changed much more with, well, I think both with enhanced memory, but also Pulse, which came out last week, which is a kind of like natural evolution from memory as well. But I think with memory, like, I started sharing with ChatGPT more context. So instead of just going there with a question that I wanted an answer to immediately, like, I knew that this context would come up in like Future Conversations with ChatGPT. So if you share, like, hey, I'm looking for a fish forward restaurant for tonight, like, help me find that. It's kind of unclear to ChatGPT, like, why am I looking for that? Am I looking for it because, like, I'm pescatarian and so it shifted my behavior into be being more along the lines of like, hey, I'm pescatarian, I'm looking for a restaurant tonight. Can you like, help me find something? And like, I've started sharing facts along the way as opposed to just like going in with a question like I would in like a traditional search product. And then like down the line, as Samir mentioned, it's like a slow burn. You see those facts come up in the future. And so now Chatbot, if I'm like asking about a restaurant, it will suggest like fish forward pescatarian restaurant friendly restaurants. And, and I can just like expect that now. And so I've started just sharing like facts about myself and context that I like, expect to be helpful in the future.
- [15:28]
B
It seems that this, this approach with Pulse and with the data you guys are amassing is solely meant to kind of help a better user experience for the person using ChatGPT. Right. It's like this personalized experience that helps them improve and get better. But I do also want to address the elephant in the room, which is you guys now own so much data, and not just any data, very personal data. I tell ChatGPT everything and very willingly, very consentingly. And I have to ask, what are the plans with this massive honeypot of data? Are there any kind of like, thoughts on new products or maybe new experiences that you could do with something like this?
- [16:06]
C
Yeah, I mean, I think we're very focused on like, helping people achieve really meaningful things in their life. I definitely, you know, Sam has said too, like, ChatGPT will probably be the most like, sensitive account that people have and like, we take this very, very seriously. But there's also so much to be gained for this. Like, I have many personal experiences in the health domain. For example, I was going to get vaccinated for a trip I went on in July and I had already shared some of my prior, like, vaccination forms as well as my labs that I had done with ChatGPT. And I asked the nurse that I went to go get vaccinated with, like, hey, what vaccinations do I need to go to this trip? And in like a month or so? And he gave me four vaccinations that I needed. And then I also asked ChatGPT, like just to check and see what ChatGPT thought. And ChatGPT gave me five vaccinations that I needed. And I was like, what is the difference here? And the nurse had looked at my prior vaccination forms and ChatGPT had pulled out that I had low immunity to varicella from a lab I had done early last year and was like, based on your vaccination forms, you have these four that you need and then you also have low immunity to varicella, so you need to add that one as well. And I think that's an example of like, such a, such a sensitive use of my data. Like, I have literally uploaded all of my, like, labs and vaccination forms to chatbots. But it's helped with such a, like, individualized outcome. I wouldn't expect anyone to be able to go look at every single form I've ever shared with them before for every single, like, doctor's appointment. And so now I know that I can like go to ChatGPT and ask like, hey, I'm going to this doctor's appointment, like, what's relevant for that? But at the same time, like, I am a user of ChatGPT, sharing my like, health information with ChatGPT. And I think we take that like very, very seriously and want to make sure that people are able to get a lot of value out of ChatGPT in the way that they want. But yeah, at the same time, like, really taking data privacy and like, security really, really seriously in the work that we do.
- [18:14]
A
So I could imagine some property, something like Sign in with ChatGPT on this website and I sign and I go to some non OpenAI website and I sign in with ChatGPT and now it's my agent, my LLM, my chat that is piped into this, this website, my identity signed in with this website and all of a sudden this website can query is like, hey, what does this user want to see? And my LLM could, could answer them. These are just my thoughts about what you could do. But there is some notion about, like, the data and knowledge that ChatGPT knows about me could be extended to the broader Internet and the Internet can finally be truly personalized to my interests. Do you, when you guys think about the product is there, how over the mark am I?
- [18:59]
D
I think that, you know, if you look at it back to our like, like, assistant, intern analogy, interns in the room. You go in, you talk about some stuff. You mention the, you know, in my case, that, like, I recently had a son who's nine months old now and I come back tomorrow and the system's like, oh, hey, like, how's it going? Maybe here's a tip around, like, what you should expect in terms of, like, the development cycle and like, when they're speaking or they're babbling, this is what it means. If we extrapolate that, you know, and we just keep following that arc, I would love for that assistant to be able to go into the real world and, you know, do things on my behalf or represent me on my behalf to solve my goals. And it's just, you know, today chatu et is a property, can deliver so much value. But if you look at the arc of this, there's things that happen all over the Internet. And I think, like, aspirationally, if you had an assistant, you'd want them to be able to go and solve problems all over the Internet.
- [19:57]
B
Talk to me about how you align an AI towards a human's goals though. Right. Like, when I think about, like, a problem for us. Yeah. When I think about meeting a new person, like making a new friend or learning deeper about a person, it takes so many different types of inputs. I'm looking at the expressions on their face, I'm listening to the tone of their voice, and then I'm listening to the actual words that they're saying. Not to mention that they have a lot of past experiences that influence the words that actually come out of their mouth. When I think about the ChatGPT relationship, I'm just. I'm slamming a bunch of letters. Sometimes they make no sense. Sometimes there are a lot of typos. How do you process that into an AI and say, this is the thing David needs to do, or this is the thing Ejaz needs to build?
- [20:42]
C
Well, we are at an amazing research lab, so we. One thing that's been really special about, about the personalization team too, is we have, like, an end to end research product team. So we're able to make progress really quickly on the research side based on what our goals are in the product. And, and we have, as Samir said at the beginning, it really like takes a village to make personalization happen. And we have some amazing researchers working on this problem.
- [21:05]
D
And it works other way too. From the product end of it. If you pull out ChatGPT Pulse today, you know, you get to the bottom of the page and it will say, you know, curate for tomorrow. And that's a effectively, you know, a goal that you could set for tomorrow or something that you want for tomorrow. And it doesn't have to be for tomorrow necessarily. You could set a longer term aspect like I mentioned, like, oh, keep me up to date with developmental milestones. Or, or it could be somewhat, you know, trivial. Like I'm a Formula one fan and these races happen like at absurd times in the morning and I'll always say, like, keep me updated with Formula one news, but do not spoil the race for me and otherwise I'll be very angry. And there's no, like, that's, there's no way you could communicate that in any other tool that has existed prior to chat and get that level of like, you know, captures level of nuance to it and provides me utility in that case.
- [22:02]
C
Yeah, I already am doing this with like the curate function and impulse as, as Samir mentioned. Like I'll say, hey, I'm going to a trip next week. I'm like going on a trip next weekend. Help me plan for my trip and like, that's a goal. I have to like plan a great trip to like London for a wedding that I was just at. And ChatGPT is like sharing like coffee shops and pescatarian restaurants that I might like and helps me kind of like look ahead and see like what's upcoming in my life. And I'll just like start sharing things like this is my goal, this is what I'm doing and sort of expect ChatGPT to like meet me where I'm at and help me, help me do that better.
- [22:36]
B
Let's get into Pulse. This is a brand new feature that Sam Altman announced and credited you both specifically as the next step in memory and personalization. I kind of think of it as like my own personalized chatgpt. I've gotten into the habit every evening actually before I go to bed of talking to it Samir, you just give an example of saying, you know, tell me about the F1 news, but don't tell Me, who won because I want to watch the race. I've been doing similar things for all the other tropes of my life. You've explained kind of how it works, but let's pop open the hood, like what's actually happening underneath that. You've explained memory and kind of like putting together the identity behind David, the identity behind Ijaz. Tell us about this next step of personalization and how it works.
- [23:22]
C
Well, yeah, first of all, this is like a feature that took a village and we have an amazing, amazing team that help build Pulse, but it really is that next evolution of ChatGPT. So you have an assistant that you go into the room, you start talking to them and now they know who you are every time you come back, but when you leave the room, they stop working. And so you have an assistant that basically only works in your one on ones effectively. And that's not someone that I would hire to be my assistant is someone that might know me and understand me, but only do work when I'm there that I generally have to watch them do. And with Pulse, we started to think about how can chatgpt in understanding you, understanding what's important to you and your goals in life, help you when you're not there so you don't have to spend all of your time in ChatGPT. And the goal here is not that people are spending all their time in ChatGPT, it's that actually more successful when ChatGPT is doing things and helping you and you don't have to be there. So you come back every morning and there's like new value for you to get from, from your assistant that was like effectively doing work when, when you weren't there. So basically behind Pulse is like, we try to understand, you know, what's going to be important to you in the next few days upcoming. We introduced calendar and email connectors as part of Pulse as well. So not just looking at memory, but also, you know, what's upcoming in your calendar, what's just happened in your email and how can chatgpt like understand all this information, understand what's important to you, and then help you prepare for your day.
- [24:50]
D
It is important to note that you know, Jaz, when you wake up and you scroll to the bottom of your Pulse page, you know it ends. And so to sort of reiterate Christina's point, there, we are trying to get you prepared for the day, help you accomplish whatever that goal is, whether it's a gym goal or it's a, you know, some sort of event you're planning for and then, you know, get back to your day. And if we do that, then we can take the time in the background. ChatGPT can figure out what's important and, you know, spend the rest of the time that you're not interfacing with it getting work done, which is kind of what you'd want from most of your assistants.
- [25:27]
C
Yeah, you can kind of think of it as like while you're asleep. ChatGPT is like trying to process all this information you shared with it to understand how it can help you the next day. So then you wake up and there are like things that are ready for you that right now are mostly content based. So like, what can we, what can your assistant share with you that's helpful? But you could imagine you wake up and your assistant's like, hey, I wrote this email for you. Do you want to send it? Or like, hey, I, you know, prepared this for you. Do you want to go do that? And so I think the goal here is like becoming more and more, like helpful and actionable over time.
- [26:00]
A
Hey everyone, if you want to listen to the rest of Christina and Samir, I'm going to make you go over to the Limitless feed to do that. We're cutting you off halfway. There's a lot more good stuff left in this episode and also on the Limitless feed in general. So to listen to the rest of the episode and to also just stay ahead of the curve when it comes to AI and other world changing tech, go subscribe to the Limitless podcast. There's a link in the show notes.