Loading summary
Transcript41 lines
- [00:01]
A
Hello and welcome to the Harvard Data Science Review Podcast. I'm Liberty Vidert, feature Editor of the Harvard Data Science Review, or at least the AI version of me. Yes, AI is reading this introduction and yes, it even helped write it. But don't worry, our interview is all real people, real voices, and real conversation. I'm joined, as always, by my co host and editor in chief, Xiao Li Meng. Today we're looking at AI's impact on the job market, how it's affecting both young workers entering the workforce and those already established in their careers. Is AI taking jobs away, reshaping them, or creating opportunities we can't yet see? To explore these topics, we've invited two distinguished guests. Ben Waber, Visiting Scientist and Lecturer at mit, and Rafaela Sadden, professor of Business Administration at Harvard Business School. So let's dive in with actual humans in conversation. In honor of AI and the workforce episode and how AI is transforming the workforce, we decided to generate all of our questions today completely using ChatGPT. So if it goes really well, then ChatGPT is working, I guess, and totally changing the workforce. But it was funny because we were talking about it and something that would usually take us a couple hours to really put together the questions was about one minute. So we'll see how this goes. But we thought it would be generated, you know what, in real time, if you want to generate the answers. There we go. Then we'll see. Then, you know, this would make.
- [01:49]
B
It's just the ouroboros of AI just eating itself. But yeah, anyway, that's your job.
- [01:56]
C
Yeah.
- [01:57]
A
So when people talk about AI really transforming the workforce a lot of times and to a lot of people, it feels very abstract and it's maybe something you've seen in the movies or it's this, in 10 years, no one's going to have a job. But from your perspectives, what's happening right now? What are these most immediate and concrete ways that AI is already shaping how people work day to day beyond just generating interview questions?
- [02:27]
B
So let's talk about generative AI for a second because I think that's what most people right now are talking about. Things like large language models and things like ChatGPT. Right. But what they've done mostly is give executives at companies permission to credibly cut large numbers of the workforce or give their existing workforce more work with not more compensation, and juice their profit margins a bit by doing that. Now, I think there is going to be long term costs to that in terms of performance of these organizations. And you've Already seen some organizations go back on some of these moves in terms of saying they weren't going to hire any more people or getting rid of people and then rehiring. I think that's actually one of the most near term things that's happening. A big reason for that. I think a lot of the experience that people have with these tools, which are damn impressive and pretty cool, is using them in very short term situations where for example, the idea of mocking up a mobile app in five minutes and something that as a former computer science student would have taken me three days in the past and that's amazing. That's very cool. That's very different than building actually a working app. And I've actually, I'm doing some work right now with mapping. I'd have to use Irvine on this specifically, but this is really where companies are hurdling without fully understanding the technology. Maybe I'll stop my rant there and pass it off to Rafael to maybe have a probably different perspective than I do on this.
- [04:00]
C
I'm so glad you started. So I think at a high level there is one piece that we agree with and I think it's the notion that everybody talks about AI and nobody really knows what they're talking about, maybe including ourselves, because this is such a, you know, this being a general purpose technology or at least having the potential of being a general purpose technology. Where I differ from your perspective Ben, is that there isn't necessarily an excuse in using AI in this hype term because we have some malicious intent. Maybe some companies are like that. But I think that this is a technology that necessarily as it happened with past technological waves, requires a huge deal of firm specific experimentation before we can really understand what it does. So you were asking what is the most immediate use? I think that right now there are some very well documented individual uses which could be as AI, basically as a copilot that helps us find information or summarize information or maybe visualize it better specific tasks that AI can do. Which is very far from the notion that AI substitutes entire jobs. By and large we are still talking about individuals using AI potentially seeing some benefit in terms of time, but I would say not yet super well documented. And what we know is that there hasn't been the organizational transformation that I think is a potential for this technology, but still needs to be discovered. And I think this is in my mind when I think about AI, I think immediately about the J curve, which is a pretty well notion of what happens to productivity when you are in the wake of A new technological wave. There is a period of losses even in terms of productivity because you have to figure it out. And I think this experimentation, I will reiterate, is firm specific. Nobody has the playbook yet. If people tell you that they have the playbook, I don't trust them. I think companies need to figure it out. Then there will be organizations that I think will get stuck in this process. But there will be also, I think potentially organizations that are either native with this new technology or that will have the ability to learn and understand from different use cases and most importantly, bring their organizations with them, because that's another piece of AI that we don't talk about. The systemic application of AI will require some fundamental changes to organization. And it's not clear to me that everybody wants to go there.
- [06:37]
D
Well, thank you to both of you. I think what you said, especially the J curve, I certainly can relate. I'm still waiting for the J curve to happen to me because I keep wasting more time. Speaking of time, I also appreciated that we have these questions generated by the AIs very quick and I'm going to read one of them and the listeners will know that I'm reading because this one, you won't detect any chinglish in it. So you know I'm reading. Okay. There is an ongoing debate will AI mainly displace jobs or make works dramatically more productive? Where do you each see the balance tipping in the next decade? Rafael, you go first.
- [07:23]
C
Yeah. I can tell that this is a question generated by ChatGPT because it makes no sense. Right.
- [07:29]
D
Okay, good. Even better. Yes, tell me why it makes no sense.
- [07:34]
C
I don't. So let me tell you where, where I am with this. I think that it's impossible to talk about the effects of gen AI in aggregate terms right now. I think we are going to see some pretty big heterogeneity or differences in how AI is adopted across different organizations. So it's possible that if you look at the aggregate data, you won't see much because you will see some organizations doing nothing and other really going out there, which I think is already happening. And there is some evidence of this, for example, in the census data.
- [08:11]
B
The Denmark paper recently also showed that, for example.
- [08:15]
C
Right. So this is Anders Humlund, who is a Danish researcher who did a very nice study of the adoption of AI across Danish workers. And he found that there is a very large heterogeneity across workers that dependent on the policies of the firms, how much they were invested, how much they were training. So I think that's like, first, I would focus more on the variance than the aggregate in this phase. And second, what happens is a choice. There is no predetermined trajectory. If AI is going to replace workers, there will have been a choice of automating certain jobs, for example, which some jobs may be automated. If AI ends up creating new jobs, I think that this is, again, will be a conscious choice that might come out of a discovery process, but it's not written anywhere. And so it's a little bit up to the creativity and the imagination and maybe even the courage of people to go out and find out new ways of doing business where we will end up this time.
- [09:20]
B
I'm glad you went first. I think you framed it much better than I could. I mean, I want to point out that one can still get a paper published in an economics journal talking about the employment effects of the introduction of the typewriter. Okay, so the typewriter came out a while ago, and we still don't really know everything about the employment effects of that. So whenever people say, oh, first of all, it's not even a single technology when it comes to, like, AI today. But when people make these prognostications about certain things, not only is it very impossible to know, but as Rafael said, these are choices that we make around how we design these technologies, around how we adopt them. Right. And I think that right now, first of all, not only is it very hard to play some of these things out, but a lot of the predictions that people are making are based on, I think, very fundamental misunderstandings of even what the current class of technology can do. Even people talking about summarization, these things actually can't do reliable summarization. And we can have a long discussion about that. Some great work by some folks at UMass Boston. But part of the reason I think this discussion has gotten, I think, framed, not in this conversation, but in general, incorrectly, obviously you have people with certain incentives that want to frame it in a certain way. But beyond that, a lot of the way is just how we tend to think about even experimentally testing new technologies. And people say, oh, chatgpt is superhuman on math, which is an insane thing to say. We don't say a knife has superhuman cutting ability. I mean, it's true, but it's just like you don't. I wouldn't use my hand to cut a sandwich. Like that is incorrect. You know, the question, of course, is, what is the correct tool for solving a problem? And again, how should we design those tools, as Rafael was saying? But I just want to Use one example of actually addition. All right? So if you have people with a high school diploma in the US and you have them do four digit addition, they'll make an error about 0.7% of the time, okay? So now let's pretend I have a generative AI calculator, which you shouldn't do, but let's say I make one, right? To make a generative AI spreadsheet that's better than people on four digit addition, it only makes an error 0.5% of the time. So you could say, oh, well, that's better than people, right? But just like I wouldn't have a person calculate every cell of a 1 million cell spreadsheet, you know, if I now give you a generative AI Excel and I say, oh, a random 0.5% of these cells are wrong. That is unusable. Like, that's actually unusable as a tool, right? And so I think we've really painted ourselves into a corner in terms of, first of all, how we compare these tools. The correct comparison is not a generative AI program versus just, you know, a naked human that you pull out of the forest. It's like, okay, using a calculator, right? And then beyond that, the question is, all right, like, how should we design these tools? I think it's fascinating that we've brought back command lines, right, as an interface, despite the fact that they're a terrible interface, but now we've also made them stochastic, right? Like, it's just interesting when you think about, like the computer mouse or the keyboard. Like, actually there's literally. And Raphael can correct me, I'm wrong, but I think there are zero papers written on the productivity effects of the computer mouse, right? Which is, which is fascinating, right? No, no, because I actually looked at this. I've looked at this. I haven't checked the last month. That's shocking about it. Imagine I removed the mouse from every computer. You know, like, people's productivity would plummet, right? But the thing is, it doesn't exactly automate what people have done. Like, we can do new things. And I think this is where a lot of the interesting things in the AI space in general, I don't even just mean generative AI. I think a lot of the interesting things in the AI space will happen much more around interfaces. I think a lot of the way that these systems are built, even things like cancer detection, it's like, output a prediction. But we don't have to output a prediction. It could, if we get an X ray, it could highlight a part of the image and a person looks at it. Now that's the exact same technology, it's a different design. I think that's where the interesting things are going to come. But I do think to the extent that we keep honing in on, okay, can this do better than a person at this task, we're going to keep getting, frankly, like a very limited, crappy instantiation of the technology versus what we could do if we think there's a whole field called human computer interaction that does amazing things that I think people in the space have been ignoring. And I think that's where a lot of the interesting stuff's going to happen.
- [13:26]
C
Yeah, I mean, if I can. I think I agree with you, Ben, and I think that the fundamental issue, and you should know I teach strategy, so maybe this is my personal bias, but I think that the fundamental issue is focusing on the specific technological applications rather than thinking about the opportunities for value creation. And that I think is a different way to think about the technology. And I want to be specific here. What excites me about AI and generative AI in particular is this notion that I think David Autor from mit, I've seen it from him first, maybe other people said it before him, but it's this idea that generative AI can free tacit expertise. We would think of expert as knowing people as experts. These retrieval opportunities and generative applications can help us access expertise that might have previously been completely bottled in individuals. If you think about this type of application, I've actually run experiments that show in a controlled setting that these type of applications, which are not AI substituting humans, but it's specific tasks potentially being augmented by expertise that happens to be channeled by AI, could lead us in new places and new value creation. So, for example, I'm working on projects where we are trying to see whether AI can help blue collar workers solve problems in autonomy. Because usually to solve a problem on the line, you would have to call an expertise an expert, an engineer. Whereas now there are organizations that are thinking about interfaces that allow workers to access this knowledge in a way that is much more intuitive and easier than before. And so those are the type of applications where I think if we can think in this frame and if we can validate that they actually add value, this could potentially be very exciting.
- [15:34]
A
I feel like this all makes a lot of sense to me, but I would say the general person, but I think even people with a lot of expertise in data science, data analytics, when you hear these statements about there's not going to be any jobs for people under 30 in five years. There's not going to be any in 10 years or five to 10 years. It creates a huge amount of fear. Is there any validity to these fears?
- [16:02]
B
There could be validity, but I do think that there's some short term problems that are going to be very real for people. And I want to use an example, actually from mit. And again, I don't want to scoop myself because I'm writing a piece on this, but just to give a preview of it. So I was being a mentor at one of the hackathons there at the beginning of the summer. Okay. And these are mostly MIT undergrads, grad students, but they'd all completed at least a year at mit. Okay. And what they had to do was they were mocking up an app and they had to show working demo. And then if you won, you actually got like tens of thousands of dollars. Like it was for real stakes. Okay. And, you know, I'm walking around just trying to help some of the groups out. All the groups are using Copilot technology to code up their apps. And again, like I said earlier, you can make these things that would take you days. You can make them in minutes, which is amazing. But they actually had to make something that worked, right? At least, you know, for like a demo purpose. And so I had a group come up to me and they said, all right, well, you know, we have an error in our app. Can you come look at, like, the code? And just to be totally clear, like, yeah, I have a BA and MA in computer science and I got my PhD, you know, in this, but I haven't coded in like 10 years. Like, seriously. Right? So. And I told them this. I was like, guys, you really don't want me to look at this code. Like, I haven't looked at things and they're like, oh, no, but, you know, it would really be helpful for us. So it was fine. So I start looking at this code and it's. I don't know if any of you know, any of your listeners or you have actually looked at, you know, code generated by generative AI that's like a larger program, but it's inscrutable. Like, it works, but it is. Like, it's unreadable. Like, you have no idea why it works basically, right? And so after playing around with for like 20 minutes, I'm like, listen, guys, this can take me hours to figure out. Like, let's just look at the error messages. It says there's a variable here that's Got a problem with it? Let's, you know, try to debug this the regular way. So why don't you put in a debug print statement just to figure out what's going on. And these, these MIT students asked me, how do we do that? Like, if you have even taken an introductory programming class, like, you know, how to write a debug print statement. So this is very concerning to me. So. Okay, but first you could say, oh, well, that was just one group. Okay. Then another group came up to me and another group and another group, the exact same. And these are people, some in grad school, I mean, you know, who clearly had at least, you know, done passively well in college level classes with that kind of, you know, ability. And so Liberty, to your, to your question. I think this is the concern that I have is that in terms of what I actually think that in the near to medium term we're doing is deskilling, or at least we're not allowing this, you know, core skills to develop in a whole class of students and workers. Then again, in the future, I think this will be corrected with better designs and other things, but I think in the near term it's going to be a really big problem. And so I don't think that this means that the technology itself is going to like, replace the jobs, but I do think that there's going to be an awful lot of people who might be hired for jobs that on paper they look qualified for, but they will be very unqualified. And we're going to have some really big problems in the, you know, in the medium to long term, actually, for this cohort. I don't think it's exactly your question, but it is something that I'm extremely concerned about from the workforce perspective, actually, even more than a lot of the, like we automate away all the jobs.
- [19:12]
C
Yeah, I mean, I think that there have been some recent papers that have shown this decline in hires among the youngest cohorts and a corresponding increase for specific occupations that are more exposed to AI and increases in older, more experienced workers. So understanding what's going on, I think is relevant. I don't know if it's an issue. I actually don't think that it's an issue of AI already substituting for these workers, because as Ben was saying, at least as far as I know, for many jobs, we are still very far from where you have entire jobs that are substituted and automated away. But what might be going on, which is entirely consistent with Ben, is that this is a technology AI that is most productive when you have contextual knowledge. And so you put AI in the hands of an engineer that knows the debug statement and you can do a lot of cool stuff. It could save time, it could maybe increase creativity, who knows? But that person would also be able to understand when is it that AI is making a mistake or maybe go inside the code to understand exactly how to use it. And I have a friend, he has an AI company who was telling me, look, my senior engineers are very concerned about hiring young people not because they believe that their jobs can be automated, but because if I give AI to an experienced person, they can destroy the code for the whole organization because they actually don't have these foundational skills that are needed to make good use of this technology. But then the implication, and I very, I'm very sympathetic with Ben and the problem that we have to figure out is how do we bring young people to the point at which they can make the best use of this technology or vice versa? How do we think about a way to, to be able to nurture people inside organizations knowing that there is a technology that eventually will be very helpful for them? And that's a much more complicated argument than just saying there is no jobs for young people.
- [21:20]
D
Well, looks like the AI we used had anticipated your answer. So here's one. And again, I love the ideas that we humans say, hey, that does not make sense at all. Okay, let's see if this one makes sense. If you were advising a student or mid career professional today, what specific skills or ways of working would you say are most future proof in an AI driven labor market?
- [21:47]
B
All right, I also don't, I don't really know. I think I always like, I mean, listen, I always like, I think statistics is always useful, but I don't know if that future proofs it. It's more like, I think, I think having a statistics, at least a basic grounding in statistics is helpful because it helps folks interrogate claims that people make about whether that's technologies, whether that's about improvements more broadly. I think that's helpful. Whether that's specific to, because we have more AI introduced. Like, I don't know. I mean, I do think that in general, like what are people really good at? People are really good at being creative, coming up with new ideas and communicating with other humans. We're very, very good at that. You know, the ability to collaborate and build relationships effectively with other people is always useful. I mean, that's all I got, which is not very sexy.
- [22:31]
C
But I Don't think that I've got much more, if that's any consolation. I think one piece, however, that I would really push for if somebody asked me that question, and I keep asking this question at education experts and nobody really has one answer. There's a lot of uncertainty and we just have to recognize that. But what I would prioritize is getting access to organizations that give you the permission to make decisions, to develop experience, to develop that contextual knowledge that I think will be really, really critical going forward. It's like, how do I apply the technology? How can the technology help me? Perhaps where we will go is not so much the idea that you do college and you're a fully formed, skilled person, but actually maybe we have to consider ourselves more as apprentices and that college experience is not going to be where we are fully formed. It usually isn't. Anyway, we learn a lot on the job. But I think it's going to be even more important to think about giving students time to get experience inside the real organizations and making real decisions. Because my suspicion is that that type of contextual knowledge will become very, very relevant to distinguish yourself from, from others.
- [23:53]
A
Rafael, your research seems, according to ChatGPT, emphasize management quality as sort of a real key driver in productivity. So how specifically will AI change what good management looks like? Because I feel like we've talked a lot about, or people are talking about AI a lot for jobs for people under 30 or sort of these jobs that aren't what we would call management. So how could management really use this?
- [24:23]
C
Yeah, so look, I mean, going back to what we discussed at the very beginning, think about AI as for companies is like taking a walk on the J curve. Okay? Now at that period when you experiment and when you have to figure out what to do, that is typically not a good time to put very specific and quantitative measurement or target revenues and so forth. So my sense is what you do and what's considered optimal depend very much on where you are in that stages of adoption. I think that at the early stages of adoption, it's more about leadership, giving people a direction. And I insist on the strategic direction, knowing more or less where we are going and giving people space to experiment and learn from this experimentation. Once you are over that and you have some understanding of how value can be created, that's where management and the, the creation of KPIs, the creation of metrics, monitoring how things are used, that could be helpful for the implementation stage. I think that there are different stages of adoption. One that is more uncertain and one that could be more certain and more about scaling things that we've figured out. And for each of these stages, you might need very different types of different attention to different things and perhaps different people too.
- [25:53]
D
Well, Ben, you have studied, according to ChatGPT, how behavior data inside companies reviews hidden patterns in collaboration. How might AI accelerate or even automate organizational decision making based on these kinds of insights?
- [26:09]
B
Yeah. So I think that again, it more or less got the gist. I think this is the thing, it's sort of like a horoscope, is that you can say vague things that are mostly right. It's like fine and then we're filling in the blanks and it's fine. But no, I mean, I think that I am very bullish on using large language models for data labeling, which is importantly different than summarization. Right. But I do think that if in the past I would have paid people on like Mechanical Turk to like label, like, is this email happy or sad or something like that, can. I'm quite convinced that one can fine tune an LLM to do that. And I think there's some very interesting things that one can do with that. I think in the past, a lot of my work had been on looking at patterns of collaboration and how that relates to outcomes. For a lot of reasons, looking at things like content or other different types of patterns was a lot less predictive. But there was obviously a pretty hard limitation on some of the approaches one could use to try to characterize that kind of data. And I think there's some very interesting things that can be done on the labeling side. I mean, I really think actually that's where the vast majority of value from generative AI technology, at least in areas where the truth matters, I think creative tasks actually is a different thing. But in areas where the truth matters, I actually think that application is going to be the biggest one. It's probably going to be a lot smaller than what a lot of people are saying in terms of automating jobs or whatever. But I do think that even as Raffaello was saying, really making use of that kind of capability does require really rethinking how one designs processes within organizations, what sort of applications one has. And so there is some new work in that area that's just getting started. But that's sort of what's interesting to me.
- [27:55]
D
ChatGPT was very smart here. It said the question spotlights Ben's expertise, then invite Rafaeli to connect back to organizational structures. It gives us the kind of a way to do it. So I don't know. Rafaeli, do you want to say a few words in terms of the whole structure, the organization structures?
- [28:14]
C
Oh, yeah. I mean, yeah. So caveat is I think there are different models right now and, you know, companies are figuring it out. I think the optimal organizational structure to implement AI doesn't exist yet, or at least I haven't seen it. What I believe might be, you know, connected with what Ben was saying is one talent that will be really necessary here is to identify who, who helps you learn at this moment in time. And this is not trivial in the sense that what I'm seeing also in very tech advanced organizations is that people with very strong technical expertise may not be the best people to help you understand how to creatively use a technology for a new source, for a new way of creating value. Because they have been trained in a certain way, they've been using technology in a certain way for their whole time, they might have some justified reluctance to, to change what they're doing. And so I think that one specific piece that that organization you need to figure out is who do you recruit at the very beginning of your adoption to understand what's technically feasible? And then who do you recruit to help understand what are the potential use cases? And typically one possibility would be to think about skunk teams, people who are, you know, far from the established organization so that they can invent. And then lastly is again, how do you bring back this knowledge so that it gets the funding that it needs, it gets implemented, it gets scaled. And so organizationally, as you can see, there isn't just one model. I think it's really understanding how do you position yourself to be able to learn and select what's good for you and then enable the organization to adopt?
- [30:09]
A
Rafael, you were talking earlier about how we're going to see some companies really adopt this, some not do anything at all. And, you know, this sort of widespread. And so AI can really, I guess, sort of widen the productivity gap, not just in companies or in, you know, sectors, but also between countries. And I think that's one of the big questions. I mean, what do you see as the risks going forward for something like that?
- [30:38]
C
Oh, I'm seeing what I've seen 20 years ago with my first PhD, the first chapter of my PhD dissertation, and it was about the adoption of computers in organizations. So the famous Solow paradox. We see computers everywhere except the productivity statistics. We were talking about this stuff more than 20 years ago, 30 years ago, and it turns out that was stuff that I studied and many Others people studied too that the productivity impact of computers was really dependent on the type of organizational managerial practices that companies were using. In particular, this ability to be flexible about the ways in which you were allocating. You were thinking about talent and the ways in which you were allocating talent. Give you one concrete example that again, has been studied a lot. The introduction of ATMs in banks created the very different ways. You know, these people that now greet you at the entrance of the bank, that was the substitution of the cashier and the introduction of new jobs that were much more based on human skills, skills and not so much in these routine jobs. So that's the parallel. That's why I'm sort of bullish on the idea that we will see a ton of heterogeneity across firms. And that's why I also think that one potential risk is that you will end up seeing heterogeneity also at the macro level. And in particular, you can hear from my accent, I'm not British and I'm not American. I'm actually Italian. And one concern I have for countries such as Italy, which are already behind from a digital adoption perspective, is that we will not catch up fast enough. Italy and other parts of continental Europe were already, and are already behind in terms of digital skills for basic applications and may risk not being agile enough to get into this continuous experimentation mode that I told you. And therefore they may be passing on opportunities that this technology may provide to them. Think about the importance of manufacturing in some European countries. This could be fatal. It could be really potentially very problematic. And in fact, the adoption gap, I think, for Europe is potentially much more problematic than the innovation gap or the creation of new LLMs or new AI companies. So I see this as a real risk. And I think my hope is that there will be a way to spur experimentation much more, not just in the US but also in other parts of the world.
- [33:17]
D
So we have been talking about the human machine interaction. I think, what do we call a human computer interaction? That's a buzzword. So here's a question from ChatGPT. What does an ideal model of collaboration between humans and AI look like? There are examples today that give you optimism about how that relationship might evolve. Maybe. Ben, you go first.
- [33:42]
B
Sure. So I also think, like, I think that is a flawed. Like that's not the correct model. It's not a person, it's not. Not an agent. It's like we used to call things computer programs and now we call them agents. It's the same thing like that's it just, it sounds fancier like it's a tool. Right. And the question is, how does one use it effectively? Right, but you don't say, I'm collaborating with a calculator and collaborating with Excel, like that doesn't make sense. And so organizations that then, you know, implement it this point, I mean, to the extent they're actually experimenting with it and actually measuring outcomes, you know, to the extent they do use it incorrectly, hopefully they would have metrics in place to understand that they're not getting the outcomes that they want. I do have a very large concern that most organizations do not have that. And the problem is they look at things that are very easy to measure, that are short term. Right. So I know a very large technology company that shall remain nameless has mostly replaced their HR department with large language models. Okay, but again, that doesn't mean that tomorrow they lose a billion dollars. Right? And again, in terms of saving time, right? Like just like writing the questions here, like you all save time by writing those questions, which was, you know, whatever, it's fine. This is not like a big deal. But now let's say a large language model that writes the employee handbook for the company, right? 50 pages, let's say now again, should a person read through those 50 pages and like check that everything's correct? Like, yeah, but listen, people are people. They're not going to do that. That's the whole reason you use this is to save yourself time. And so now let's say there's not a section in there about sexual harassment because most of these systems are fine tuned to avoid sexual content. That doesn't mean that tomorrow you lose a billion dollars, but it could mean that in a couple of years something horrible happens, right? And then you get sued for hundreds of millions of dollars. Now, are you going to be able to go back and say, oh, that's because we used a large language model. No, you're not. That systemic risk is still in there. And so I think part of this though, gets to how one thinks about using these tools. It's about being both a good programmer as well as having a good mental model for how these tools work. And the problem is the mental model that most vendors and even a lot of academics are promulgating of these tools is as people, but they're not people. And that's a problem because the way you would treat a person is just so fundamentally different from how I would use an Excel spreadsheet. But I think having an appropriate mental model is going to be important and I think the mental model that I would suggest folks have of these tools is it's like a cookbook written by random people on the Internet. Okay, so you would never say that a cookbook knows how to make chocolate cake. Like that doesn't make sense. Like the information is there, but also if you don't know how to make a chocolate cake. So let's say I turn to the recipe page for chocolate cake and I look at it. If I already know how to make a chocolate cake, I could discern. Is this a good recipe? Probably. But let's say I haven't made it before, right? Now this recipe could have been written by a great baker, but maybe it's written by someone who doesn't know what the hell they're doing and it's actually terrible. So I need to have the ability to actually discern that. But beyond that, now let's say I bake the cake but I don't even taste it myself. I give it to someone else and I don't ask them what they think. And so this gets to the way that you should bring these tools, especially as currently constructed, into your workflows. Very much is going to require both people who can discern that those outputs are like appropriate, that you have an understanding of actually how these tools function. And I worry that rather than doing this in a methodical, experimental way, I think most companies are trying to jam these in very quickly in a non thinking way again without having metrics in place. And I think that's going to have big problems. I'm not actually sure that I really answered the question, but I at least answered the question that I wanted to answer. So that's hopefully.
- [37:22]
D
Okay, that's what a human does. So it's good.
- [37:26]
B
There we go. Exactly.
- [37:27]
D
And Rafaelic, do you have any responses?
- [37:30]
C
Yeah. So your question is about the optimal human computer interaction. Right. So where should we go? Yeah, so I think that that's where what I was saying before comes back and it connects also with what Ben just said. I think that first of all, so it's hard for me to understand or to evaluate something that is taken off the shelf and just introduced in an organization. Because there are many things that are very firm, specific about what is a good output. And I think that the way the good human computer interaction starts with the design of the technology, which I think if we believe in that promise that there is some expertise that can be disseminated and leveraged by others, you have to, to spend quite a bit of time thinking about and training the machine and making sure that it does have a good model of the expertise and it has some guardrails that allow you that prevent things from going really badly. I think that that's where you start. And then the second piece is human computer interaction. We need to understand interaction. What does it mean for a technology to add value? So first of all, you have to have hypothesis of how value might be created. And these hypotheses are necessary to form an experiment. And then you need to be able to measure and to have a good understanding of whether things went according to your hypothesis or not. And so what I'm getting with that is that I think that human computer interaction is one piece technology, but a lot of it is actually how you train the technology. And an even more important piece is how do you learn how this technology helps your people. And so I know that Ben really believes in that and I believe in that too. Companies and organizations in general, you need to get much, much better at becoming experimenters. This means formulating the hypothesis, monitoring, measuring, learning, and doing this continuously, not just once. Because that's I think at the basis of creating a productive human computer interaction.
- [39:47]
D
Thank you to both of you so very much. And I'm sure we could have asked ChatGPT to generate another two hours questions, but we need to wrap up. And as Liberty and I said, we were trying to experimenting here using AI to generating all these questions, but we do have one coming from us. That's the magic wand question. So if you could wave a magic wand and put the AI genie back into the bottle, so to speak, which part of would you put back?
- [40:20]
B
Okay, so it'll be a very specific thing. Well, I would, I don't know if this is putting it back in the bottle, but I would make it so that systems could not respond with terms like I or say. I'm thinking. Yeah, I mean it would also not have a text box interface because having an Omni tool makes no sense. Right. Like why do we have one thing to try to do every. Like that's just, you know, that's a minimal ask. There's probably bigger things that I could ask that would take way too long to expound on. But that's maybe my, my, maybe not so radical ask. I think it's still doable. Like we could just, even Rafael said this, we could choose to do that tomorrow. We're choosing not to, but we have full control over what we're building.
- [41:03]
C
Yeah, I mean, it's a tricky one. I'm going to go in a different place. I think I am intrigued at the same time scared by the ability of AI, of making people feel heard. And I think that that's like, it's not so much putting it inside because I can see how this could actually help with loneliness or, you know, some aspects of solitude. But I'm very, very scared because at some point when we've seen it, unfortunately there is a limit where you really need a human. And right now that threshold of helping people feel a little bit less lonely, but you know, really knowing when is it that that they need another human being to be with them, there is no clarity. I think that the machine is not good there. And I think that that's actually something that concerns me because I see a lot of loneliness around. And if only we could use this technology to help alleviate this loneliness, but also to help people get the human help when is needed, I think we would have. It would be terrific. But I don't know if that's where we are going.
- [42:19]
D
Thank you again to both of you for this really insightful conversation and normally for HDSR podcast, because we're facing, you know, the other data science community, we will have a question more about data science, but this is the AI generated. We didn't prompt it, so it doesn't have have much in it. But I want to emphasize that as a statistician, I was so pleased to hear Ben, your emphasize your emphasis about everyone should know something about statistics. And also Rafael, your emphasis on everything started about thinking about hypothesis, you know, then experiment, then measurement. These are very much things we teach our students about how to do research, how to think about, you know, any problem. So I appreciate that. And with that, I want to close again by thanking both of you and thank you so much. And Liberty, I know you are with your little baby, the future workforce.
- [43:17]
A
So I know his grandma's like a green or not a green.
- [43:22]
D
That's the ultimate test is there. So again, thanks everyone.
- [43:29]
A
I'm Liberty Vitter Capito. And on behalf of Shauley Meng and our guests, thank you for joining us. And a special thanks to our producers, Rebecca McLeod and Tina Toby Mack. This was the Harvard Data Science Review. Everything Data Science and data science for everyone.