← Artificial Intelligence Masterclass
Artificial Intelligence Masterclass
Metaprompting with GPT-3 to dynamically generate arguments | Artificial Intelligence Masterclass
00:55:04
Loading summary
Transcript4 lines
- [00:00]
A
Study and play come together on a Windows 11 PC. And for a limited time, college students get the best of both worlds. Get the unreal college deal everything you need to study and play with select Windows 11 PCs. Eligible students get a year of Microsoft 365 Premium and a year of Xbox Game Pass ultimate with a custom color Xbox wireless controller. Learn more@windows.com studentoffer while supplies last ends June 30th terms at aka mscollegepc.
- [00:31]
B
Happy Monday morning everybody. David Shapiro here with another video. Let me go ahead and introduce today's topic. So I posted on the OpenAI forum, just say, hey, bring me your research and business problems and I'll see what I can do. And so Anna R. Mills is an expert in constructing arguments and asked me to take a look at doing this with GPT3. So I'll show you the data in a second. But the most important thing is the instructions that she gives her, I guess, students. And it's basically teaching them how to construct critical arguments against something. So write a thorough summary and critical assessment of the argument. The summary should describe the key ideas of the argument, including the main claim, key reasons, counter arguments, rebuttals, and limits. The assessment should discuss the strengths and weaknesses of the argument. What was compelling, persuasive, troubling, unclear, or problematic. Choose phrases like elegant argues throughout to show that the writer's purpose to show the writer's purpose at each point, write most of the essay in your own words, but consider using the occasional direct quote where original word choice is critical. The introductory paragraph should include the title of the argument, the author's full name, the argument's main claim, and your overall assessment of the argument's validity. Okay, so this might sound like one task, but this is actually many, many, many tasks. And so let me show you some of the data that she sent over to show you what I mean. And so this is, this is one reason that a lot of people think that, like, oh, GPT3 is really stupid, because you ask it to do, like, all these things at once and it only does one of the things. And that is because what we need to do is we need to break these things down into separate cognitive tasks. So here's the instructions that she gives. And really, this is a procedure, right? It says do this, this, this, this is how it should look. GPT3 will be able to get some of this. But let me show you an example of one of the inputs. So here's one of the, here's one of the essays. It is 1600 characters long. And then we go get the, the critique. Look how long the critique is. The critique is way longer than the, than the, than the actual essay. It's 5,800 characters long. Okay, so that's, that's a lot of critique. Unfortunately, this is probably a little bit too long to fit in, in a single prompt, which is another reason this. So let's see how many tokens this is. Oh, only 1500 tokens. Okay, never mind. So this could actually fit in original DaVinci. However, what we're doing here is, is basically I'm going to break it down into prompt chaining. So with all that said, we are going to. I've already got the repo created, so let's go ahead and clone this down and let's get clone. Yes, I fat fingered that. There we go. Okay, so then we've got our brand new repo here. Critical argument. Yep, that looks good. And then we'll open another one. Increasingly verbose bot copy my git, ignore and OpenAI key. Okay. And then increasingly verbose, just go ahead and copy a script over. I'll rename it paste. Okay, so this will be Critical Argument. I'll just call it Critical Argument. So what we're going to do within this though is we're going to set up prompt chaining. So the first thing we need to do is let's look at the instructions. Write a thorough summary and critical assessment of the argument. That is the highest order instruction. And then this describes the summary. The summary should describe the key ideas, including the main, main claim, blah, blah, blah. Okay, so we'll say this is, this is subordinate to the first instruction. Right? Let's see. The assessment should discuss the strengths and weaknesses of the argument. What was compelling, persuasive, troubling, clear and problematic throughout. Okay, so the throughout is. Global instructions. Write most of the essay in your own words. Also global instructions. Let's see. The introductory paragraph should include the title of the argument, main claim in your overall. Okay. Okay. So basically what I'm doing here is I kind of chopped up the instructions to show that it's like, let's see that it, like it's, it's separate tasks. I don't, I don't think I'm wording this correctly, but I hope that by showing you. So like there's an order of operations. Basically the introductory paragraph, right, should include this. Choose phrases like elegant, argues throughout to show the writer's purpose. Actually, I wonder if this should be subordinate because this first instruction is kind of the global Purpose, right? That is what this paper is, right? Most of the essay in your own words. Okay, yeah, I think that's good. And then. Yeah, okay, so the reason that I'm chopping it up this way is just to show how the mind, how your mind might organize this. Because the instructions that she gives kind of talk about this part and this part and this part. But what, what happens in your mind as you build it up is you're kind of constructing an image. So our brains are kind of like object oriented coding where you know, it's like, okay, so you start with this primary idea and then you attach a new feature to it or a new, or a new characteristic and then you modify that characteristic and then you add a new, you know, value or whatever or a new function to it. And so you keep operating on this object in your mind, which is constructing this argument. Now before we do that, let's just do an experiment because right now I'm operating off of intuition. So it's like, what if I'm wrong? Right? What if, what if GPT3 can handle, can handle this just sight unseen. So let's go and test that real quick. So write a thorough summary and critical assessment and then argument actually here. Now we don't even need that. Just go grab it. Sh. Critique. Add some more new lines. Okay, critic, let's see. We'll say summary and critical assessment. Now this is DaVinci02. So, okay, not bad. May contain sensitive content. Alright, so in Wouldn't We All Cross the border? Yep. Anna Mills argues that illegal immigration is not inherently wrong or unethical. Mills points out that if most people or in a situation where their children are starving and they had a chance to get to the US they would take it regardless of the legal consequences. Therefore, Mills concludes that we must find a border policy that treats migrants with empathy and respect. This is a compelling argument that highlights the human side of illegal immigration. Oh, that's still really hot. Mills effectively uses hypothetical situations to illustrate her point that most people would do whatever it takes to get their families to safety. However, the argument does not address some of the more practical concerns around illegal immigration, such as national security. Overall, this is a thought provoking piece that challenges readers to consider the issue from a different perspective. I would say that is a phenomenal opening paragraph. However, let me show you how long this is in comparison. And part of the reason that it's shorter is because we're using text davinci02 so text davinci02 is fine tuned on the instruct series data. So it's basically saying, okay, I'm going to put out, you know, one sentence, a couple paragraphs. Sometimes you get run on stuff. But with DaVinci O2, its output is generally very short, especially compared to some other ones. So this is, this is what GPT3 put out. And then this was what a professional wrote. I don't know if Anna wrote it or if, if a student wrote it. But you see like, okay, so we've got, let's see, 906 character response versus 5,800 character response. So we're not quite there. But the quality of this response, phenomenal, right? This, this, I would say like the construction of this, it is very obvious that someone is responding, that this author is responding directly to this essay. Whereas you say like, you know, the, the response title was Contested Territory. Okay, that's not even the same title, right? In recent years, like someone is just kind of summarizing it. But honestly, this, I don't, I don't think that this is as good of an opening because someone is like, whoever wrote this, they're not actually saying that this is what Anna Mills claims. They're just stating this as a fact. And so it's not clear that this is actually a critical response just from the get go. So that's my personal opinion. I'm not an expert in this. This is just, I'm reading this and if this was an art, like if I saw this as like a blog and someone was like, you know, critiquing someone else's blog, I'd be like, what are you, who are you? Like contested Territory. Okay, I would search for contested territory. Like, is that what you're responding to? The second paragraph seems spot on. Mills questions the ethics of enforcing immigration laws. That was, you know, GPT3 picked up on that as well. While the essay argues for empathy, it limits the scope of its argument from offering comprehensive vision. Okay, so we've got all the different components. So for the opening paragraph, it seems like if we just use this whole thing, that's fine, but what I'm wondering is if we can break it up into sections. So first let's just copy this prompt down because it worked. So we'll do essay summary and critical assessment. Okay, so we'll save this as prompt 01. So we'll use this to generate the opening paragraph. Let's run it again just to see if it produces something similar. So I'm guilty of this. Sometimes, like I see one good output and I'm like, oh, it's good. This it almost identical. Okay. Yeah. So the output is darn near identical. Let's turn up the temperature just to see if it gets really weird. So one thing is that the quality of the instructions, you know, you see, like, there's a whole paragraph of instructions. Whoops. That actually can sometimes help. Sometimes if the instructions are not clear or if you're trying to do too many things, it'll get confused. Okay, so let's see. Anna Mills challenges the morality. Yeah. She appeals to the reader's empathy, asking to them to imagine strengths is lioness emotional appeal and examine examination of moral implications. Yeah, okay, so we've got that. But because we. So there's a few. There's a few, like, universal rules. Right. So what we want to do is break it down into smaller parts because, you know, I think we do have weakness. Oh, nope. It didn't address the weaknesses. So here's what I typically do when I want to deconstruct something is I'll break it down into smaller and smaller parts. And so what I'll do is let's. Well, here, let me make sure I've got a copy of the instructions. No. Okay, got the instructions here. And let's close that. We don't need that. Here's where I deconstructed it. I actually don't like that. Let me close that. I made a mess of it. Don't need that. Okay, we'll get back to the script once we do some prompt engineering. Yes. So the key ideas, including the main claim, key reasons, counter arguments, rebuttals, and limits. All right, Let me now copy all this into a new prompt. And all we're going to do is we're going to. We're going to pare this down. So we want to summarize. Well, here, I'll borrow the same language that Annie is. So we'll say. Write a summary of the following essay. While focusing on the counter arguments, rebuttals, and limits. Assess the strengths and weaknesses of the argument. Okay, So now let's take this and we're going to assessment of strengths and weaknesses. And if it's not apparent yet what I'm doing. Oh, here, let's just grab this. Yeah, we can grab the whole thing and then we'll go grab the. Oops, I closed it, didn't I? Go back here. So by breaking it down into smaller parts, I'm basically going to say, okay, write each section separately. So let's see what this does. Let's see if this works. It might not. Let's turn this back down to zero.
- [16:29]
C
Some follow the noise. Bloomberg follows the money. Whether it's the funds fueling AI or crypto's trillion dollar swings, there's a money side every story. Get the money side of the story. Subscribe now@bloomberg.com.
- [16:49]
B
And here we go. So this is. This is a little bit better. The essay's weakness is that it does not offer any specific solutions to the problems. The author admits that open borders would be dangerous, but does not say how we could regulate the border without criminalizing immigrants. This lack of specificity leaves the reader wondering what exactly the author is advocating for. Not bad. So I pointed out on the. On the forum that GPT3 was trained on a lot of Reddit data. So it's really good at constructing arguments. It's frustratingly good. Let me run this again just to make sure that it's consistent in its performance. Let's see. It also does not consider not fleeing difficult circumstances, but are instead seeking economic opportunities. Not bad. Okay, let's do thorough assessment. Let's add thorough back in because I'm wondering if we can get a little bit more. We can squeeze a little bit more juice out of this. Because if it writes one paragraph. Oh, here we go. Oh, much better. Okay. Yeah, yeah. Okay. All right. So this will, this will looks like it'll reliably generate two to three paragraphs. Sometimes one. I really like it, though. So we'll update our prompt. Whoops. Where is the prompt? Go ahead and save this as prompt. Oh, two. Because this is like kind of the second part of the paragraph. Oh, hang on, my dog's outside. Gotta let him in. Okay, I'm back. Yes. So write a summary. Write a thorough summary. So it all comes down to right word choice to get the. To get the desired effect. So the prompt 01. This will generate our introductory paragraph. We remove that verbiage from this one where we say, what was compelling, persuasive, troubling. And actually, so it did a really good job of doing strengths and weaknesses. Let's actually separate prompt O2 out into a third prompt and we'll. We'll do so like this. This Prompto 2 is going to be strengths and weaknesses. So let's focus on the compelling, persuasive, troubling, unclear, or problematic parts of this one. So if we do thorough assessment of. Let's see what was compelling, persuasive, troubling, unclear. Oh, whoops. Crap. Crap. No, I'm modifying the wrong one. Copy that. Go here. Whoops. Okay. Sorry. My brains are a little bit like scrambled eggs this morning, apparently. Prompt. Oh, three. Okay, so let's leave, let's leave the original prompt as it is. But then here we'll just, we'll just copy this and do it all caps. So someone asked why do I do caps? And that is because I think I've explained it before, but just in case you haven't heard, a caps lock version of a word looks different to GPT3. It uses a different set of tokens. And so GPT3 will see like, okay, this is like. And it looks like a block of instructions or something. Okay, so let's just change this and let's see if it. Let's see how much just changing the final kind of thing, you know, final instruction changes the nature of the output. Well, that was short. Maybe, maybe, maybe my intuition here was wrong. Okay, this is pretty similar. All right, I think, I think my intuition was wrong. I don't think. I think prompt two is fine as it is. It looks like it got most of it in there. So let's go back to the original problematic. Choose phrases like, okay, word choice is critical. Introductory. Okay, well, I mean, Let's just, let's just start here with prompt one and two and just kind of run it a couple times and see what's missing. Because my brain is not giving me clear, like, signals as to like, okay, what is the next cognitive task that we're missing here? Okay, so I've got this. Let's see. I don't think we need to do chunks in this one. We'll just say text equals open file, input dot text. So that means we need to have an actual input. So we'll do essay to critique and we'll save this as input text. All right? And then result equals list. So what I, what I often do when I have, like when I have prompt chaining is I'll do. I'll do a variable called like prompt files equals. And then I'll do, you know, prompt01.txt and then prompt o2.txt and actually, there's an even easier way since I'm following the same format. Make sure I import OS. So I'll do prompt files equals OS.lister do the current directory and then actually put this in a list comprehension. So this is a fun thing I for I in OS lister, where no if. If prompt in I. So what this does. So Python has this fun thing called list comprehension where you can actually declare and, and populate a list in one go. And so I for I, you Would see you would recognize this as instantiating a for loop. So it's, it's. So the variable name is going to be I and the for loop is for I N O S Lister. So it's going to list all the files in this directory and then it will include it, it'll include I. So it'll include the file name if prompt is in the file name. So this is a fun little one liner that you can do with Python. So then we'll do for file in prompt files. So we'll just iterate through them and then we'll do prompt equals open file. So we'll actually just replace this with I because I is now the file name and do essay and then we'll replace because we don't have any chunks anymore. So then we'll say argument equals GPT3 completion from the prompt. So we've populated the prompt and then the result is we're just going to accumulate it all together, right? And so we'll replace this with argument argument and then once it's done we can get rid of all this. We're just going to save the file and we're going to have a double new line between each one and we're going to save it as output. Txt. So this is pretty compact. And so again what this is called is called prompt chaining where it's like sometimes you take the output of a previous one. I did this very extensively with like the summarizing and novel writing one. But when you're constructing like especially nonfiction, where it's just kind of like one idea after another or one kind of dissection after another, it's much, much more procedural than especially than fiction. Because each passage or paragraph in a nonfiction essay, particularly if it's an argument where you're skewering one essay, each paragraph can kind of stand on its own. Okay, so let's just run this real quick. What is this? Okay, I don't need that. Yeah, and we'll actually, we'll leave that in just so that you can see that it'll, it'll, it'll stick three arguments together. Okay, so CD GPT three critical critical argument Python critical argument Py. Well, that was fast. Something is amiss. Did it generate an output? No, hold on. Hold the phone. I bet I did something wrong. I wonder, I wonder if my, I wonder if that was it. Print prompt files. Am I not getting anything? Mmm, I have done something wrong. Duplicate. I know I fat fingered the syntax somewhere. So let's just say os, Lister and Then we'll add an exit right after that. Just. Okay. Yep. Okay, so Lister works. Oh, wait. Hey, there we go. Okay. Just Lister with nothing, with no arguments. That's how you. That's what I did. I is not defined. What do you mean, I is not defined? Oh, derp. I instantiated it. Here. This is I. So typically, here's. Here's what I did wrong. Typically for shorthand is you'll say you'll use I in a. In a. In an iterative iterable variable in Python. So it's for I in iterable thing. And then I copy that instead of switching it to file. Sometimes you. Sometimes you be like for I and prompt files. That's bad practice, though, because if you reuse the same variable name over and over again. I've run. I've run a foul of that before. Okay. Figured it out. That's what I did wrong. Do, do, do, do, do, do. So let's see what it generates. No such file or directory. Ah, darn it. I forgot to create my gpt3 logs. Always something GPT. Third or fourth time's a charm. Okay, and then what we'll do is I'll post this back on the forum and see what she says. Okay, so now we've got the. Now we've got the output populated. Reload it. Yes. Okay, so it's only 2,000 characters long, but that's not so bad. Anna Mills argues that illegal immigration is not unethical. Migrants with empathy and respect. Okay, didn't use the title in this one, though. All right, so some stuff is starting to crystallize in my head. Let me make sure I get a copy of the original instructions. And ideally, what we would do is actually have GPT3 generate the prompts that it needs. So that is going to be a big thing in the future with artificial cognition is what's called cognitive control. But we'll get into that at another time. Okay, so let's change this. We're going to write the opening. Okay. Right. The. Let's see. No. Oh, this is a tough one. Oh, oh, oh. Okay, I know what I should do. I'm gonna pause it for a second because I need to do a little bit of tinkering. Okay, we're back. This video, I realize, probably seems like it's a little bit rambling and unfocused, and I'm taking a risk here because this is not a problem that I've thought about a lot. And. And I just went in cold. So I'm showing you my Whole process of like, okay, let me take kind of initial stab at it and see what happens. And then we kind of pull it apart and then go at it again. So what I've done is rather than me design the argument, I'm asking GPT3 now to design the argument. So what I've done is I've written a prompt that says, read the following argument and brainstorm the outline of a critical assessment. Write a thorough list of the key points in the essay with some critical evaluation of each. Focus on constructing a thorough evaluation and deconstruction of the argument presented below. So then I said, thorough, brainstorm outline points. Yeah, so that's that. You know, just. Let's see. Let's take thorough outline of points. And I've ran. I've run this a couple times and it's pretty consistent. There we go. Oh, much better. Okay, so then what we do. So this is, this is cognitive control. So what we're doing is we're asking first for GPT3 to plan its argument. Let's run this one more time just to be sure. This is writing first person. Okay. I don't like how it did that. We can also increase the maximum length. Let's see, let's say, describe the points in your own words. Yeah, there we go. The author argues. Okay. All right, cool. So let's copy this. This will be our new first prompt. Let's just nuke this one. And I know some of you will be like, ah, no, save the thing. Essay, thorough outline of points. Okay, so this will be prompt 01 and then prompt oh, two. We will keep. We will keep some of the original instructions. And so then we will. But, but, but what we will do is we will ask to focus on a specific point. Okay. Write a thorough summary and critical assessment of the argument. The summary should describe the key ideas. We've already got that. The summary should focus on the following. Point. And so then we'll do point. And so what we'll do is we'll. We'll copy paste the output one of these items. So she poses the question of whether or not illegal immigration is actually wrong or unethical. And so then we'll say, let's see, We'll put this in quotes. That should be fine. Let's see your assessment of the argument. Your assessment of. Let's say of this point, of this point should. And we'll go down to here, discuss the strengths and weaknesses of the argument. What was compelling, persuasive, troubling, unclear, problematic? Choose phrases like throughout to show the writer's purpose. Write most of the essay in your own words, but consider using the occasional quote. There we go. And then we can remove the introductory paragraph. Actually, I think we need to bring that back. So this will actually be like. Yeah, okay, I know I'm jumping all over the place and I promise there is method to my madness and I'm going to give these better names. Okay, so we'll start this with essay and then we'll do. We'll go back to kind of what I originally had, which was, I think it was something like critical assessment and evaluation. Thorough. Okay, that's probably good enough. Okay, so this, this will be prompt intro. And this one we're going to rename to prompt Brainstorm. So I can tell what the heck it's doing. Prompt brainstorm. And then. No, don't keep it in the editor. This one is going to be each point. So prompt point. Whoops, I've already got it open. That's fine. Close all those prompt point. And then we probably need a conclusion. Let's. Let's not get ahead of ourselves because we're already breaking this down into a meta prompt, which is the cool thing. Okay, so let's go back into here. So now we can't just enumerate all the prompts and just run them in order because that won't make any sense. So we'll do prompt files equals. Actually this is going to be super procedural. So we have to just do this one at a time. Alright, so write intro equals prompt. So what did I name this? It's prompt intro dot text. And then we will do. Argument equals GPT3 completion prompt. Okay, and then we will do. Actually here we can just shorten this. Do result dot append. Okay. And then let's do the brainstorm. Brainstorm the rest of my essay. Actually, we'll say critique to use the right terminology. Okay. And so then we do. We do this and we do brainstorm. And so in this case we'll do key points equals. Dot split lines. Okay, so what we're doing here is when GPT3 sends this back, we're going to split it into lines. So that way then we can enumerate for each line. Right. For each point. Let's see. And I think we need to make them standalone. That's not right. Okay, this is better. Yeah, so this is okay when it says the author believes. So what I'll do is I'll add the author to the beginning of the prompt. So what? Especially. This works especially well with Davinci Instruct it. If you want a list of things and you want it to follow a particular format, you just kind of start it. And it's like, okay, the author does what? And then it'll follow that pattern. So then what we do is we just add this back. The author said something to the beginning of the first one. Yep, that should be fine. Key points. So then we'll say key points zero equals key point. Actually, we'll say the author plus key points zero. And the reason that I'm doing this is because this is going to get cut off. It's not going to be part of the response. So we want each. Each bullet point to be the same. Okay. Four point in key points. Now we get back into kind of this for loop prompt equals open file. And we're going to use the prompt. What did I call it? The prompt point. I come up with such original names, prompt point, dot, text, replace essay. And then the argument is going to be GPT3 completion. We'll do that. Yeah. Okay, let's run this. I think. I think you guys will be surprised by this. So basically what I've done is rather than me construct the argument, I'm asking GPT3 to construct the entire argument. Because all I did was it said brainstorm. I said brainstorm, you know what the key points are. And then in the first one, the intro, I just said, you know, write the intro and especially so the introductory paragraph. So that's what it's writing here. Okay. Yeah. So we've broken this down into three cognitive tasks, which is write the intro, brainstorm your points, and then make an argument for each point. So let's see what happens. Cls to clear screen. And then it blows up. Always does. What happened? Did I forget to do something? See? 625. So it did. Thorough assessment and evaluation. Yeah. Okay, that looks right. Thorough outline of points. The author. Oh, that's what I did wrong. I. Yeah, okay. Oops. I used this one. Got to get rid of that. We want that. We want to preserve the new lines. Okay, let's try that again. So what happened was I went back through my logs. This is why I keep logs. And you see the response, I said the author. And then it begins by discussing. But it's all on one line. It's because sometimes what I like to do is compact the responses, but in this case, I don't want to compact the responses. Okay, so let's try that again. All right. That's pretty typical for what we saw before. Yep. It looks kind of repetitive. So let's see what the final output looks like. You see there's actually quite a few more steps. So at 627 it ran it. You know, there was the introduction, the brainstorm and then there was looks like four points that it made. So let's go back out here and look at the output. See, this is a persuasive argument based on empathy and personal experience. Mills makes a compelling argument that illegal immigration is not necessarily wrong. One weakness is that does not offer specific solution. It looks like it's actually repeating itself almost verbatim. Let's take a look at some of those, some of these logs, especially these last four, because these are going to be where each point was. Oh, that's what I did. Oops. Oh, no wonder they look identical. I'm not populating it. We're not using that. We got to. We also got to replace. Replace point. Good grief. With the point. Because let's go to the GPT3 log and you see here where it's like the summary should focus on the following Point. Point. Okay. Good grief. I'm dumb. You can tell I haven't had enough caffeine today yet. It's also still 6:30 in the morning. How long is this video while that's running? Oh, this is 45 minutes. Yeah, we're, we're calling it here regardless, but I think you guys get the point. There we go. I think this is probably good. Okay, we're generating some output. Mmm, there we go. Oh, wow. It's kind of going back and forth, but that's fine. It is a little repetitive, but this is certainly better. Okay, so now let's see what the final output looks like. Reload it. Oh, look at that. This is long. This is a 6000 character response. I'm not going to read the whole thing, but I'll save it in the output so you can see kind of what we've done. And now you've seen meta prompting or prompt chaining in order to construct a very thorough argument. I imagine with the right fine tuning and refinement you could do the same thing for any kind of argument, legal argument or otherwise. Okay, so I will go ahead and get this set up and uploaded and you can check it out on GitHub. Thanks for watching.