← Enterprising Investor
Enterprising Investor
Brian Pisaneschi, CFA: Demystifying AI for Smarter Investing
00:24:48
CFA Institute Research and Policy Center senior investment data scientist Brian Pisaneschi, CFA, breaks down the future of AI in investing—covering retrieval-augmented generation (RAG), agentic workflows, and why human creativity remains your...
Loading summary
Transcript25 lines
- [00:00]
A
Foreign hello and welcome to the Enterprising Investor, the flagship investment podcast for CFA Institute. I'm Mike Wahlberg and today's guest is Brian Visaneski, cfa. Brian is a senior investment data scientist at CFA Institute who was impressively recently named one of Investment Week's asset management game changers of the next 30 years. With a background in quantitative finance and a research focus at the intersection of data science and sustainability, Brian is helping shape the future of investment automation. Today we're going to talk about a research series Brian's been working on that dives a little deeper into how AI tools do what they do and what opportunities that may create for investing. His work aims to demystify AI and empower investor to experiment with low risk, high impact applications. And I for one look forward to our little tumble down the rabbit hole. Welcome to the show, Brian.
- [00:57]
B
Thank you very much, Mike. It's good to be here.
- [01:00]
A
So let's, let's start with the basics. In this piece that you put out for CFA Institute, Brian, you talk about retrieval augmented generation. What is that and how does it fit into the big black box that we know as AI in quotations?
- [01:14]
B
Yeah, so I think a lot of us are probably used to going on ChatGPT, uploading documents whenever we want to try and bring in some external information. And essentially what that is is a retrieval augmented generation system. And so it is a way that we've, essentially us in the machine learning community have been able to provide more updated information or also more real time information. When we can connect it to say, the Internet. We can pull in more information because these large language models have cutoff dates. So they have training dates. At that point they no longer have any information, so they hallucinate things that they don't have. So we've used retrieval augmented generation to essentially, you know, you get a question and it is connected to this, what's called a vector database. And essentially it's going to go into this vector database and find chunks of text that are most similar to the question. So you know, if you were, you know, asking the retrieve augmented generation system that has uploaded with proxy statements, which is the example in the case study about, you know, executive compensation of the CEO, it's going to pull in say the top 10 chunks of text into the context window of the large language model. So that's what the large language model can see. So once it has that, then it can ground its answer off of that information. So it's essentially just a way for us to reduce those hallucinations, have More updated real time information. And yeah, it's been kind of the go to tool for that use case right now.
- [03:06]
A
So you've said that rag or retrieval augmented generation works well then for the qualitative data. And that's, that's what you looked at in here, but struggles sometimes with quantitative analysis.
- [03:18]
B
Yeah.
- [03:18]
A
Can you walk us through why is that and is there anything that investors should be thinking about when, or keeping in mind when they're using these tools?
- [03:24]
B
Well, first off, the large language models aren't meant to essentially, you know, perform calculations. That's never what they, they were intended. They are these autoregressive generating of words. So they're just taking previous words and they're coming up with the highest probability which would be the next token, which is a word in our, in our example here. And they're just generating. So like for example, like math, if, if you asked it, you know, a highly complex mathematical question, you know, it actually might get pretty close. It'll actually get to a, a certain, you know, probably a decimal point to the right answer. But then it just starts hallucinating because it's seen those questions and it has a, you know, a probability of what the answer is, but it's just hallucinating it. So it's not, so it's not meant to do those quantitative tasks. Now in terms of when we are interfacing with these documents in a rag setting from, for the example in the case study, which is interfacing proxy statements, a lot of where the quantitative information fails, it stems from its ability to extract it from tables. So you have a lot of that quantitative information in tables and a lot of the times the tables are just, you know, in different structured formats. And it just is lacking in its ability to pull out the right information versus say in a, in a passage of text, it has, you know, these, the context of the words leading up to, you know, answers. It's pulling that in to its context window and it's being able to answer that question. So it has a lot to do with the structure of the text and how the structures differ between different reports as well. But one of the things that I mentioned in the, in the piece is that really a lot of, you know, the quantitative information, the qualitative versus quantitative information, if you're just doing single queries, you know, just interfacing, you know, what was CEOs X's, you know, executive compensation for this period, it does a pretty good job just single query interfacing. It's really, it starts to get Complex. And it starts to lose its accuracy when you start asking it, you know, highly complex and nuanced questions, including say ex, you know, in, in a case study, extract executive compensation for, you know, company X, company Y, company Z. And then it has to pull in from multiple sources and it starts to lose some of its accuracy there. So it really stems from, okay, it is, it is fairly good at single query interfac, but it starts to lose some of its accuracy when you start to, you know, add really, you know, complex nuance questions. And this is where agents come in, this is where agentic workflows come in, where you actually can split up a complex task and really leverage that single query, you know, accuracy.
- [06:50]
A
So, so would this be the hybrid automation, this idea of kind of having multiple tools that are doing the things that they're good at. So language, large language models are good at language, at figuring out that, summarizing, generalizing. And then we have calculators. Right? Like, exactly. Why don't we, we don't really need these AI tools if they're not good at it. Plug a calculator in. Can you talk about, about that? Like, what does that look like?
- [07:13]
B
Yeah, absolutely. I mean that's, that's literally what everything, you know, VC companies, if you look at say Y Combinator right now, it's like 73% of VC right now is all trying to build these agentic workflo. So the agentic workflows essentially do exactly that. You know, these large language models aren't meant to do math problems. You. Okay, so then you connect the large language models to accurate data. That's that you can pull in similar to retrieval, augmented generation. But it's just, it has access to functions to pull in correct data, you know, to pull in correct financial metrics so that it doesn't have to try and do these calculations and make mistakes. So that's where agentic workflows come in handy, where if you have a highly complex workflow, you can break that up into steps and then at each step you can, you know, go out and have the large language model connected to APIs. So application programming interfaces that are connected to accurate data. You can have it connected to a calculator, to a Python interpreter so it can run some data analysis which, you know, these large language models have quite, you know, grown in their accuracy for coding abilities. You know, coding and software engineering is a big cost center for firms. So that has been the main kind of objective is, is to, is to get these large language models really good at coding. So that they can, you know, hopefully reduce some of the costs that go into.
- [08:56]
A
I was just trying to think is that irony is that like punishing them, the, the creator of the, of the monster, basically you do all this work to make something so good that it puts you out of, puts you out of the job for these poor developers.
- [09:11]
B
I think that there's a lot of irony in that. I think that the LLMs are still just not, they're not good enough to completely get rid of software developers, that's for sure. And I would strongly recommend people who are interested in finance and also interested in the tech side to still, you know, learn about the tech, learn what's under the hood. Because, well, for one, you know, competitive advantage is found really on the edge of that existing technology. So if I tried to create an agentic workflow right now just using a large language model. So there's a term in ML space called Vibe coding. So that's just where we, we work with a GPT or anthropic Claude and we just ask it a question, code up some, some process and we copy it, we throw it into our Python interpreter and then we, we see if it works. If it doesn't work, then we ask Claude, ask GPT, we just Vibe. So if you just tried to Vibe code some of these new ways to create agentic workflows, you're going to run into a lot of problems because as I said before, you know, the, these large language models have cutoffs. They have to pull in the correct information. They don't always go out to the Internet and pull in the correct documentation to be able to do these more novel, you know, coding tasks. So you still, you know, you will very quickly run into frustration. But that's just one aspect. The other aspect is the fact that these workflows that are being used to automate these processes are still very much rooted in the computer science kind of way of structuring a, an entire workflow and being able to think like a computer scientist, being able to, you know, really outline how a workflow could be automated, is going to help you even in a no code fashion. So this is when we start talking about, you know, these, these, these platforms like N8N or Zapier, these are designed for people who don't want to code. And they're these kind of graph structures that allow us to create these agents where we connect them to outside data platforms. If we want to create some valuation model or scenario analysis model or something like that, we have one node and we connect it to the input data we are in. And then just like you would do in GPT, we give it a prompt, analyze this data or clean this data and then it moves on to the next step and then it brings in other data or some other step in the process. And this is all done without code. But the, the entirety of building this, this structure is, is very much rooted in being able to think about how to automate this entire process. So it's a way of thinking that I think is still very much important going forward.
- [12:36]
A
So you're describing taking sort of the data science approach and applying it across kind of how we do our day to day jobs. Talk about the inverse of that, talk about human creativity and curiosity. What do you see as their role in all of this?
- [12:52]
B
Well, for one, I think that that's the biggest thing that we as an industry need to be sending out the alarm bells. I think in our industry, you know, we've downplayed the need for creativity because a lot of our workflow didn't involve highly, highly creative tasks. But if we really want to differentiate our ability from AI's ability, I think that that is, is it in 10 years time, I still think human creativity and curiosity are going to differentiate us from AI's capabilities. So we need, the industry needs to be giving their teams a lot of space to be able to exercise that part of our brain. Because if you don't give the teams the exercise, the ability to exercise that part of your brain, you know, it's, it's the same kind of discussion that you know, we talk about even with software engineers. It's like, okay, well right now GPT as an intern is like a 247 coding intern that you would give to a software engineer. Which leaves the junior software engineers lacking the ability to get any experience. And so if you don't have something set up in place that is going to give them that space to still really, you know, exercise that ability, I think that we're really, you know, shooting ourselves in the foot. So I think that absolutely curiosity and creativity fundamental going forward. I think that the industry is kind of on a path to be being a lot more scientific, a lot more creative in that aspect. You know, you need to be able to know what these tools are and the technology that exists. But most of the competitive advantage is going to come from your domain expertise and really having that holistic idea of the entire, you know, finance industry so that you can find those little pockets of value that you know, are hidden and also not the norm because you Know, as we can see, like within GPT, there's, they're biased, like, they're, there's a lot of things that are built into the training data that just kind of guides an LLM to the majority because it's seen it so many times and that's what kind of allows it to become bias. So, you know, it really is important for us to understand where those biases are, how we can mitigate them.
- [15:42]
A
And that's, that's, that's so interesting because it means, I mean in this industry and I've worked as an analyst before and you know, it's part science and part art always. Right. As a fundamental analyst anyways and to a certain extent quants as well. But if you're just, what you're describing is where this, you know, you're going to really need to lean into the art more to, to on the one hand, lean into the art more to find your edge, to find those opportunities. But I, I find that really interesting, this idea that if everybody starts relying on the same large language models and they, they're all, it's, it's, it's like anything in investing, right. If you start to drink the Kool Aid and you believe what, what market consensus is, but now the market consensus is going to be coming from, from the biases in these models to the extent that people are relying on those models for the science part of their job.
- [16:31]
B
Yeah. And I mean even, but to go back to, you know, the retrieval augmented generation essentially, that really allows the analysts and the PMs to, you know, inject their own creativity into the models. So the large language models are, they're based off of the attention mechanism. So they've been trained to really, you know, place attention on certain words in a passage of text in order to accurately output, you know, the next word, a next token. And so if you, you know, if it has all this bias information, but then you give it updated information in your retrieval augmented generation system or you've given it a prompt that is very guided towards your secret sauce, it's going to attend to your secret sauce. So it does still give the analyst the ability to be creative and, and add their, their own competitive, their own alpha that they've found and over the years of developing it, so, you know, as much as it will, you know, kind of go and I would say default to the, what it's seen in its training data the most, you know, you still have that ability with, you know, some, some prompt engineering, as you mentioned before, you know, it's that art. I've personally believed that prompt engineering is a lot more of an artist than it is a science. There's a lot of, you know, basic structures of prompt engineering. You give it a rule, you give it a task. You know, you, you kind of give it some limitations or guardrails of what it should and shouldn't do. You give it examples and you give it a format. But those, those things are just kind of like basic things. Most of what I think to be really good prompt engineering stems from your domain expertise or your ability to communicate. Like, it's just, it's the same kind of like, really good communicators are gonna produce much better prompts and they're gonna get better answers. Because if you can add that nuance to really get down to what the, the task is, you're gonna be a better prompt engineer. So I really think that, again, is just like what you're saying. It's just more of this kind of art part that, that we can inject into these large language models.
- [19:07]
A
What's a big misconception that investors have about AI right now?
- [19:14]
B
Well, I, I mean, personally, I, I think that, that there's been a misconception on. They, maybe they'll look at GPT, they'll have some answer that it doesn't actually, you know, produce its. Produce it correctly or it failed at whatever task it, it gave it, and then they just kind of brush it off and they say it's, it's no, it's no good. I think that, that. Miss that level of perception and the capabilities of these models is, is lacking. The nuance of the fact that when we really can connect different external data sources and when we can define a process, a workflow with guardrails, with evaluations that we can do that the capabilities of these things are far greater. And so I think there's a misconception of just kind of brushing off these capabilities just from the use of, say, ChatGPT, the platform that's on ChatGPT. And I think that if you, if you have a team that can help you build these workflows, I think that that's, that there's a lot of capabilities and I think kind of rooting on that. I think that what a lot of, when people talk about agentic AI or AI agents, they look at these kind of autonomous systems that are going to be, you know, super difficult to audit, super difficult to keep track of, and really think that, you know, okay, this is just not something we can work with. But going back to the workflows. I think that absolutely, you know, giving an agent so much autonomy is most likely going to be a disaster. But again, if you can break down a workflow into individual steps and at each individual step you've established what is an accurate kind of output from that and you've run evaluations, I think you can do some really great robust things.
- [21:26]
A
So breaking it down into steps is, it sounds like it's a way to kind of create your own audit trail, avoid the hallucinations, be sure that it's not just like making something up at the last.
- [21:36]
B
Yeah, yeah, I think that that's where we're headed for a good amount of time. I think that the really autonomous agents are quite a ways away in a highly, a lot on the line in our industry. High risk of output, which is really what you need to think about when you're thinking about automation. You have to think about, okay, well, is the risk of this task something we can, the output of it? Can, can we handle that? If it is, then, you know, maybe that's a, a task for, for automation. But I think a really good way to kind of think about, you know, automation tasks is to think about where you had naive approaches for doing things in the past or where, you know, being able to scale up or just like manual process that you couldn't do to a high degree in the past can now add more nuance. So if you had a stock screener that you had some kind of like binary filters on that stock screener to create, I don't know, some sort of custom index for, for a client that those thresholds that you just put in place, you can add more nuance. Now with large language models that can pull in different context for those thresholds that I think is basically like looking at a problem and saying, okay, this is a nuanced task and I've kind of, because I'm limited of time and effort, we've created this kind of process that is, you know, naive in its, in its attempt. Well, now we can add more kind of complexity and robustness into our stock screeners with, with large language models and things like that.
- [23:28]
A
So, so Brian, we're, we're coming to the end of our chat here today, unfortunately. I'm going to hit you with our two part final question here. What was your first job in the industry? And if you could go back and take yourself for coffee on your first day, what key piece of advice would you offer yourself?
- [23:44]
B
So, first job was a, an accountant, corporate accountant with a property casualty insurance company. I. What would I say to myself? Well, I would say that be as true to yourself as possible and take as much risk as you can and everything will work out. I think that that's it.
- [24:08]
A
I've been speaking today with Brian Pisaneski, CFA senior investment data scientist at CFA Institute and author of Rag for Finance Automating document analysis with LLMs for the CFA Institute Research and Policy Center. You can find it there on the CFA Institute website. Thanks so much for coming on the show today.
- [24:25]
B
Thank you, Mike. It's a pleasure.
- [24:27]
A
I'm Mike Wahlberg, and this is me, the enterprising investor.