← How I Invest with David Weisburd
Loading summary
Transcript58 lines
- [00:00]
A
How does super forecasting play into how you manage $26 billion today?
- [00:04]
B
This is a great question and I love that we're starting with Dr. Phil Tetlock. So Phil Tetlock spent decades proving that like most experts are terrible forecasters. Not cause they're like dumb in any way, it's just that they're like most of us, overconfident and like they don't keep score. So, and I know this firsthand, but back in 2010, Dr. Tetlock organized a team to compete in what was really a four, a four year forecasting competition for the Department of Defense. And I was on that team, so I was a guinea pig. And over like the subsequent four years, which I think went from 2010 to about 2014, grinding through geopolitical questions with real scoring, I learned what we call calibration and what calibration thinking actually feels like. So he eventually published his findings in the book Superforecasting, which became a bestseller on Amazon probably around 2016 or 2017. And I basically brought that methodology back to my investment team. So what does that mean for us right now? Every investment decision requires a very explicit probabilistic forecast. And let me explain what that means. It's not like I think this fund looks good, but it's. I believe there's a 55% chance, for example, this state manager will outperform the MSCI healthcare index by 400 basis points over the next five years or five years from the close of the fund. So it's specific. You have to have a probability forecast. You have to have a clear definition of success and a time horizon and a confidence level. So the discipline isn't just about, you know, being right, it's about being calibrated. So when you say you're 90% confident, are you 90% right? And so it's about knowing the difference between what you know and what the information allows you to know and what you believe.
- [01:45]
A
So how do you track your team's investments internally based on the super forecasting methodology?
- [01:50]
B
We use the Breyer scores. And the Breyer scores are something that were championed by Phil Tetlock and he has since commercialized his research into what's called the Good Judgment Project. So the Breyer score is just a mathematical formula that ranges from zero to two. So lower score is better. So a zero Breyer score would be that you're 100% confident and 100% right. So you're basically like an all knowing kind of omniscient. Omniscient being. And a 2 is that you're 100% confident all the time, but like 0%. Right. And just for reference sake, a Breyer score of 0.5 would be that you're sort of a, you're sort of a coin flip. But it's not just about the Brier score. You also have to attach kind of a rationale, you know, how did you get here? And so we're tracking the quality of the decision making as well. And we can measure, you know, hey, if somebody's overconfident or underconfident, what would have happened to specific investments if you actually had that, if you're actually appropriately calibrated. So that's, that's one way that, that we do it to make sure that when, you know, first of all, there's an objective score, but also that people are right for the right reasons. And that that level, I think, of introspection is really important and that's the main way that we track it.
- [02:57]
A
So this is a fascinating thought experiment in and of itself, but you've actually implemented at Arizona PS PRs. What are the second order effects of this?
- [03:05]
B
First of all, intellectual humility just becomes absolutely contagious. So when everyone's score is on a wall, we're all competitive people, right? Nobody walks into a meeting claiming some kind of certainty that they don't have. So that social vindication element really is looming large. Second, I think we make fewer big mistakes because the process forces you to articulate what has to be true for the investment to work. And step one starts with setting what we call the base rate, meaning you take this outside view and you ask, like, what usually happens, what usually happens to, say, a private equity fund or whatever it is that you're looking at and that keeps you from falling prey to sort of like, you know, this, this illusion of control. The third one is, and maybe, maybe the most important is that it changes the culture of disagreement. So in some, I think, investment committees, it's like this really combative, sort of, you know, gladiatorial event where there's a clear winner and a clear loser, or where you have people who are just really good at debate, who can take up a lot of air in the room, or maybe it's just the senior people in the room talking and the junior people don't, you know, don't speak up. All of that goes away when you have this objective scoring and this, and the, and this objective, very quantitative approach. So it sounds more like, hey, I'm sort of thinking 60% confidence in this investment, but you're kind of at 80. Like how do we bridge, how do we bridge the gap?
- [04:28]
C
Expert calls have always been one of the most powerful ways to build conviction. But today investors are asked to to cover more companies, move faster and do it with leaner teams. With AlphaSense AI LED expert calls, their Tejas call service team sources experts based on your research criteria and lets the AI interviewer get to work. The magic is in the AI interviewer purpose built and knowledgeable based information to conduct high quality context stretch conversations on your behalf, acting as a trusted extension of your team. Then they take it one step further. Your call transcripts flow natively into your AlphaSense experience and become queryable, searchable and comparable. So your primary insights plug directly into earnings preps, digital work streams and pitchbooks with zero tool switching. And with AlphaSense expert call services, the AI led expert calls are just one option because we know the importance of a hybrid expert research approach. AI for coverage and efficiency, humans for complexity and conviction. It's the institutional edge that scales research without scaling headcount. For hedge funds, that means validating thesis assumptions across dozens of experts before earnings instead of a handful. For private equity, it means faster pre IOI scans and deeper commercial diligence. For investment banks and asset managers, it means pulling real operator perspectives straight into models and sector positioning without disconnected tools or manual handoffs. All of it lives inside the AlphaSense platform trusted by 75% of the world's top hedge funds. Alongside filings, broker research news and more than 240,000 expert call transcripts turning raw conversations into comparable auditable insight. Take advantage of Alpha Sense AI led expert calls. Now the first to see wins. The rest follow. Learn more at alpha-sense.com/how I invest so
- [06:14]
A
everybody wants to improve their score. So they're actively looking for feedback from the rest of the team in order to better calibrate their position versus this being the zero sum, this is my investment and somebody being against investment and you've kind of butt heads in the sear some way.
- [06:27]
B
Exactly. And it's, it's very, it's really fascinating. Some of the best forecasters on the team are people who have come to us with absolutely no prior investment knowledge. Now they're sort of probabilistic thinkers and, or they're what we call numerate, meaning they just sort of think in terms of sort of quantitative. But it's not like they're running real complex models. Sometimes it's just that that's, you know, they're Thinking in terms of likely outcomes, likely ranges, you know, and again, they start with the base rate, what normally happens, and then they start looking for data. They can kind of move them off of it and it's not so robotic, you know, but you can sort of say, hey look, on average there's kind of a 30% chance of these things. But I'm looking at this other data and that kind of is moving me. You know, I moved up kind of 5 or 10% here, right? But then this kind of moving back down 5%. So you're, you're putting guardrails around a conversation and it's not, it's not a subjective. And I find that that's just really important.
- [07:15]
A
I've been going down this rabbit hole of figuring out how the most innovative people in the world become the most innovative. And prime example, of course, is Elon Musk. I've interviewed a lot of his friends and a lot of people around him. And one of the things that I've learned about how to tell whether somebody's crazy or extremely innovative, one way is, does it actually work? So you could tell a post facto, in the beginning, Elon Musk was trying to create a private space company. Even his closest friends, his most innovative friends, had an intervention, showed him all these videos of space space rockets crashing. But he really had first principles. So he had these ideas, these building blocks of logic, and he had this thesis on why it would work. And the thin line between whether somebody's crazy or extremely innovative is actually not based on them or even people's reactions. It's based on the first principles, logic. So I know A, I know that sounds crazy. Okay, fine. Now B, now look at my logic. Poke holes in it either to improve it and to improve the thesis or to kill the thesis. But if the first principles hold even though it's completely crazy, then it's actually innovative.
- [08:20]
B
Not, not crazy in the context of our organization. I think that's what this, this forecasting application really does. Because what you're trying to build over time is calibration, right? Are people, when they're, when they say they're 90% confident, are they 90% right? If they say they're 60% confident, are they 60% right? So we write down, of course, all the training questions and also all the investment recommendations, and we're looking at those resolutions, right? And we, for any individual on the team, we can say, well, here are all your 60% confident forecast, right? Here are all your 80% confident forecast and where you write 8 out of 10 on those 80%, 6 out of 10 on. On the 60. So when you get to innovation, that becomes really important, because what you want as a CIO is to know that your team is calibrated. Somebody. Somebody comes to you and they say, well, here's an idea we probably haven't thought of. We start thinking through, you know, okay, well, is this more like a. Is this a 50, 50 bet? Right. And that we can get into, you know, how that works with a specific investment example. But like, that is really important to help, like we said, help parse the ideas that are truly crazy from the ones who. That are actually innovative.
- [09:20]
A
This reminds me of Bridgewater and how Ray Dalio created the principles where everybody kind of openly fights their positions in order to get better. But upstream of that is if you, if you have this organization where people know that they could come to your organization to basically tune and calibrate their investment skills, that's a hell of a recruiting advantage over other organizations that might be seen as tribal or political.
- [09:42]
B
Now that we do it this way, I can't imagine doing it, you know, any other way. Because when you, when you have that level of objectivity, it just levels the. It just levels the playing field. And people who wouldn't normally speak up, you know, because they're newer or for any other reason, actually have a voice. And it's not necessarily that they speak up more, but it's the people who tend to be overconfident. Right. That will start to defer. And I think that's just a really healthy team. Team dynamic for sure.
- [10:06]
A
There's this paradox of knowledge where the deeper you go into topic, the more you realize how much you don't know. So said another way, if you don't know a lot, you're gonna have a lot of times a very high confidence. So there's. There is this negative correlation between somebody's confidence and sometimes their depth to their knowledge.
- [10:19]
B
That's well said because there is like the paradox in decision making, which is the, the faster you admit what you don't know, actually, the better you are at making decisions. And we see this pattern in the forecasting results of our own team. So what'll happen is they'll make a few forecasts early on and let's say the results are really good. So then they build up all this confidence, right, without really getting any. Any better at the calibration process. And then sort of like mid, you know, mid sort of forecasting career, right? Then they get a bunch of really bad scores and Then they figure out, okay, wait, wait, you know, maybe I was overconfident. So then they, then they start to pull back and they think, okay, on the first ones, maybe I got lucky here, and I, I confused that luck with skill. And then I got overconfident. Now you start to break them down, and that's the process of calibration. And they start to, they start to pull the confidence down and you and their accuracy will start to go up. And that's what you want to see.
- [11:11]
A
Do you go through this super forecasting process as well?
- [11:15]
B
Let me give you an example of how this would work on the team, right? So you have high, you know, kind of have high and low confidence. And let's just take like a low, like a low confidence example, right? So the low confidence example. Oh, I always get asked this, right? Can you, can you make a low confidence investment? And low confidence doesn't mean, you know, don't invest. It's just like poker, right? You don't fold a hand just because you're not holding the aces, right? So this is classic, the way we approach this, classic expected value theory, right? You multiply the probability of being right times the upside alpha subtracted by the probability of being wrong times the downside, right? And then if the number's positive, you sort of have a case. So, like, here's an, like here's an example. I'll make the math easy on myself on the podcast here, right? So say you're like looking at a private equity fund and it's focused on a specific sector. We'll benchmark them, say on, on like a PME basis against the sectors like public, public index. And let's just say, for the sake of argument, right, that history tells you in any vintage, say 2 out of 10, you know, sector specialists will add a thousand basis points of alpha over the sector benchmark. So your starting base rate is 20%. You can be 20% confident with, you know, that's the outside view, what normally happens. 20% confident that a manager is going to add a thousand, a thousand basis points of alpha. And then you go hunting for information that moves the needle. So maybe their prior fund did it. And that maybe moves you up from a 20% to a 30%. Maybe the team's intact, maybe there's a proprietary sourcing angle, they've got a playbook. But let's just say for sake of argument, that kind of moves you to 50%, like a coin flip. So 50, 50, right? Meanwhile, let's say if you're wrong, you Estimate that you lose about a hundred basis points relative to that sector index. So you take the average of the other eight managers and let's just say they're a minus, you know, they're minus 100. And so the math is you have 50% times a thousand basis points, so that's 500 minus 50%, you know, times 100. So your expected value is 450 basis points. Right? That's not certainty, but that's a solid investment and that's basically how we approach it.
- [13:14]
A
Last time we chatted, you mentioned that you're being benchmarked against the S&P 500 whether you like it or not. Why is that?
- [13:20]
B
So here's the uncomfortable truth of institutional investing. Every dollar we put into private equity, real estate, hedge funds, you know, what have you, it's a dollar. We didn't put the s and P500 index or a bond index, but nobody, nobody compares alternative investments to the bond index. So the S and P wins the benchmark conversation. Conversation, like just by default, probably because it's like the one number everybody already knows. It's familiar. It's on the news every night. So even if it's not your explicit stated benchmark, it's the one in the back of every journalist's mind, every legislator's mind, every board member's mind, every constituent's mind. And like the S and P has been, you know, extremely good for the last 15 years, which makes the conversation even harder. And my job is to justify the complexity, the illiquidity, the fees of a diversified portfolio. And the only honest defense is long term risk adjusted returns over full market cycles. And we're not trying to beat the S and P every year. We're trying to not be the fund that blows up when the cycle turns. But like, as I say all the time, diversification is a absolute punishment for sure in a bull market, but it's a miracle in a crisis.
- [14:30]
A
And it gets even more complex given that you manage for police and fire men. So it's not just one single decision maker where you could basically explain and educate them on the market. There's thousands of underlying members that are de facto going to use S&P 500 as this mental benchmark.
- [14:46]
B
Absolutely. And you have to keep your audience in mind and that you just. And I think it's good that you sort of keep, you know, keep that idea in the back of your head because it does keep you really honest when you're, when you're adding investments. And I think that's you know, not a bad way to construct a portfolio. I do think you have to, you have to have kind of like, you know, what your constituents understand in the back of your mind, you know, at all times. So as annoying as it is, I think it is a healthy way to build a portfolio and I think it's a, you know, a fair way from a public policy standpoint to interact with your constituents.
- [15:16]
A
And you have to construct your portfolio in these two co centric circles and what, who your audience is and what is the ideal portfolio and find a way to kind of match both of those.
- [15:25]
B
By and large, you want to kind of keep the portfolio constructed in a way that like, people will understand it. Right? I mean, that's the, I think that's, that's really critical. I mean, these are people, right, who didn't sign up to study asset allocation and their retirement is not in their own hands. And it can be kind of disconcerting to kind of like trust other people and, you know, with your retirement money. So I do think you have an obligation to construct something that they understand. Now at times there are things where you just know, look, this is not going to be well understood to a layperson, but we feel strongly that it's going to be helpful in the portfolio and then you have that conversation with your board. But I do think you have to keep what your constituents will understand in the back of your mind at all times.
- [16:02]
A
Tell me about your portfolio construction today and maybe how that's evolved over the last few years.
- [16:07]
B
So when I became the CIO in 2019, we had something like 10, 10 asset classes, I think is what we had may, you know, maybe more so. And I've changed that to simplify it remarkably. So we start with a really simple premise, right? Our liability at the end of the day is a long standing stream of payments to first responders. Right? And so for me, the mission is in this order. One, have money when money is due, manage portfolio volatility so it doesn't translate into contribution rate volatility that employers and members just, you know, can't absorb because it makes it hard. And do all of that for do all of that. I think spending as little money as possible. That's the whole job in that sequence. And so like I said, we do feel an obligation to make, to make it understandable to the people whose retirement this is. Again, they're not signing up to study stocks and bonds. They signed up to serve. So we really break the portfolio at a really high level into three broad categories. And for us that's called capital appreciation, contractual income, and diversifying strategies. Because those labels, I think, describe how investments actually behave, not sectors or anything like that. And we tell members within each of those, there's publicly traded investments and there's private ones. That's basically it. So the edge in my mind isn't having, like, all these exotic buckets. It's just having fewer of them and filling the buckets really well.
- [17:28]
A
You could think of the buckets as essentially constraints. So if you have a specific public equity and private equity, and for some reason you see private equity as more risk adjusted, then you're arbitrarily constrained into this public equity where you don't want to be playing just because somebody came up with these buckets.
- [17:42]
B
I get asked, why have the loosely defined buckets? And I think, to your point, I think the rigid buckets can. I mean, the glib answer is the rigid buckets can lead to just, like, stupid decisions, right? So I think the problem with defining asset classes by, like, sectors having an asset allocation, I mean, where you have sectors and sort of capital structure, so you have a real estate allocation, but then you also have a public equity allocation, right? So you get into these conversations about, well, what do you do with a real estate equity guy or a real estate credit guy, right? And I just think that that gets conceptually messy pretty fast. So, sure, real estate, you know, sometimes behaves differently than stocks, but, you know, how often does it really. Right. And with, like, what degree of confidence? And take probably the most recent example would be commercial real estate. Right? Commercial real estate has been, like, a longtime darling of the inflation protection crowd. And then Covid hit followed by actual inflation. And commercial real estate didn't help because other forces were swamping the inflation signal, right? So the theoretical benefit just got buried in the specific facts on the ground. So I think making the buckets wider increases competition among investment ideas. And for a pension fund with a longer horizon where we can stomach some of that volatility, the more competition for capital, I think, just leads to better outcomes fundamentally. And so loosely defined ranges lets us be flexible and opportunistic, which is a huge advantage. The buckets are, in our minds, just kind of a map. But, like, you know, any navigator knows the. The map's not the territory.
- [19:09]
A
Are you essentially implementing TPA into your portfolio? And if not, why not?
- [19:13]
B
I'm gonna be honest, I. I can't describe tpa, and every time I hear somebody, like, define it, I just think, like, that kind of sounds like what we're already doing. And, you know, maybe we all arrived at it independently and maybe that's a sign, right? Maybe the logic of it is pretty sound. But I think either way, the core question is within each sort of bucket or sort of risk category, what's the best complement of assets available to us right now given our size? It's not like, hey, is this a good credit investment? Or to your point, like, hey, is this a good private equity deal? And maybe private equity is just not all that attractive. But it's like, is this a better use of capital than the next best alternative with a similar risk and correlation profile? And so I think that reframing is just really, I think really important. And I think it's generally hard to execute institutionally because it just requires overcoming this gravitational pull towards asset class silos, which is just run a huge deep trail down all of our cerebral cortexes. People get comfortable defending their buckets, right? And I think, like, my understanding of TPA is that you're just, you know, you're trying to compete across all the buckets all the time, right? And compare things with common risk exposures to things, other things with common risk exposures.
- [20:19]
A
One of the benefits of this bucketed approach or endowment approach is that you have your best athletes in specific asset classes. So you have your real estate credit, your real estate equity, you have your private equity, public equity. How do you build a team that's able to be so flexible? And how do you build incentives mechanism in order for your team to be flexible?
- [20:39]
B
That's a great question. So I think the answer at the end of the day is, you know, you just have to hire the right people. Public funds have like various constraints. You have to hire people that are intellectually honest. You have to hire people that just never stop asking questions. And the nice thing about having fewer buckets is you don't have anyone that's territorial, right? I can say to someone on my team, hey, listen, you've got to manage the private, you know, the private equity investments right now. But let's be, let's be clear, right? We could, this could be zero percent. There's nothing obligating us to have any money in this asset class, right? So I think the good thing about that, and in fact, I love the PM that we have who's running real estate right now. He has spent probably the better part of three years just saying no to everything. But that hasn't been his only assignment. He's worked on a bunch of different asset classes. So I think you just have to going in, tell the people, look, you're going to be athletes, you're not going to be married to a single asset class. And that just makes people really honest. And it's not hard for people to say, look, I don't think there's anything to do here in fixed income right now, so I'm not making any recommendations. Or real estate where we've just been saying no, not re upping with any of our partners and nobody's worried about their job. It's all about what's the best allocation. If we're going to take this kind of risk, where's the best opportunity in that market right now? And I think that's just a better way to run the portfolio. And I think it's resulted in improved performance for us for sure. When I started as the CIO we were in the bottom decile pretty much across all the time horizons. And now we're in the top third across all the time horizons. And I think just having the right people is part of that. And then also simplifying the asset allocation so you have greater competition for deals because it just raises the bar how
- [22:17]
A
big of your strategy is co investments and has that changed?
- [22:20]
B
So we haven't been real active co investors in a lot of the credit like instruments within what we call global private equity, which encompasses private real estate, private real assets, buyouts, venture capital, all of that stuff. Probably about 10 to 15% of of that allocation. So it's probably maybe 5% of our overall portfolio and increasing. We have a big focus on just trying to get closer to the assets. Co investment is going to become a more important part of our program for sure.
- [22:55]
A
When I interviewed Chris elman, who is CIO of CalSTRS for 23 years, he said one of the ways that they really generated alpha was structural alpha in co investments. And one of the only things that really worked in co investments was rules based co investment. So creating a diversified pool of co investments, lo and behold the alpha there was the fee savings. So it's somewhat of an interesting view on private equity in general. But they were able to use their size to an advantage and capture this co investment structural alpha.
- [23:23]
B
It makes a lot of sense if you focus on a handful of partners. We tend to like the sector specialists and co investments I think from sector specialists are just of a higher quality than say maybe a more generous or diversified manager. Now like obviously there's exceptions here and there, but I think that's not a bad way to do it. And what you're effectively doing is just creating your own private equity fund. But the deals have already been vetted, you know, by a GP that you've underwritten and trust. And so you can have effectively a private equity program with substantially discounted fees or no fee and no carry. Even if they perform in line with all the other deals in a private equity fund, you've maybe increased your return on those by 3 or 400 basis points because you've eliminated that gross. Gross to net spread.
- [24:02]
A
You're one of the best first principles thinkers that I've interviewed in terms of cio. I want to get your opinion on something. So I've been thinking about this concept of the semantics on call it investing. Some people call it allocation, other people call it investing. And then I had Sam Zell's long term partner, Mark Soder, and he calls it own owning. He said, Sam Zell never said I allocated or I invested. He said I own this asset. Something about that to me feels more powerful when you're thinking about which assets do I want to own, not where do I want to allocate away, do I want to invest? You think there's something there?
- [24:31]
B
I do. So in fair disclosure, I have not heard that, but I've always hated the, I've always, always hated the term allocator. Right. I mean, I don't see much of a difference whether you know, you're buying a stock or investing in a fund. Psprs, we do everything. We've got some, we've got some direct investments, we can buy and sell stocks and bonds, we've got ETFs that we buy and sell. We've obviously commit to private equity and private credit funds, we've done co investments. But even if we didn't have all of that and we were just allocating to funds, the practice is the same. I mean, it is an investment for sure. And I think we, I think allocators becomes just more of a semantic. I mean we're just trying to distinguish between the players in the market more than anything else. But I like ownership. I like that idea. I'm going to have to use that and I'll give full attribution to them because obviously for me accountability is a huge thing.
- [25:17]
A
To be fair, this podcast is how I invest. If I could call it how I own. If that sounded better, we would also call it how I own.
- [25:25]
B
True.
- [25:26]
A
But I have to be intellectually honest. I do think own is stronger than invest, and I think invest is still passive.
- [25:32]
B
You could say maybe semantically ownership is. Maybe something was given to you or you inherited it or something like that. So it wasn't maybe necessarily a certain decision with an outcome that you have to realize. But yeah, I mean, I certainly prefer invest over allocate any day. But I like, I like the ownership idea just because again, start to finish, you own the work and you own the outcome. And I think that that's your semantics. You know, words matter and it's important for enforcing behavior. So I, I do like that idea.
- [25:59]
A
What's something you've changed your mind on in the last 12 to 24 months?
- [26:03]
B
I'm anchoring to, you know, the most recent geopolitical events and some of the conflict in the Middle east, but we've been on a rampage to really humble the confidence of our, of our investment partners. I probably would have thought three or four years ago that the confidence that our partners have in their specific outcomes were merited. I no longer think that as we're looking at our managers across the board, because we do sort of implicitly benchmark them using sort of the same kind of forecasting process that, that we have internally. And a lot of them are, are overconfident. And I, and I think, you know, that's been a fundamental shift in my mentality is that, you know, we, it's not, it's not a 50, 50, whether they can do what they say they do, but like, we have to be a lot. I mean, the default has to be. Look, there's a, you know, maybe a 20 or 30% chance that they can do what they say they're going to do. There's a lot of marketing. But I think treating partners with a much higher level of scrutiny in this environment is really important. So I've gone from, from thinking that, you know, look, they probably have a pretty good handle on things to just thinking, certainly in some conversations, you know, there's marketing. I understand that from a business building standpoint, right. That, you know, certainty is what sells, you know, or if you're on the other side of it, pessimism. If you're, if you're shorting things, pessimism just always sounds smart. But I think what everybody has to do right now is just reduce their certainty in whatever they're doing, whether they're long or they're short. And I think that's critical.
- [27:24]
A
There's such a performative aspect to this confidence. We just invested in a company called Lagora and LP was asking me about horizontal versus vertical AI. I had this itch to say, really? Nobody knows. How do you know horizontal versus vertical AI? And if Marc Andreessen is on the record saying he doesn't know. Peter Thiel doesn't know. Why would I know? But you have this pressure to justify this position, and everybody expects it. You have to basically justify this unknowable variable without saying the more intellectually sound thing, which is, well, if vertical AI wins, or in the 50% of cases that wins, this has an expected value of 5x. But in half the cases, this might. This might be a 1x or could even lose money.
- [28:06]
B
I would be in that camp. I would think, look, I think vertical AI is probably, like, the way to go. It's probably still where I think things are going. Because, look, with vertical AI, it's a huge moat, right? It takes a lot of resources to just understand the sector, because every sector is extremely complicated, right? So there can be a strong case made for that. But I think with AI generally right now, everyone has to be careful. We don't know the full impact of AI. And I would be real hesitant to take a view on whether horizontal or vertical is going to be the winner. And I love the way that you position it, right? I mean, if these guys were the experts in the field. Don't know, then I think it's fair for us to say we don't know. And a similar discussion going on with, like, home builders and interest rates. You know, if you listen to, like, the earnings calls of home builders, you know, they don't really have a clue where the housing market and interest rates are going, right? They can kind of speak for, like, you know, the next year or so. But if they don't know, and you would think that they're probably as close as anyone to it, then you have to say we don't know. And that's really important in terms of how we make decisions when we're trying to create those. Those base rates. You know, I'd be real hesitant if anyone on my team said, hey, look, I'm 60 or 70% confident. I know. Like, you know, the interest rates will go up by at least 100 basis points in the next year. Or I'm 60 or 70% confident that horizontal AI is going to win. Because you'd have to convince me that, like, the base rate right now, according to the experts, is 50. 50. It's a coin flip. And that's max uncertainty. And you'd have to point to some pretty substantial evidence that would move you off of that. We all have to be very humble right now about what we know and what we don't know. And like, I said at the beginning, the paradox of improving is just admitting what you don't know. And the more you do that, then, you know, I think the better, the better you are about making decisions. Now, that doesn't mean you're paralyzed, right? But you have to approach investments and say, Look, I'm, I'm 50% confident. So if I'm looking at this investment on an expected value basis, meaning I'm 50% right, what do I think I gain if I'm right? You know, times like the negative alpha, if I'm wrong and, you know, 50%. So you could still be making investments into that. But I think you have to say this is a coin flip, as close to a coin flip. And there's another sort of behavioral component to this too, which is we're often really good cognitively at sort of the yes scenario. Will horizontal AI win or will vertical AI win? Because you can kind of point to various things and you can build a case in your mind because our human brains are reasonably pretty good at that. But what we're not good is generally the no case. Right. So will horizontal AI fail to win or will vertical AI fail to win? Because that's really cognitively expensive. You have to like, look at all the reasons it possibly could and just blow them all up. And so it is a delicate balance right now, but I would certainly be closer to that 50, 50 threshold just
- [30:43]
A
to pressure test that. Do you have GPS that come and pitch you in probabilistic manner, and if so, how do you react to that?
- [30:51]
B
No, I would say not usually. You know, the conversation with gps, you know, usually goes something like this. We aim to perform over a market cycle. Right. So then our team's job is to, is to pin him down and say, well, what's like a market cycle for you? Well, like five to 10 years. Well, which is it, five or 10, right. And generally partners are. You want to keep that flexibility because. Right. You know, we all understand, you know, the, the future's uncertain and, you know, there's compliance and things like, and things like that. But it's also a huge problem in the LP community. If you can't get your potential partner to define, like, what success looks like, then any result is going to do. Right. This is the classic just drawing the bullseye around wherever the arrow landed. So I think what you have to do is say, hey, listen, we, I need a minimum, a minimum success definition here. Like what? And they'll say, well, we're looking for mid teens over market cycle. Okay, well, like what is that like 15, 13, 18. So you pin them down on that. Well, okay, at least 13. All right. What's that market cycle look like? Well, it's not really a market cycle. We're trying to do the beat, the benchmarking. Oh, okay. So it's a relative performance. So our team, their job is to just sit there and pin down what does success look like. Cause I maintain as an lp, you're never going to be the expert in any one area. Right. You're talking to venture capitalists, you're talking to CTAs, right. You're talking to global macro funds. And I maintain if they can't define success, that's problem number one. Right. Two, if they're defining success and you finally get them to say, because we do have GPS that'll do this and that's sort of like the threshold for getting into our portfolio. They'll say, look, we're looking for 12 to 15% return on a rolling 18 month basis about 60% of the time. Okay. And then we're just watching them, you know, every 18 months and we roll it forward. Hey, are you hitting that, hitting that return 60, 60% of the time. And I maintain like you're, since because you're not an expert, you don't have to know everything that they're doing, but if they're miscalibrated and they say 60% of the time, we're going to hit that return threshold, but they're only hitting it 40% of the time. You don't have to know what they're doing, you just know that they don't know what they're doing because they wouldn't be that far off. And in fact, you see a lot of GPS who are overconfident. And that's a problem because if you go back to the expected value construction, if you think your success is 60% and it's actually 40, well then your, your equation has a negative expected value and you end up with performance disappointment.
- [33:07]
A
You're essentially forcing super forecasting on your gps.
- [33:10]
B
Yep, yep. Well, we will impute it and we will ask them directly, you know, so if it's a liquid fund, we'll certainly ask them that. If it's a private equity fund, obviously we're getting sort of like a whole. We'll kind of walk through the example I gave earlier. Right. But then you can also go portfolio company by portfolio company and you can say, hey, this portfolio company, where do you think revenue is going to end up 12 months from now? And they'll Say kind of here, and you say, well, how confident are you? Ah, we're 60%. Right. Okay. Portfolio company B, what do you think? Right. And then fast forward a year. You know, you can, you can start to compile some data on them and you can say, look, you guys are actually pretty well calibrated, or you seem to be routinely overconfident. And I just think LPs really want to work with groups who are appropriately calibrated because then they're sizing bets appropriately. And that's a little bit how we do it.
- [33:55]
A
If you could go back to 2007, you just left JP Morgan and joined PSTRS, what is one piece of timeless advice you'd give a younger mark that would have either accelerated your career or helped you avoid costly mistakes.
- [34:07]
B
The number one thing is to communicate to any young investor that most of what you look at will probably not do what you think.
- [34:15]
A
Right.
- [34:16]
B
So when we first started doing, when I stepped in as the CIO and we were forcing everyone on the team to assign sort of confidence levels to investments, the confidence levels were really high. 75% confident that this fund is going to achieve the success definition, 60, 80%. And it was a bit of an awakening. Let's say you're looking for top quartiles, private equity funds. Okay, well then that means by definition, most of the people across the table from you are not going to be there because only 25% of them can be and 75% of them will not be. So I, what I, what I hear from other investment teams or individuals, they sort of kind of go into these meetings thinking like, okay, this sounds like a good idea. I'm going to listen to them and then I'm going to try to poke holes in what they're saying. But that's coming from a position like, like you're giving them, you know, you're basically saying they're going to do what they're going to do and I have to find information that disproves it. Or they'll come in and say maybe, which is a little bit better. Like, I'm going to be on the fence. I'm kind of 50, 50, let's see if they can convince me. But it's completely different to sit there and say there's a 25% chance they're a top quartile fund. Most likely they will not be. Right. And they're really going to have to convince me. And I've got to point to very, very specific criteria, right. That would move me from 25 to, you know, say, you know, 50 or 60% confidence. And I just think that's a, that's a healthy frame, but I think it takes some time. But doing that. And again, starting with that outside view that most funds actually do not perform, I think is a much healthier way to approach investments. And when I was back in 2007, when I was starting, I didn't realize and understand that I do now. And that's one thing I would definitely beat into my 2007 self from the get go
- [35:56]
A
set slightly a different way. The market, when people look at historic returns, they look at an asset class and maybe returns 8% or 11%. They intuitively think it's just like this linear growth. But especially in the most spiky assets like venture capital, some years you're getting 40%, some years you're losing 18%. It's really quite chaotic back and forth. So what do you do with that? You do two things. One is you have to discount any kind of projections. So if you're trying to get top quartile, you probably need to pick a top decile fund. If you're going to pick a top quartile, the odds of it staying top quartile, according to University of Chicago Steve Kaplan, is 52%. But that also says 48% of the time you're not going to get that. So you have to be more ambitious in order to hit your target. And then secondly is when you're right, you better make a count. You better make sure that the asymmetry is there. Because if you have just an asset class that is going all over the place and the median and the mean is basically the same amount, then it's not a good asset class.
- [36:48]
B
So part of this is just sort of when we, when we start with this outside view and creating base rates, what you're really trying to do is kind of control for skill and luck, right? And so that's why it's really important to, to pin a potential partner down and ask them to define the success metrics. I'll use this analogy all the time. Like, you know, take the University of Michigan stadium, right? Biggest stadium in college football. If you had everybody stand up in that stadium, and if you flip heads, you stay standing, you flip tails, you, you sit down, you do that 15, 15 times, 20 times. I, you know, I don't know the math off top of my head, but it, you know, be 0.5 times 0.5 times 0.5, you'd probably end up with a handful of people Standing at the end of maybe, you know, 15 coin flips. But it's not like they did that intentionally. Right? But you can imagine, like, what happens. Like, imagine they were investment, you know, they were like venture capitalists or whatever. You know, they had some home runs. But what you don't know is whether that happened on accident or, you know, whether that was skill. Going back to my analogy, imagine that. Let's say there's five people. Imagine that those five people, or one of them said, I'm going to flip 15 heads in a row ahead of time right before it started. And they actually did it. Right. That would matter a lot more. So. So the. I think defining that success and the outcome beforehand really helps you interpret the results, because otherwise there's a lot of revisionist history. Right. And they'll take credit for anything that turned out really well. Right. And they'll sort of. They'll sort of just like, you know, uncritically accept that, and they'll assign bad luck to anything that didn't turn out well. So I think from the. From the jump, having that definition of success is just really important.
- [38:18]
A
Well, Mark, it's been two years since I've been trying to get you on the podcast, ever since you won Innovator of the Year for Institutional Investor. Congratulations on. Belated on that. And thanks so much for taking the time and for sitting down.
- [38:29]
B
Hey, no problem. This has been fun. Thanks for having me.
- [38:31]
C
If you found this conversation valuable, please click Follow How I invest so that you don't miss the next episode with the world's top investors.