Loading summary

Transcript144 lines

[00:00]
A
Our goal is to get to a billion people in our audience and then to be able to stratify and know what exactly is this person an expert on? And it might be, you know, even something like sneakers. You have some people who are influencers and kind of early adopters. And if you're able to find that audience and interview them first, the insights are much more valuable. And we can learn across all of the interviews that we do. We build profiles of people as we do more interviews in the platform, and then we can search and find the right person.
[00:51]
B
Okay, today we're sitting down with Alfred Walforce, founder and CEO of Listen Labs. Listen is an AI first customer research platform that can run thousands of voice interviews simultaneously. You launched about a year ago, and you now serve 20% of the Fortune 500, including iconic brands like Microsoft, Anthropic, Sweetgreen, NBC and others. And Konstantin, I'm very, very excited to sit down with you today and talk about market research and how it's getting transformed with AI.
[01:19]
A
Yeah, thank you for having me.
[01:21]
B
Maybe just to get started. So you are building an AI enabled platform that scales market research. What does that mean?
[01:26]
A
Yeah. So we have this AI agent that can understand your customers better than you can. And the way we do that is by talking to them. So to give you an example, you can ask a question like, how can you improve Curtis Onboarding? And then Listen will create an interview guide, which is an instructions for the agent to make the interviews. And then we have an audience. We have 30 million participants. We can find pretty much anyone from an oncologist to a software engineer. And we'll go and actually talk to them and have hundreds of those interviews and then analyze the data, give you recommendations. And now the final step that we're just launching in a couple months is simulation. So after you've done tens of thousands of interviews in the platform, can you predict how your customers will answer questions in the future? Put it another way, as we get closer to AGI, it will be easier to build things, but the hard part will know what to build. And that's what we're building at Lucent.
[02:22]
B
Awesome. Do you have any favorite customer stories?
[02:26]
A
Yeah. So Chubby's is one of our customers.
[02:29]
B
Like the Shorts brands.
[02:30]
A
Yeah, They've been like one of our early customers.
[02:32]
B
What they use you for?
[02:34]
A
They use us for everything. So a lot of marketing testing for testing shirts to understand what products perform well and what doesn't. And one of my favorite examples is they discovered that chest hair interface really poorly with one of the Materials they have. So it's like really uncomfortable to wear one of their shirts. And they changed the shirt and it became like radically more comfortable. So we solve the small things to the big things. Manscaped, Changed their Super bowl ad with insights from lesanne. So.
[03:07]
B
So never heard of that. But I'm not going to ask.
[03:11]
C
So you got the men's hair market cover.
[03:14]
A
Yes. That's our niche.
[03:15]
C
From shaping to clothing.
[03:17]
A
That's right.
[03:18]
C
Wow.
[03:18]
A
We do other things. Skims is one of our customers.
[03:23]
B
Don't know what you're talking about, but I know context clues, so that's awesome. I'd love to understand as you framed it, as we get closer to this AGI future, one of the questions I have is, you know, traditionally I've always been very skeptical actually of surveys because people get paid to take surveys. So you already got a selection bias issue. The things that people say they would do or the way that they describe how they would behave is different from how they actually behave in practice. And so I guess I come from the school of thoughts where like actual just telemetry in the real world matters so much more than asking people about what they would do. And so I'm curious what you think of that and how you think AI or listen labs can help bridge that gap.
[04:06]
A
Yeah, so we've done a lot of research on this. One of the things we've done with surveys, for example, is we went back to the same person and asked them a multiple choice survey again, and they were radically inconsistent. So even if you go back to the same person and ask them a survey question in a multiple choice fashion, they're much more inconsistent. But we did the same thing with listen, when you actually have to think and you have to really reason through your answer, and then you're much more consistent with at least how you answer the same question. And then we're constantly tracking. For example, with Chevy's, when we test their different charts, we a couple of months later look back and see how did that perform with the actual sales data. And I think it depends on different use cases. I agree that a B test is kind of the holy grail, but in practice it becomes really difficult to get right because you need a very large volume of users. And it's really useful to have some kind of input than no input at all.
[05:13]
C
Does listen do voice to text? As in the actual customer who's answering the survey can speak their answer and then you guys transcribe it? Does it also do text to voice? It's a two way Conversation. What does listen start with and what does it finish with? For the user experience?
[05:32]
A
Yeah. So it's essentially a zoom call that you have with the agent. So you're on video and you can also detect their emotions. So that's another way to bridge the gap between what they say and how they actually think and feel. So it looks at your eyes, the way you say it, and that's kind of much closer to how you actually behave in the real world.
[05:54]
C
And have you seen Persona's point that actually having the person's face and their emotions and their voice and whatnot yields more engagement, truthfulness? Have we been able to have any studies or at least data to point in that direction?
[06:13]
A
Yeah, specifically with advertising, it's a huge benefit because you might have people say on a Likert scale, which is like five questions that you click, are you extremely likely to buy this product versus when you. You might have very high scores on a survey question like that. But when someone also reacts very enthusiastically, it's going to be like performing much higher. And we've seen that those ads then perform better in performance marketing, for example, on meta and LinkedIn.
[06:48]
C
And can you, if you're the customer and you commission this and you get all this response, can you actually click in and if you ever wanted to watch the interview to get that level of granularity.
[06:59]
A
Yeah. So we built a platform around traceability so that for every data point, you can always click and then look at the video or see the quote. So. So you know that AI is not just hallucinating kind of where it's coming from.
[07:13]
C
That's awesome.
[07:14]
B
How'd you come up with the idea to build this?
[07:17]
A
So my co founder and I actually built a consumer app that did what that went viral. It was called bfix. So you could create an AI avatar of yourself. It was an early version of the ChatGPT images and you could fine tune stable diffusion and put yourself in that world. And that ended up going super viral. And overnight we had 20,000 users. And we were also kind of experimenting with different ways of using AI. So we built this AI interview for ourselves because we had a bunch of questions of how we had a ton of churn. So we wanted to understand why, how they thought about our positioning, different use cases, and it was really useful for us. And that's how we got started.
[08:01]
B
Maybe just walk us through how the industry is changing before and after. Listen Labs. Historically, let's say you're somebody with an app with 20,000 users. You understand how users are using the App what they want next, why they're turning. Historically, how did people go about doing that?
[08:17]
A
Yeah, so what we discovered was that there are these survey tools that are pretty old school, like Qualtrics, but then there's also this very large services industry because it becomes harder and harder, especially if you want to do market research where you want to talk to your prospective customers, not your current customers. It becomes harder and harder to do that as you scale. So that's a multi billion dollar industry. And what they do is come up with questions to ask, which is an academic subject in and of itself. It's actually really hard to know how do you ask questions to a point that get to how someone actually will behave? You can't just ask how much are you willing to pay for this. There's different methodologies that work better than others to finding the audience. How do you source the participants to then analyze hundreds of these calls? And in the traditional industries like cpg, even in Microsoft, they spend tens of millions of dollars on focus groups to bring people in a room and interview them. And we can help speed that up much faster.
[09:26]
B
Okay, so that's the old world of how this used to be done. Maybe describe the new world. And then it seems to me that there were obvious kind of first order benefits. Like there's probably much more scalable, probably much more cost effective, but there's probably also less obvious benefits. Maybe just talk about some of those benefits of what is it like when you actually do AI first market or customer research?
[09:50]
A
Yeah, so most decisions that gets made are not based on the customer input. Right. And the reason for that is it's just a lot of friction to even talk to customers. So when you can lower the barriers of talking to customers, you end up making much smarter decisions. So the speed advantage is actually huge for us. You can get input within five minutes from real people. And it's a really magical experience when you see hundreds of people populate in your interview. And so that's one thing. And because it's asynchronous, it's also much more affordable. So you can pay people much less than if you would have to run like synchronous interviews. So actually that's an interesting thing that people often ask us, do people even like being interviewed by an AI? And the objective answer is yes, because you can pay them less to talk to an AI than to talk to an actual interviewer.
[10:54]
B
Why is that?
[10:56]
A
I think it's mostly because it's asynchronous and people are very busy. But then Also what we found. Yeah, lower pressure. You can kind of go on and off. We've also found that people are more honest talking to an AI. We've had people really open up. It's a very therapeutic experience because it's a non judgmental entity that's really interested in you. And we can also have sensitive conversations like interviewing kids, how they react to different products. And so I think that's another advantage as well, that people can be brutally honest talking to the AI.
[11:33]
B
Okay. So historically, for example, if you want to do research on the kids market, very, very hard to access that market. Is that a regulatory thing? Is it a scheduling thing?
[11:40]
A
Yeah, you need parental consent. Kids are really busy, they go to school, they have extracurricular activities. How do you find time with them? And you need to find the right kind of kids. One of the things we realized is the audience is extremely important. And that's actually where we spend 80% of our engineering resources. Every company is driven by a power law in customer segmentation. So even a product like sweetgreen, which you would think is for everyone, the right audience is typically urban, high household income, mostly female. And by the way, they need to know what seed oils are, which only like 1% of the population does. And then you find that some people go to Sweetgreen every single day and that's 80% of their revenue. So if you can find that segment, the research is so much more actionable.
[12:35]
C
Yeah, there's probably a network effect to it as well, where when you get of a certain scale and people use it, you can access the same kind of person that otherwise might be really difficult to access. Or maybe it's a scale economy, something along the lines of accessing those really, really specific people that are really valuable for the type of product that you're trying to introduce.
[12:57]
A
Yeah, it's really about, you know, we have. Our goal is to get to a billion people in our audience and then to be able to stratify and know what exactly is this person an expert on? And it might be even something like sneakers. You have some people who are influencers and kind of early adopters. And if you're able to find that audience and interview them first, the insights are much more valuable. And we can learn across all of the interviews that we do. So we build profiles of people as we do more interviews in the platform and then we can search and find the right person.
[13:36]
C
So someone might say, in a totally unrelated interview, I'm a total sneaker head. And you can keep that in the database on that person. And then when Nike or what have you is launching their new product line, you can offer that person up.
[13:47]
A
That's right.
[13:48]
C
That's amazing.
[13:49]
A
And that was not possible to do before because it was usually like separate entities. And it would be a very manual process where you would have an email list and you would just spam email.
[14:00]
C
I've been on the receiving end of this. Yeah, they're terrible.
[14:03]
A
And one of the problems with that is that you then need to have a extensive screening process. So you have something called an incidence rate, which can be 10%, which means only 1 in 10 people gets qualified to even take the interview. And that causes significant churn on these databases because it's really annoying to.
[14:26]
C
Super annoying.
[14:27]
A
Green out 10 times to even get paid the first time.
[14:30]
B
Why do these brands even need you for access if. Let's take Sweetgreen. Sweetgreen knows who that 80% is. Can't they just reach them? Don't they have direct relationships with them already?
[14:40]
A
Yeah, so they can. And we do that as well. We connect to their CRM and they can send that out. But then the really interesting part is how do you talk to prospective customers, people who also may not be kind of current power law users? And how do you compare those two? And then also what we found is that the CRM is typically really unorganized and sometimes there's also like regulatory issues. If you're at Google, you can't just send emails to people who use Gmail. And it gets much easier to use an external third party and you run
[15:18]
C
the risk of spam, which can get you totally blocked. I have seen that at some of our companies over the years where, you know, you do outbound and then eventually you're in the Google filter and next thing you know, you're in Microsoft purgatory, I guess, going through you guys, you don't have to deal with that.
[15:33]
A
Yeah, exactly.
[15:34]
C
It's cool.
[15:35]
B
What does this mean for the McKinsey or whoever is of the world that are, you know, the people that are building the hundred slide decks that, you know, reach 3,000 people to reach some set of insights?
[15:45]
C
Did you do that, Sonia?
[15:46]
B
Wasn't that former life, you know? No, Konstantin. But I'm glad that that's what you think of me.
[15:51]
C
Isn't that what banking does?
[15:55]
B
We. We hired cons. I used to hire these people.
[15:59]
C
Got it, Got it.
[16:00]
B
So I was, you know, I was a layer on top of the. Layer on top of the layer.
[16:03]
C
Okay, got it.
[16:03]
B
Okay. I was even more redundant. But what does it mean for all these people, do they still have a role to play in this new future?
[16:11]
A
Yeah, I think AI is changing all roles very quickly. And we work a lot with Bain, for example, so they use us to speed up their traditional processes. And I think they still have a role to play. I think traditional services and being able to then implement these changes is still extremely valuable. But a lot of margins are going to drop and you have to make sure you kind of unbundle a lot of your services to maybe allow for AI agents to help solve some of the problems that you would go to traditional consulting firms before.
[16:52]
C
Maybe I'm an optimist here, but why wouldn't it be more? Why wouldn't I, if I'm running a business, say, oh, great, I want to find five new areas to expand to now that I have AI and these tools. And I will pay you, Bain or what have you, the same dollars you use. Listen and just explore those new areas and tell me where to. Where to build. Is that overly optimistic?
[17:15]
A
No, I think it's one of those areas where the ceiling is very high. You can kind of learn more about your customers and you can build more things. And so I think, I think you're right.
[17:27]
C
I was thinking the chest hair shirt thing
[17:32]
B
that 20 minutes. Still thinking about chest hair.
[17:35]
C
There's so many little things that I'd love to tell the companies that I am a consumer of, like the smallest little. Even the way they laced these shoes. I'd love to give that feedback.
[17:46]
B
This is why you're a venture capitalist.
[17:48]
C
Details, details.
[17:51]
A
We hope to live in a world that finally works the way people want.
[17:54]
C
That would be great.
[17:55]
B
Please. Are you seeing any price and compression already hit the industry like I would imagine if I am Bain's customer, I'm thinking, well, you're able to do this survey a lot more efficiently now with AI than before AI. Who. Who's getting the benefit of that economic surplus?
[18:13]
A
So because you're able to do it faster, I would argue you should be able to charge more for it.
[18:18]
B
And is that what's actually playing out?
[18:20]
A
We have done some studies where we're able to charge hundreds of thousands of dollars to speak to 20 doctors across eight countries. So maybe over the long term, the individual interview will become more affordable. But I think you'll be doing kind of two orders of magnitude more of research. And I think what's really exciting is also simulation, which is something we're building now, where you're able to unlock the 99% of use cases where you would never have time to talk to real people.
[18:54]
C
I think that's so awesome in part because there are so many areas where they don't even listen to the customer. Like medicine. There are a million little problems with the medical system. I hear about it all the time. And these are, you know, their doctors are busy, important people. But it feels like the companies haven't even invested the time in figuring out where all those paper cuts are and the doctors are really busy so they're not going to go schedule an appointment and have some long conversation and meet with some group. But if they could do it at any time like in an app on their phone as part of the normal homepage app and give feedback on their EHR or something in the operating room or something along those lines that seems like a life saving use case for listen over time.
[19:41]
A
Yeah, I think what I'm really excited about as well is taking all those small things and then telling another agent to go and solve that problem. And we're getting pulled in this direction by some of our customers where they will have a churn interview and then they will connect. If you find a bug for example, they'll connect that to another coding agent to go and solve the problem.
[20:05]
C
That's cool.
[20:06]
B
Let's talk about Generative Agent simulation. It seems like the entire industry has gone From Market Research 1.0 call 100 people one by one, collate them manually to Market Research 2.0 ainative where AI designs the question track is able to talk to thousands of people simultaneously synthesize the answers. It seems like we're maybe moving to Market Research 3.0 with Generative Agent simulation. What do you make of that? And I both see the dream of it. I see how synthetic data has changed for example self driving cars. And then I also am inherently skeptical of it. Is a bunch of synthetic data just remixing what's already in the pre training set and are you actually learning anything useful or with alpha there? And so I'd love to hear your take on it and how you guys are taking on the three level and
[20:55]
C
maybe what is it too to start.
[20:57]
A
Yeah. So the way we are building simulation is by interviewing a single person. Say if I interview Konstantin for one hour I can probably start to predict your preferences to some degree.
[21:11]
C
Fascinating insights about chest hair.
[21:15]
A
And it turns out that LLMs are quite good at this as well. So you can essentially try to feed in as much information as possible on a single individual and then in some cases we're able to get 95% accuracy to predict how they will answer certain questions.
[21:30]
B
What sorts of things are you finding that you can predict well versus can't?
[21:35]
A
One of the most useful things is message testing. So that's the idea of like what's the tagline on the billboard? Or I was actually using it this weekend. So I have created a panel of our customer base and I had to come up with the title for a talk at a conference. And it's like a small thing, but it actually does matter because it will increase conversion if people show up. And I came up with 100 different titles for my talk and inputted that into our simulation. And then the top talk was like twice better than the next one.
[22:14]
C
Wow, cool.
[22:14]
A
And I don't know if it's correct, but it certainly felt correct. And it was really helpful to have guidance in making that decision. And I also think even if it's wrong, it's just nice to have some help in making a decision. It's also nice to outsource your decisions.
[22:32]
B
And how does it compare to just asking ChatGPT the same thing?
[22:35]
A
Yeah. So then I inputted the same questions into ChatGPT and actually I had another talk, I did that that was not so successful and I inputted a competitor or another talk that was more successful and I showed both of them to ChatGPT and both of them to our simulation. And in ChatGPT it picked the wrong one and in our simulation it picked the right one. So it's early for us. We're going to release this in a couple of months, but it seems like it's performing better than the general models. And the models are trained on the average person and you want to build for a very specific niche. And that's how we can kind of essentially train the models to follow that
[23:20]
B
niche and just to push on this because I think it's so fascinating, can't you kind of force the models into a specific niche or personality? Like, hey, ChatGPT, you're a 35 year old, really grumpy software engineer that likes using your terminal and then it does then take on the preferences of that niche is sort of my mental model at least. And so I'm actually surprised that ChatGPT wasn't able to arrive at the right answer. And then bootstrapping off real user data was because ultimately it all is kind of a reflection of real user data. Right. And so actually what is the intuition for why kind of SIM only unpretrained data isn't sufficient?
[24:03]
A
Yeah. So we've tried many different inputs and that certainly performs A little bit better than just vanilla chatgpt. But what performs much better is we've tried credit card spend kind of behavioral data purchasing behavior. But what we found was the best data set is interviews because it's more kind of allows you to go off tangents. It understands you can ask behavioral questions. So also it can't just be any interview. Like the way you design the questions is also really important. And the intuition, I think, is that the models don't have clean data on how a specific Persona acts and how they think.
[24:49]
C
It's anecdotal, but it makes perfect sense. Because if you want to understand someone, what better way to understand them than asking them a lot of questions? That's why we're all here. It's kind of the purpose of this type of format and if you have enough people that follow a certain group as opposed to the average, that can tell you a lot about other things that they might not have explicitly said. All of AI is this generalization of some sort of compressed data of some sort. And so if you have this compression in a slightly different part of this hyperspace that you say now complete this orbital of what everybody is thinking in this category of person, you know, Listen can fill that out because it has enough interviews.
[25:30]
A
Yeah.
[25:31]
C
Do you think that you will offer that package as a product? As in if I wanted to understand my customer, and for me, for us, our customers, founders, they're very different though, so extremely different people. If I wanted to understand my customer, could you do active interviews? The normal Listen lab interviews have 1,000 or 10,000 cumulatively and then offer a little special purpose Listen labs bot that then I can use instantaneously for any ad hoc question.
[26:04]
A
Yeah, that's exactly what we have.
[26:06]
C
Okay.
[26:07]
A
So that's what we call augmented responses. The cool part of this as well is that it can also live in your coding agents or your other agents. So I think in the future you will want to have almost a human API which where the agents are able to call the preferences of your users to be able to know like what to build, how to do it or who to invest in or how to help them the best today.
[26:31]
C
Is it all rag? Is it fine tuned? Is it something else? How do you take those conversations and then combine them with the models that you're doing the rest of the Listen labs with?
[26:43]
A
Yeah, we are doing post training typical RAG as well. There's like a bunch of different techniques. Some of them are proprietary, but yeah.
[26:54]
C
All right, we'll do customer interviews on all your engineers report Back.
[26:58]
B
I'm curious what you think of multi agent systems and their role in helping us kind of iteratively use, you know, at inference time iterate to a better answer. Is that part of how you're doing simulation or not?
[27:13]
A
Yeah, like the way we do simulation is essentially you have one person that you model really, really well and then you scale it up with a thousand people so you have a representative sample and it's essentially multi agent.
[27:29]
B
But you're not having those thousand people debate each other. That's what I'm asking.
[27:32]
A
Oh yeah, no, we don't have that yet, but that's a good question.
[27:34]
B
Do you think that would help?
[27:35]
A
Potentially, but there are these other competitors that are doing that approach more. I think the worry is that again, chaos theory tells us when things kind of compound, it becomes really hard to predict how the things are going to interact with each other. And it's something we definitely should explore more. But I'm a little bit skeptical of the approach.
[28:01]
B
Maybe the analogy I'd make is the AI council approach of send the same query out to three different LLMs and then have one LLM act as judge in synthesizing them. I do think on average you get a slightly better response.
[28:13]
A
Yeah, cool.
[28:14]
B
So where else do you see yourself going from here? Then you're going from Market Research 2.0 to Market Research 3.0. Now with kind of generative simulation, do you expect that 3.0 takes over as the majority of queries over time and then what else is ahead?
[28:30]
A
Yeah, I think you'll still need human input, but I think there would be many more use cases that are now opened up where you can get customer input. So for the large decisions, if you're doing a Super bowl ad or things like that, you will still need to run real interviews. But for the smaller things, like what should be the tagline for your billboard? If it's a small billboard, then you can use simulation to answer that. And I still think that there's a lot of alpha on the core product as well to improve. I mean, when we started, the core idea was just making the interview less annoying to go through. Like we had an eval that looked at repetitive questions or looked at is the AI even able to follow the instructions? And with GPT4, sometimes we would ask the same question 100 times. And in the beginning that eval was like 20%. Now we've been able to climb that eval to be 85%. But now we created a new eval that's much more advanced. So it's able to understand what are you doing on your screen when you're screen recording or can you skip questions that are not relevant anymore. And now we're back at like 20%, which I think is one of the values that vertical AI companies can have is that they have this proprietary eval that they can use and essentially climb that eval. And that's your advantage as a vertical AI company.
[30:08]
C
Keep pushing forward. Better data, harder problems, better data, repeat.
[30:12]
B
It seems to me like you're in the middle of a very interesting infinity loop. Right. Because I mean fundamentally a company is figure out what to build, build it, figure out what to build, build it,
[30:23]
A
write code and talk to users.
[30:24]
B
Exactly. And, and the build it is coming up rapidly up in exponential. And the figure out what to build is the thing that you are pushing forward. And then it's not only even outside of product and engineering, the broader loop is actually strategy execution. Strategy execution. And so much of what AI is enabling us to do is it's making execution faster, cheaper, better, all these things. And the thing that you guys fundamentally are positioning yourselves to do as a company is the strategy part from what to build to what to say. Is that a fair synthesis?
[30:59]
A
Yeah. And I think when we have that one person billion dollar company, we'll be part of that loop. So I have a coding agent and listen and then run that in a loop and we'll have these autonomous organizations,
[31:16]
C
even the big companies still back to this idea of you can implement things faster. Let's say you have an agent. I mean if you can be a big company and we're talking in software because software is native to us. But in software, if you could talk to a customer, figure out a bug, create a pr, have a coding agent, close it, chip it, customers happy, that seems like a really important left hand side of the equation. Find the bug from an actual human. But I imagine it's the same thing in a big atoms company. Like if you're consumer packaged goods, if you're clothing, if you're any of those things, I imagine that's even more important because you have to figure it out because once you actually do the thing, it's done.
[31:58]
A
Yeah, exactly. Like Procter and Gamble when they're launching in a new market that can be tens of millions of dollars, if not more. And you have to make sure that that is right when you launch. And that's one of the reasons why they're big customers of Listn.
[32:12]
C
Who has done this historically really well. Like who are the companies that are admired in history have done a great job of listening to their customers either in the, you know, consumer space or in the software space.
[32:30]
A
I think Procter and Gamble is kind of the archetype of the best market research organization where they're essentially marketing companies that are trying to figure out what are niches that people really care about and then build specific brands to solve those problems. I mean, one example is the Tide washing machine pods. Pods that they were able to figure out that it was really uncomfortable to use the washing liquid. And they discovered that people wanted something that was much more easy to use. And through customer interviews, they found this insight, made this new ipod and became really successful. Another example, which is in the acquired podcast when they talk about Mars, they did the first one of the first market research studies in the 1950s where M& Ms. Were originally designed for being used in the army because they were like, sweet treat, they don't melt in pocket. And they discovered through market research that another great segment was young kids. And they then decided to pivot the entire advertising strategy to focus on this because it doesn't melt and ruins your furniture, for example. And things that like, like that.
[33:54]
B
As we progress towards this, you know, listen, labs future, future vision of the world. What are the things that you're confident will work and what are the things that you're still not sure about?
[34:04]
A
I'm confident that in the future you'll still need to have human input, because even if you have a perfect rational being like AGI, humans are still irrational and they will still want kind of be chaotic in their nature where they all of a sudden get obsessed with a new product, you know, a new TikTok trend that shows up and you have to change your entire marketing strategy towards that. And so I think that will remain a really huge part of how we do things. I think. I'm still uncertain of what level simulation will play, and I'm confident that it will work for certain questions. But we'll see how, like, how good the models get to predicting human behavior.
[34:54]
C
I mean, I'd imagine that it's actually even more important the better AI gets to have the delta because the competition. If companies are about serving people, which I think we can all agree on, like, at the end of the day, every company is about serving humans.
[35:10]
B
Constantine, our resident humanist.
[35:13]
C
I'm a humanist, absolutely. But if companies are about serving people, because that's what I. That's why we're all working, is to help someone else in some way. And intelligence gets better and better and better, and you kind of have what the human wants here, and the intelligence is approaching that asymptote, then the delta in that asymptote, which is what is in a human's mind that isn't in the AI's mind, only becomes more important.
[35:34]
A
Yeah. And yeah, one of the things that we've also realized is that there's a lot of talk around what's the mode of these AI vertical AI companies and.
[35:46]
B
Yeah, what's your moat?
[35:48]
A
What is our moat?
[35:49]
C
We've got network effects and scale economies.
[35:51]
A
Yes, those are nice.
[35:52]
B
We're on an episode of Acquired right now.
[35:54]
C
Hey, I'm feeling it. I'm feeling it right now. It's a good book. Recommended. Seven Powers.
[35:59]
A
Yeah, Seven Powers.
[36:00]
C
Love.
[36:01]
A
Love. Seven Powers. Yeah. On the moats, I mean, we have the clear modes, which are the network effects on the panel where you have supply and demand dynamic. We also have the network affects the data mode. As we do more interviews, you get better simulation. And then the product is very sticky because you have all these interviews in your platform and you don't want to lose that. You want to track things over time. But even like the simplest things, I think in terms of product advantages, one of the first things that Brian Shire said, one of our sequel partner, was that founders want to build something that's complex, but customers want something that's stupid simple and it just works. They don't want to configure their own workflow. They don't want to sit and build custom software. And just one example of this is creating the interview guide is really difficult. It's actually an academic subject and it's one of the reasons why you have services firms because they know what methodology to use. If you want to understand pricing or brand perception, these kind of things, you
[37:14]
C
don't want to lead the witness.
[37:15]
A
You don't want to lead the witness. And it's really hard to get that right. In the beginning, we just used the vanilla LLM models and the customers would create interviews, they would get the data back, and then they come back to us really frustrated, saying, like, yeah, what is this? Like, I can't use this data for anything. And we took the blame for that. Now we've trained it to follow the best practices so that you always get good data out of the interviews. And I think that's the advantage you have as a vertical AI company, that you can essentially train this agent to follow best practices and the work that you do.
[37:54]
B
So I want to go back to the concept of tide ponds that you had mentioned earlier. I think it's really interesting. And so much of market research, as I understand it today, is almost more inviting people to pass judgment on ideas that you feed them. But it seems to me that one of the, you know, hallucinations can be a bug. They can also be a feature with generative AI. And, you know, do you think we're going to see user research actually evolve into live product ideation? I could almost imagine, you know, AI inventing solutions as customers are going about their interview process, even helping visualize those solutions. Are your customers doing that already, or do you think we're going to have the moment where I kind of can create a Tide Pods idea in a market interview anytime soon?
[38:41]
A
Yeah, I think that's really exciting. Today they do that manually, you know, use AI to generate images of different concepts and feed that into the interviews. But I think specifically also with simulation, it becomes really powerful. So we now have an MCP as well, so that you can feed that into Claude and then you can tell Claude like, hey, run, listen in the loop, and then come up with a bunch of ideas for how to market something or different concepts and then you can have it run like that.
[39:15]
B
I'm even thinking in the course of an interview as somebody's complaining about, you know, the Tide, I can't sound very portable for the AI to be live brainstorming with you solutions, not just.
[39:26]
C
This is what it could look like an image generator too. Yeah, that'd be cool. Sonia?
[39:30]
A
Yeah, I think it's a good idea. You should be on our product team.
[39:34]
B
Awesome. Well, Alfred, we really love what you're building. Thank you for taking the time to share insights both on the broader market, which I think is just so fascinating, and also what it takes to be building in the application layer right now. We really admire the business that you've built and thank you for your continued partnership.
[39:50]
A
Thank you so much,
[40:14]
B
Sam.

Summary8 min read

Podcast Summary: Training Data – "Knowing What Your Customers Want, All the Time: Listen Labs' Alfred Wahlforss"

Podcast: Training Data
Host(s): Sonya Huang, Pat Grady, and Sequoia Capital Partners
Guest: Alfred Wahlforss, Founder/CEO Listen Labs
Date: June 2, 2026

Episode Overview

This episode delves into the transformative role of artificial intelligence in customer and market research, featuring Alfred Wahlforss, founder and CEO of Listen Labs. Listen Labs is an AI-first customer research platform capable of running thousands of voice interviews simultaneously. The conversation explores the evolution of market research, the impact of simulation and generative AI, the power of network effects, and the platform’s vision for the future of customer input. The hosts press for insights on AI’s technical, practical, and economic implications for businesses, consulting, and society at large.

Key Discussion Points & Insights

1. What Listen Labs Is and How It Works

Listen Labs employs AI agents to conduct nuanced customer interviews at scale, leveraging a vast panel of 30 million participants.
The platform builds detailed user profiles over multiple interviews, enabling highly targeted, valuable insights—even down to niche interests like sneakers.
Alfred Wahlforss [00:00, 12:56]:
“Our goal is to get to a billion people in our audience and then to be able to stratify and know what exactly is this person an expert on? ... If you're able to find that audience and interview them first, the insights are much more valuable.”
Simulation: Listen Labs is moving toward predicting customer responses based on extensive interview data—pushing into the domain of AI-powered market simulation. “The final step that we're just launching in a couple months is simulation… can you predict how your customers will answer questions in the future?” [01:25]

2. Real-World Impact and Customer Stories

Early Successes:
- Helped apparel brand Chubby’s identify that “chest hair interface” was uncomfortable with a shirt material, informing a product redesign.
  “They changed the shirt and it became like radically more comfortable.” [02:33]
- Insights from Listen Labs led Manscaped to change their 2024 Super Bowl ad.
- Listen Labs also works with brands like Skims, Microsoft, Anthropic, Sweetgreen, and NBC.
Tangible Feedback Loop: The platform tests its AI-derived insights against actual business outcomes, such as sales data.
“We a couple of months later look back and see how did that perform with the actual sales data.” [04:05]

3. Why AI Interviews Are Different from Traditional Surveys

Inconsistency in Traditional Surveys:
“We went back to the same person and asked them a multiple choice survey again, and they were radically inconsistent… with Listen, when you actually have to think and you have to really reason through your answer, and then you're much more consistent.” [04:05]
Voice/video input:
- Interviews resemble a Zoom call for a richer experience, including emotional cues.
  “It looks at your eyes, the way you say it, and that's kind of much closer to how you actually behave in the real world.” [05:31]
Honesty and Comfort: Participants are more honest and open with AI—sometimes even finding the experience therapeutic.
“We've also found that people are more honest talking to an AI. We've had people really open up. It's a very therapeutic experience because it's a non judgmental entity that's really interested in you.” [10:55]

4. The AI-Driven Future of Market Research

Traditional Approach vs. Listen Labs

Old model: labor-intensive surveys, expensive focus groups, and lots of friction to talk to actual or prospective customers.
Listen Labs model: scalable, asynchronous, affordable, and accessible to hard-to-reach populations (ex: kids, busy professionals). “When you can lower the barriers of talking to customers, you end up making much smarter decisions. So the speed advantage is actually huge for us.” [09:50]

Importance of Audience Stratification

Identifying and interviewing the right people is critical—many purchasing patterns follow a power-law curve where a small subset drives most revenue.
Alfred: “It's really about, you know, we have… to be able to stratify and know what exactly is this person an expert on?” [12:56]
Network effects and scale economies emerge as the panel grows—a differentiator for Listen Labs in accessing even niche experts or early adopters.

Platform Traceability

Every insight is linked to a source (video, quote) to avoid AI hallucination and add trust to recommendations.
“You can always click and then look at the video or see the quote. So… AI is not just hallucinating where it's coming from.” [06:58]

5. Implications for Consulting & Industry

Shifting the Value Chain:
- AI reduces margins for traditional consulting, requiring them to unbundle services and adopt tools like Listen Labs. “A lot of margins are going to drop and you have to make sure you unbundle a lot of your services to maybe allow for AI agents to help solve some of the problems that you would go to traditional consulting firms before.” [16:11]
Faster, More Accessible Research:
- Pricing models are changing, but demand for deeper, more frequent insights is increasing. “You'll be doing two orders of magnitude more of research.” [18:19]
Simulation & Automation: In the future, customer interviews may trigger direct actions (ex: bug identifications routed to coding agents), accelerating the “execution” loop in business operations.
“Some customers… will have a churn interview and then… connect that to another coding agent to go and solve the problem.” [19:40]

6. Market Research 3.0 & Agent-Based Simulation

Market research is moving from 1.0 (manual), to 2.0 (AI-aided), to 3.0 (generative agent simulation).
Generative Simulation: Models can predict respondent behavior with 95%+ accuracy in some cases, notably for tasks like message testing. “If I interview Konstantin for one hour I can probably start to predict your preferences to some degree… in some cases we're able to get 95% accuracy to predict how they will answer certain questions.” [20:57, 21:14]
Superiority Over General LLMs: Niche-trained simulations outperform vanilla ChatGPT on targeted queries due to more specific, high-quality training data (active interviews). “In ChatGPT it picked the wrong one and in our simulation it picked the right one… the models are trained on the average person and you want to build for a very specific niche.” [22:34]
‘Augmented Responses’ Product: Users can deploy simulated customer bots for instant feedback—usable by coding agents, marketers, and more. “That's exactly what we have… we call [it] augmented responses… you will want to have almost a human API.” [26:04]

7. Multi-Agent Systems, Challenges, and Future Directions

Listen Labs currently models large representative samples, not multi-agent debates; skepticism remains about emergent behaviors in complex agent interactions. “Chaos theory tells us when things kind of compound, it becomes really hard to predict… I'm a little bit skeptical of the approach.” [27:34]
Human input remains critical, particularly for large decisions (Super Bowl ads, product launches), but agent simulation unlocks new small-scale, rapid experiments.
Continual product improvement is a core advantage for Listen Labs:
“We had an eval [evaluation] that looked at repetitive questions or looked at is the AI even able to follow the instructions… we've been able to climb that eval to be 85% … vertical AI companies… have this proprietary eval.” [28:30 – 30:07]

8. Moats, Network Effects, and Product Philosophy

Moats:
- Network effect of the panel, data accumulation for simulation, and product stickiness due to stored interviews and longitudinal data.
- Simplicity is key for product adoption:
  “Founders want to build something that's complex, but customers want something that's stupid simple and it just works.” [36:00]
- Creating effective interview guides is a difficult, often under-appreciated advantage. “We took the blame for that. Now we've trained it to follow the best practices so that you always get good data out of the interviews.” [37:13]

9. Generative AI for Live Product Ideation

Future potential: Market research tools that not only reflect user judgment but actively co-create solutions with users during interviews (e.g., live brainstorming with image generation and prototyping). “I could almost imagine, you know, AI inventing solutions as customers are going about their interview process, even helping visualize those solutions.” [39:15]

Notable Quotes & Memorable Moments

On differentiation:
Alfred Wahlforss [00:00]: “Our goal is to get to a billion people in our audience and then to be able to stratify and know what exactly is this person an expert on?”
On scalability and affordability:
Alfred [09:50]: “You can get input within five minutes from real people... and because it's asynchronous, it's also much more affordable.”
On honesty with AI:
Alfred [10:55]: “We've had people really open up. It's a very therapeutic experience because it's a non-judgmental entity that's really interested in you.”
On product feedback leading to real change:
Alfred [02:33]: “[Chubby’s] discovered that chest hair interface really poorly with one of the materials … They changed the shirt and it became radically more comfortable.”
On simulation accuracy:
Alfred [21:14]: “In some cases, we're able to get 95% accuracy to predict how they will answer certain questions.”
On competitive moats:
Alfred [36:00]: “We have the clear moats, which are the network effects on the panel ... and then the product is very sticky because you have all these interviews in your platform.”
On vertical AI advantage:
Alfred [37:13]: “You can essentially train this agent to follow best practices and the work that you do.”

Important Timestamps/Segments

What Listen Labs does: [01:25–02:22]
Customer stories (Chubby’s, Manscaped): [02:26–03:17]
Surveys vs. AI interviews honesty/consistency: [04:05–05:14]
Video/voice interview UX: [05:31–06:58]
Market research before/after Listen Labs: [08:00–09:50]
Advantages of AI-first research: [09:50–11:32]
Audience stratification & importance: [11:40–13:36]
Accessing the right segments: [13:36–14:59]
Implications for consulting (McKinsey/Bain/TAM): [15:34–17:27]
Simulation & future market research: [18:54–20:04]
Agent-based simulation explained: [20:55–21:34]
Simulation accuracy vs. ChatGPT: [22:34–24:03]
Augmented responses/human API: [26:04–26:31]
Product moats & vertical AI: [36:00–37:54]
Generative AI co-ideation future: [38:40–39:29]

Closing Thoughts and Future Vision

Alfred and the hosts conclude that as AI accelerates the “build” side of business, understanding “what to build”—deeply, quickly, and affordably—will be the new competitive frontier. Listen Labs positions itself at the crux of this shift, using proprietary AI and simulation tools anchored in rich behavioral data. The importance of a massive engaged panel, vertical specialization, and a constant loop of product improvement and evaluation are emphasized as key to winning in the next generation of market research.

Alfred Wahlforss [34:04]:
“I'm confident that in the future you'll still need to have human input, because even if you have a perfect rational being like AGI, humans are still irrational and they will still want kind of be chaotic in their nature...”

For listeners new to the topic, this episode offers a wide-ranging, engaging look at the intersection of AI, customer research, and business strategy, with real-world examples and a candid, optimistic tone throughout.

Loading summary

Transcript144 lines

[00:00]
A
Our goal is to get to a billion people in our audience and then to be able to stratify and know what exactly is this person an expert on? And it might be, you know, even something like sneakers. You have some people who are influencers and kind of early adopters. And if you're able to find that audience and interview them first, the insights are much more valuable. And we can learn across all of the interviews that we do. We build profiles of people as we do more interviews in the platform, and then we can search and find the right person.
[00:51]
B
Okay, today we're sitting down with Alfred Walforce, founder and CEO of Listen Labs. Listen is an AI first customer research platform that can run thousands of voice interviews simultaneously. You launched about a year ago, and you now serve 20% of the Fortune 500, including iconic brands like Microsoft, Anthropic, Sweetgreen, NBC and others. And Konstantin, I'm very, very excited to sit down with you today and talk about market research and how it's getting transformed with AI.
[01:19]
A
Yeah, thank you for having me.
[01:21]
B
Maybe just to get started. So you are building an AI enabled platform that scales market research. What does that mean?
[01:26]
A
Yeah. So we have this AI agent that can understand your customers better than you can. And the way we do that is by talking to them. So to give you an example, you can ask a question like, how can you improve Curtis Onboarding? And then Listen will create an interview guide, which is an instructions for the agent to make the interviews. And then we have an audience. We have 30 million participants. We can find pretty much anyone from an oncologist to a software engineer. And we'll go and actually talk to them and have hundreds of those interviews and then analyze the data, give you recommendations. And now the final step that we're just launching in a couple months is simulation. So after you've done tens of thousands of interviews in the platform, can you predict how your customers will answer questions in the future? Put it another way, as we get closer to AGI, it will be easier to build things, but the hard part will know what to build. And that's what we're building at Lucent.
[02:22]
B
Awesome. Do you have any favorite customer stories?
[02:26]
A
Yeah. So Chubby's is one of our customers.
[02:29]
B
Like the Shorts brands.
[02:30]
A
Yeah, They've been like one of our early customers.
[02:32]
B
What they use you for?
[02:34]
A
They use us for everything. So a lot of marketing testing for testing shirts to understand what products perform well and what doesn't. And one of my favorite examples is they discovered that chest hair interface really poorly with one of the Materials they have. So it's like really uncomfortable to wear one of their shirts. And they changed the shirt and it became like radically more comfortable. So we solve the small things to the big things. Manscaped, Changed their Super bowl ad with insights from lesanne. So.
[03:07]
B
So never heard of that. But I'm not going to ask.
[03:11]
C
So you got the men's hair market cover.
[03:14]
A
Yes. That's our niche.
[03:15]
C
From shaping to clothing.
[03:17]
A
That's right.
[03:18]
C
Wow.
[03:18]
A
We do other things. Skims is one of our customers.
[03:23]
B
Don't know what you're talking about, but I know context clues, so that's awesome. I'd love to understand as you framed it, as we get closer to this AGI future, one of the questions I have is, you know, traditionally I've always been very skeptical actually of surveys because people get paid to take surveys. So you already got a selection bias issue. The things that people say they would do or the way that they describe how they would behave is different from how they actually behave in practice. And so I guess I come from the school of thoughts where like actual just telemetry in the real world matters so much more than asking people about what they would do. And so I'm curious what you think of that and how you think AI or listen labs can help bridge that gap.
[04:06]
A
Yeah, so we've done a lot of research on this. One of the things we've done with surveys, for example, is we went back to the same person and asked them a multiple choice survey again, and they were radically inconsistent. So even if you go back to the same person and ask them a survey question in a multiple choice fashion, they're much more inconsistent. But we did the same thing with listen, when you actually have to think and you have to really reason through your answer, and then you're much more consistent with at least how you answer the same question. And then we're constantly tracking. For example, with Chevy's, when we test their different charts, we a couple of months later look back and see how did that perform with the actual sales data. And I think it depends on different use cases. I agree that a B test is kind of the holy grail, but in practice it becomes really difficult to get right because you need a very large volume of users. And it's really useful to have some kind of input than no input at all.
[05:13]
C
Does listen do voice to text? As in the actual customer who's answering the survey can speak their answer and then you guys transcribe it? Does it also do text to voice? It's a two way Conversation. What does listen start with and what does it finish with? For the user experience?
[05:32]
A
Yeah. So it's essentially a zoom call that you have with the agent. So you're on video and you can also detect their emotions. So that's another way to bridge the gap between what they say and how they actually think and feel. So it looks at your eyes, the way you say it, and that's kind of much closer to how you actually behave in the real world.
[05:54]
C
And have you seen Persona's point that actually having the person's face and their emotions and their voice and whatnot yields more engagement, truthfulness? Have we been able to have any studies or at least data to point in that direction?
[06:13]
A
Yeah, specifically with advertising, it's a huge benefit because you might have people say on a Likert scale, which is like five questions that you click, are you extremely likely to buy this product versus when you. You might have very high scores on a survey question like that. But when someone also reacts very enthusiastically, it's going to be like performing much higher. And we've seen that those ads then perform better in performance marketing, for example, on meta and LinkedIn.
[06:48]
C
And can you, if you're the customer and you commission this and you get all this response, can you actually click in and if you ever wanted to watch the interview to get that level of granularity.
[06:59]
A
Yeah. So we built a platform around traceability so that for every data point, you can always click and then look at the video or see the quote. So. So you know that AI is not just hallucinating kind of where it's coming from.
[07:13]
C
That's awesome.
[07:14]
B
How'd you come up with the idea to build this?
[07:17]
A
So my co founder and I actually built a consumer app that did what that went viral. It was called bfix. So you could create an AI avatar of yourself. It was an early version of the ChatGPT images and you could fine tune stable diffusion and put yourself in that world. And that ended up going super viral. And overnight we had 20,000 users. And we were also kind of experimenting with different ways of using AI. So we built this AI interview for ourselves because we had a bunch of questions of how we had a ton of churn. So we wanted to understand why, how they thought about our positioning, different use cases, and it was really useful for us. And that's how we got started.
[08:01]
B
Maybe just walk us through how the industry is changing before and after. Listen Labs. Historically, let's say you're somebody with an app with 20,000 users. You understand how users are using the App what they want next, why they're turning. Historically, how did people go about doing that?
[08:17]
A
Yeah, so what we discovered was that there are these survey tools that are pretty old school, like Qualtrics, but then there's also this very large services industry because it becomes harder and harder, especially if you want to do market research where you want to talk to your prospective customers, not your current customers. It becomes harder and harder to do that as you scale. So that's a multi billion dollar industry. And what they do is come up with questions to ask, which is an academic subject in and of itself. It's actually really hard to know how do you ask questions to a point that get to how someone actually will behave? You can't just ask how much are you willing to pay for this. There's different methodologies that work better than others to finding the audience. How do you source the participants to then analyze hundreds of these calls? And in the traditional industries like cpg, even in Microsoft, they spend tens of millions of dollars on focus groups to bring people in a room and interview them. And we can help speed that up much faster.
[09:26]
B
Okay, so that's the old world of how this used to be done. Maybe describe the new world. And then it seems to me that there were obvious kind of first order benefits. Like there's probably much more scalable, probably much more cost effective, but there's probably also less obvious benefits. Maybe just talk about some of those benefits of what is it like when you actually do AI first market or customer research?
[09:50]
A
Yeah, so most decisions that gets made are not based on the customer input. Right. And the reason for that is it's just a lot of friction to even talk to customers. So when you can lower the barriers of talking to customers, you end up making much smarter decisions. So the speed advantage is actually huge for us. You can get input within five minutes from real people. And it's a really magical experience when you see hundreds of people populate in your interview. And so that's one thing. And because it's asynchronous, it's also much more affordable. So you can pay people much less than if you would have to run like synchronous interviews. So actually that's an interesting thing that people often ask us, do people even like being interviewed by an AI? And the objective answer is yes, because you can pay them less to talk to an AI than to talk to an actual interviewer.
[10:54]
B
Why is that?
[10:56]
A
I think it's mostly because it's asynchronous and people are very busy. But then Also what we found. Yeah, lower pressure. You can kind of go on and off. We've also found that people are more honest talking to an AI. We've had people really open up. It's a very therapeutic experience because it's a non judgmental entity that's really interested in you. And we can also have sensitive conversations like interviewing kids, how they react to different products. And so I think that's another advantage as well, that people can be brutally honest talking to the AI.
[11:33]
B
Okay. So historically, for example, if you want to do research on the kids market, very, very hard to access that market. Is that a regulatory thing? Is it a scheduling thing?
[11:40]
A
Yeah, you need parental consent. Kids are really busy, they go to school, they have extracurricular activities. How do you find time with them? And you need to find the right kind of kids. One of the things we realized is the audience is extremely important. And that's actually where we spend 80% of our engineering resources. Every company is driven by a power law in customer segmentation. So even a product like sweetgreen, which you would think is for everyone, the right audience is typically urban, high household income, mostly female. And by the way, they need to know what seed oils are, which only like 1% of the population does. And then you find that some people go to Sweetgreen every single day and that's 80% of their revenue. So if you can find that segment, the research is so much more actionable.
[12:35]
C
Yeah, there's probably a network effect to it as well, where when you get of a certain scale and people use it, you can access the same kind of person that otherwise might be really difficult to access. Or maybe it's a scale economy, something along the lines of accessing those really, really specific people that are really valuable for the type of product that you're trying to introduce.
[12:57]
A
Yeah, it's really about, you know, we have. Our goal is to get to a billion people in our audience and then to be able to stratify and know what exactly is this person an expert on? And it might be even something like sneakers. You have some people who are influencers and kind of early adopters. And if you're able to find that audience and interview them first, the insights are much more valuable. And we can learn across all of the interviews that we do. So we build profiles of people as we do more interviews in the platform and then we can search and find the right person.
[13:36]
C
So someone might say, in a totally unrelated interview, I'm a total sneaker head. And you can keep that in the database on that person. And then when Nike or what have you is launching their new product line, you can offer that person up.
[13:47]
A
That's right.
[13:48]
C
That's amazing.
[13:49]
A
And that was not possible to do before because it was usually like separate entities. And it would be a very manual process where you would have an email list and you would just spam email.
[14:00]
C
I've been on the receiving end of this. Yeah, they're terrible.
[14:03]
A
And one of the problems with that is that you then need to have a extensive screening process. So you have something called an incidence rate, which can be 10%, which means only 1 in 10 people gets qualified to even take the interview. And that causes significant churn on these databases because it's really annoying to.
[14:26]
C
Super annoying.
[14:27]
A
Green out 10 times to even get paid the first time.
[14:30]
B
Why do these brands even need you for access if. Let's take Sweetgreen. Sweetgreen knows who that 80% is. Can't they just reach them? Don't they have direct relationships with them already?
[14:40]
A
Yeah, so they can. And we do that as well. We connect to their CRM and they can send that out. But then the really interesting part is how do you talk to prospective customers, people who also may not be kind of current power law users? And how do you compare those two? And then also what we found is that the CRM is typically really unorganized and sometimes there's also like regulatory issues. If you're at Google, you can't just send emails to people who use Gmail. And it gets much easier to use an external third party and you run
[15:18]
C
the risk of spam, which can get you totally blocked. I have seen that at some of our companies over the years where, you know, you do outbound and then eventually you're in the Google filter and next thing you know, you're in Microsoft purgatory, I guess, going through you guys, you don't have to deal with that.
[15:33]
A
Yeah, exactly.
[15:34]
C
It's cool.
[15:35]
B
What does this mean for the McKinsey or whoever is of the world that are, you know, the people that are building the hundred slide decks that, you know, reach 3,000 people to reach some set of insights?
[15:45]
C
Did you do that, Sonia?
[15:46]
B
Wasn't that former life, you know? No, Konstantin. But I'm glad that that's what you think of me.
[15:51]
C
Isn't that what banking does?
[15:55]
B
We. We hired cons. I used to hire these people.
[15:59]
C
Got it, Got it.
[16:00]
B
So I was, you know, I was a layer on top of the. Layer on top of the layer.
[16:03]
C
Okay, got it.
[16:03]
B
Okay. I was even more redundant. But what does it mean for all these people, do they still have a role to play in this new future?
[16:11]
A
Yeah, I think AI is changing all roles very quickly. And we work a lot with Bain, for example, so they use us to speed up their traditional processes. And I think they still have a role to play. I think traditional services and being able to then implement these changes is still extremely valuable. But a lot of margins are going to drop and you have to make sure you kind of unbundle a lot of your services to maybe allow for AI agents to help solve some of the problems that you would go to traditional consulting firms before.
[16:52]
C
Maybe I'm an optimist here, but why wouldn't it be more? Why wouldn't I, if I'm running a business, say, oh, great, I want to find five new areas to expand to now that I have AI and these tools. And I will pay you, Bain or what have you, the same dollars you use. Listen and just explore those new areas and tell me where to. Where to build. Is that overly optimistic?
[17:15]
A
No, I think it's one of those areas where the ceiling is very high. You can kind of learn more about your customers and you can build more things. And so I think, I think you're right.
[17:27]
C
I was thinking the chest hair shirt thing
[17:32]
B
that 20 minutes. Still thinking about chest hair.
[17:35]
C
There's so many little things that I'd love to tell the companies that I am a consumer of, like the smallest little. Even the way they laced these shoes. I'd love to give that feedback.
[17:46]
B
This is why you're a venture capitalist.
[17:48]
C
Details, details.
[17:51]
A
We hope to live in a world that finally works the way people want.
[17:54]
C
That would be great.
[17:55]
B
Please. Are you seeing any price and compression already hit the industry like I would imagine if I am Bain's customer, I'm thinking, well, you're able to do this survey a lot more efficiently now with AI than before AI. Who. Who's getting the benefit of that economic surplus?
[18:13]
A
So because you're able to do it faster, I would argue you should be able to charge more for it.
[18:18]
B
And is that what's actually playing out?
[18:20]
A
We have done some studies where we're able to charge hundreds of thousands of dollars to speak to 20 doctors across eight countries. So maybe over the long term, the individual interview will become more affordable. But I think you'll be doing kind of two orders of magnitude more of research. And I think what's really exciting is also simulation, which is something we're building now, where you're able to unlock the 99% of use cases where you would never have time to talk to real people.
[18:54]
C
I think that's so awesome in part because there are so many areas where they don't even listen to the customer. Like medicine. There are a million little problems with the medical system. I hear about it all the time. And these are, you know, their doctors are busy, important people. But it feels like the companies haven't even invested the time in figuring out where all those paper cuts are and the doctors are really busy so they're not going to go schedule an appointment and have some long conversation and meet with some group. But if they could do it at any time like in an app on their phone as part of the normal homepage app and give feedback on their EHR or something in the operating room or something along those lines that seems like a life saving use case for listen over time.
[19:41]
A
Yeah, I think what I'm really excited about as well is taking all those small things and then telling another agent to go and solve that problem. And we're getting pulled in this direction by some of our customers where they will have a churn interview and then they will connect. If you find a bug for example, they'll connect that to another coding agent to go and solve the problem.
[20:05]
C
That's cool.
[20:06]
B
Let's talk about Generative Agent simulation. It seems like the entire industry has gone From Market Research 1.0 call 100 people one by one, collate them manually to Market Research 2.0 ainative where AI designs the question track is able to talk to thousands of people simultaneously synthesize the answers. It seems like we're maybe moving to Market Research 3.0 with Generative Agent simulation. What do you make of that? And I both see the dream of it. I see how synthetic data has changed for example self driving cars. And then I also am inherently skeptical of it. Is a bunch of synthetic data just remixing what's already in the pre training set and are you actually learning anything useful or with alpha there? And so I'd love to hear your take on it and how you guys are taking on the three level and
[20:55]
C
maybe what is it too to start.
[20:57]
A
Yeah. So the way we are building simulation is by interviewing a single person. Say if I interview Konstantin for one hour I can probably start to predict your preferences to some degree.
[21:11]
C
Fascinating insights about chest hair.
[21:15]
A
And it turns out that LLMs are quite good at this as well. So you can essentially try to feed in as much information as possible on a single individual and then in some cases we're able to get 95% accuracy to predict how they will answer certain questions.
[21:30]
B
What sorts of things are you finding that you can predict well versus can't?
[21:35]
A
One of the most useful things is message testing. So that's the idea of like what's the tagline on the billboard? Or I was actually using it this weekend. So I have created a panel of our customer base and I had to come up with the title for a talk at a conference. And it's like a small thing, but it actually does matter because it will increase conversion if people show up. And I came up with 100 different titles for my talk and inputted that into our simulation. And then the top talk was like twice better than the next one.
[22:14]
C
Wow, cool.
[22:14]
A
And I don't know if it's correct, but it certainly felt correct. And it was really helpful to have guidance in making that decision. And I also think even if it's wrong, it's just nice to have some help in making a decision. It's also nice to outsource your decisions.
[22:32]
B
And how does it compare to just asking ChatGPT the same thing?
[22:35]
A
Yeah. So then I inputted the same questions into ChatGPT and actually I had another talk, I did that that was not so successful and I inputted a competitor or another talk that was more successful and I showed both of them to ChatGPT and both of them to our simulation. And in ChatGPT it picked the wrong one and in our simulation it picked the right one. So it's early for us. We're going to release this in a couple of months, but it seems like it's performing better than the general models. And the models are trained on the average person and you want to build for a very specific niche. And that's how we can kind of essentially train the models to follow that
[23:20]
B
niche and just to push on this because I think it's so fascinating, can't you kind of force the models into a specific niche or personality? Like, hey, ChatGPT, you're a 35 year old, really grumpy software engineer that likes using your terminal and then it does then take on the preferences of that niche is sort of my mental model at least. And so I'm actually surprised that ChatGPT wasn't able to arrive at the right answer. And then bootstrapping off real user data was because ultimately it all is kind of a reflection of real user data. Right. And so actually what is the intuition for why kind of SIM only unpretrained data isn't sufficient?
[24:03]
A
Yeah. So we've tried many different inputs and that certainly performs A little bit better than just vanilla chatgpt. But what performs much better is we've tried credit card spend kind of behavioral data purchasing behavior. But what we found was the best data set is interviews because it's more kind of allows you to go off tangents. It understands you can ask behavioral questions. So also it can't just be any interview. Like the way you design the questions is also really important. And the intuition, I think, is that the models don't have clean data on how a specific Persona acts and how they think.
[24:49]
C
It's anecdotal, but it makes perfect sense. Because if you want to understand someone, what better way to understand them than asking them a lot of questions? That's why we're all here. It's kind of the purpose of this type of format and if you have enough people that follow a certain group as opposed to the average, that can tell you a lot about other things that they might not have explicitly said. All of AI is this generalization of some sort of compressed data of some sort. And so if you have this compression in a slightly different part of this hyperspace that you say now complete this orbital of what everybody is thinking in this category of person, you know, Listen can fill that out because it has enough interviews.
[25:30]
A
Yeah.
[25:31]
C
Do you think that you will offer that package as a product? As in if I wanted to understand my customer, and for me, for us, our customers, founders, they're very different though, so extremely different people. If I wanted to understand my customer, could you do active interviews? The normal Listen lab interviews have 1,000 or 10,000 cumulatively and then offer a little special purpose Listen labs bot that then I can use instantaneously for any ad hoc question.
[26:04]
A
Yeah, that's exactly what we have.
[26:06]
C
Okay.
[26:07]
A
So that's what we call augmented responses. The cool part of this as well is that it can also live in your coding agents or your other agents. So I think in the future you will want to have almost a human API which where the agents are able to call the preferences of your users to be able to know like what to build, how to do it or who to invest in or how to help them the best today.
[26:31]
C
Is it all rag? Is it fine tuned? Is it something else? How do you take those conversations and then combine them with the models that you're doing the rest of the Listen labs with?
[26:43]
A
Yeah, we are doing post training typical RAG as well. There's like a bunch of different techniques. Some of them are proprietary, but yeah.
[26:54]
C
All right, we'll do customer interviews on all your engineers report Back.
[26:58]
B
I'm curious what you think of multi agent systems and their role in helping us kind of iteratively use, you know, at inference time iterate to a better answer. Is that part of how you're doing simulation or not?
[27:13]
A
Yeah, like the way we do simulation is essentially you have one person that you model really, really well and then you scale it up with a thousand people so you have a representative sample and it's essentially multi agent.
[27:29]
B
But you're not having those thousand people debate each other. That's what I'm asking.
[27:32]
A
Oh yeah, no, we don't have that yet, but that's a good question.
[27:34]
B
Do you think that would help?
[27:35]
A
Potentially, but there are these other competitors that are doing that approach more. I think the worry is that again, chaos theory tells us when things kind of compound, it becomes really hard to predict how the things are going to interact with each other. And it's something we definitely should explore more. But I'm a little bit skeptical of the approach.
[28:01]
B
Maybe the analogy I'd make is the AI council approach of send the same query out to three different LLMs and then have one LLM act as judge in synthesizing them. I do think on average you get a slightly better response.
[28:13]
A
Yeah, cool.
[28:14]
B
So where else do you see yourself going from here? Then you're going from Market Research 2.0 to Market Research 3.0. Now with kind of generative simulation, do you expect that 3.0 takes over as the majority of queries over time and then what else is ahead?
[28:30]
A
Yeah, I think you'll still need human input, but I think there would be many more use cases that are now opened up where you can get customer input. So for the large decisions, if you're doing a Super bowl ad or things like that, you will still need to run real interviews. But for the smaller things, like what should be the tagline for your billboard? If it's a small billboard, then you can use simulation to answer that. And I still think that there's a lot of alpha on the core product as well to improve. I mean, when we started, the core idea was just making the interview less annoying to go through. Like we had an eval that looked at repetitive questions or looked at is the AI even able to follow the instructions? And with GPT4, sometimes we would ask the same question 100 times. And in the beginning that eval was like 20%. Now we've been able to climb that eval to be 85%. But now we created a new eval that's much more advanced. So it's able to understand what are you doing on your screen when you're screen recording or can you skip questions that are not relevant anymore. And now we're back at like 20%, which I think is one of the values that vertical AI companies can have is that they have this proprietary eval that they can use and essentially climb that eval. And that's your advantage as a vertical AI company.
[30:08]
C
Keep pushing forward. Better data, harder problems, better data, repeat.
[30:12]
B
It seems to me like you're in the middle of a very interesting infinity loop. Right. Because I mean fundamentally a company is figure out what to build, build it, figure out what to build, build it,
[30:23]
A
write code and talk to users.
[30:24]
B
Exactly. And, and the build it is coming up rapidly up in exponential. And the figure out what to build is the thing that you are pushing forward. And then it's not only even outside of product and engineering, the broader loop is actually strategy execution. Strategy execution. And so much of what AI is enabling us to do is it's making execution faster, cheaper, better, all these things. And the thing that you guys fundamentally are positioning yourselves to do as a company is the strategy part from what to build to what to say. Is that a fair synthesis?
[30:59]
A
Yeah. And I think when we have that one person billion dollar company, we'll be part of that loop. So I have a coding agent and listen and then run that in a loop and we'll have these autonomous organizations,
[31:16]
C
even the big companies still back to this idea of you can implement things faster. Let's say you have an agent. I mean if you can be a big company and we're talking in software because software is native to us. But in software, if you could talk to a customer, figure out a bug, create a pr, have a coding agent, close it, chip it, customers happy, that seems like a really important left hand side of the equation. Find the bug from an actual human. But I imagine it's the same thing in a big atoms company. Like if you're consumer packaged goods, if you're clothing, if you're any of those things, I imagine that's even more important because you have to figure it out because once you actually do the thing, it's done.
[31:58]
A
Yeah, exactly. Like Procter and Gamble when they're launching in a new market that can be tens of millions of dollars, if not more. And you have to make sure that that is right when you launch. And that's one of the reasons why they're big customers of Listn.
[32:12]
C
Who has done this historically really well. Like who are the companies that are admired in history have done a great job of listening to their customers either in the, you know, consumer space or in the software space.
[32:30]
A
I think Procter and Gamble is kind of the archetype of the best market research organization where they're essentially marketing companies that are trying to figure out what are niches that people really care about and then build specific brands to solve those problems. I mean, one example is the Tide washing machine pods. Pods that they were able to figure out that it was really uncomfortable to use the washing liquid. And they discovered that people wanted something that was much more easy to use. And through customer interviews, they found this insight, made this new ipod and became really successful. Another example, which is in the acquired podcast when they talk about Mars, they did the first one of the first market research studies in the 1950s where M& Ms. Were originally designed for being used in the army because they were like, sweet treat, they don't melt in pocket. And they discovered through market research that another great segment was young kids. And they then decided to pivot the entire advertising strategy to focus on this because it doesn't melt and ruins your furniture, for example. And things that like, like that.
[33:54]
B
As we progress towards this, you know, listen, labs future, future vision of the world. What are the things that you're confident will work and what are the things that you're still not sure about?
[34:04]
A
I'm confident that in the future you'll still need to have human input, because even if you have a perfect rational being like AGI, humans are still irrational and they will still want kind of be chaotic in their nature where they all of a sudden get obsessed with a new product, you know, a new TikTok trend that shows up and you have to change your entire marketing strategy towards that. And so I think that will remain a really huge part of how we do things. I think. I'm still uncertain of what level simulation will play, and I'm confident that it will work for certain questions. But we'll see how, like, how good the models get to predicting human behavior.
[34:54]
C
I mean, I'd imagine that it's actually even more important the better AI gets to have the delta because the competition. If companies are about serving people, which I think we can all agree on, like, at the end of the day, every company is about serving humans.
[35:10]
B
Constantine, our resident humanist.
[35:13]
C
I'm a humanist, absolutely. But if companies are about serving people, because that's what I. That's why we're all working, is to help someone else in some way. And intelligence gets better and better and better, and you kind of have what the human wants here, and the intelligence is approaching that asymptote, then the delta in that asymptote, which is what is in a human's mind that isn't in the AI's mind, only becomes more important.
[35:34]
A
Yeah. And yeah, one of the things that we've also realized is that there's a lot of talk around what's the mode of these AI vertical AI companies and.
[35:46]
B
Yeah, what's your moat?
[35:48]
A
What is our moat?
[35:49]
C
We've got network effects and scale economies.
[35:51]
A
Yes, those are nice.
[35:52]
B
We're on an episode of Acquired right now.
[35:54]
C
Hey, I'm feeling it. I'm feeling it right now. It's a good book. Recommended. Seven Powers.
[35:59]
A
Yeah, Seven Powers.
[36:00]
C
Love.
[36:01]
A
Love. Seven Powers. Yeah. On the moats, I mean, we have the clear modes, which are the network effects on the panel where you have supply and demand dynamic. We also have the network affects the data mode. As we do more interviews, you get better simulation. And then the product is very sticky because you have all these interviews in your platform and you don't want to lose that. You want to track things over time. But even like the simplest things, I think in terms of product advantages, one of the first things that Brian Shire said, one of our sequel partner, was that founders want to build something that's complex, but customers want something that's stupid simple and it just works. They don't want to configure their own workflow. They don't want to sit and build custom software. And just one example of this is creating the interview guide is really difficult. It's actually an academic subject and it's one of the reasons why you have services firms because they know what methodology to use. If you want to understand pricing or brand perception, these kind of things, you
[37:14]
C
don't want to lead the witness.
[37:15]
A
You don't want to lead the witness. And it's really hard to get that right. In the beginning, we just used the vanilla LLM models and the customers would create interviews, they would get the data back, and then they come back to us really frustrated, saying, like, yeah, what is this? Like, I can't use this data for anything. And we took the blame for that. Now we've trained it to follow the best practices so that you always get good data out of the interviews. And I think that's the advantage you have as a vertical AI company, that you can essentially train this agent to follow best practices and the work that you do.
[37:54]
B
So I want to go back to the concept of tide ponds that you had mentioned earlier. I think it's really interesting. And so much of market research, as I understand it today, is almost more inviting people to pass judgment on ideas that you feed them. But it seems to me that one of the, you know, hallucinations can be a bug. They can also be a feature with generative AI. And, you know, do you think we're going to see user research actually evolve into live product ideation? I could almost imagine, you know, AI inventing solutions as customers are going about their interview process, even helping visualize those solutions. Are your customers doing that already, or do you think we're going to have the moment where I kind of can create a Tide Pods idea in a market interview anytime soon?
[38:41]
A
Yeah, I think that's really exciting. Today they do that manually, you know, use AI to generate images of different concepts and feed that into the interviews. But I think specifically also with simulation, it becomes really powerful. So we now have an MCP as well, so that you can feed that into Claude and then you can tell Claude like, hey, run, listen in the loop, and then come up with a bunch of ideas for how to market something or different concepts and then you can have it run like that.
[39:15]
B
I'm even thinking in the course of an interview as somebody's complaining about, you know, the Tide, I can't sound very portable for the AI to be live brainstorming with you solutions, not just.
[39:26]
C
This is what it could look like an image generator too. Yeah, that'd be cool. Sonia?
[39:30]
A
Yeah, I think it's a good idea. You should be on our product team.
[39:34]
B
Awesome. Well, Alfred, we really love what you're building. Thank you for taking the time to share insights both on the broader market, which I think is just so fascinating, and also what it takes to be building in the application layer right now. We really admire the business that you've built and thank you for your continued partnership.
[39:50]
A
Thank you so much,
[40:14]
B
Sam.