Simulating Humans at Scale: Simile's Joon Sung Park

Summary

Podcast Summary

Training Data (Sequoia Capital)

Episode: Simulating Humans at Scale: Simile’s Joon Sung Park

Date: June 16, 2026
Host: Sequoia Capital partners
Guest: Joon Sung Park (Founder & CEO, Simile)

---

Episode Overview

This episode explores Simile’s pioneering efforts in simulating human behavior and societal dynamics at massive scale using AI and agent-based models. Joon Sung Park shares insights from his research origins at Stanford, the breakthrough experiment “Smallville,” and Simile’s journey from academic research to real-world applications for Fortune 500 companies—while outlining the massive future potential of sophisticated social simulation for technology, business, economics, and even societal policy.

---

Key Discussion Points & Insights

1. The Vision: AI Simulations Inspired by Science Fiction

• Park draws inspiration from sci-fi, highlighting how mature technological societies always have "some version of AGI and simulations that really help guide the society" (00:00, 36:14).
• The conviction: "I do see an opportunity today to really take the first crack at building the simulation. I would not have said that even five years ago." (00:00, 36:14)

---

2. Smallville Experiment: First Major Breakthrough

• Smallville at Stanford: Early 2023 project simulating a town of 25 generative AI agents that live, work, interact, and display emergent human behaviors—"sort of like people would" (01:13).
  • Agents had personas, daily routines, formed relationships, threw parties, and showed realistic social dynamics.
  • Surprising emergent behavior noted (Valentine's Day party, relationship drama).
  • Quote: "You actually see her [Isabella]... gathering materials for the party, actually telling our customers, 'Hey, we're going to have this party, please come.'" (02:34)
  • Other agents had forgetfulness or unrequited love—"Some of the people did get invitation, but they forgot. [...] Klaus... decided to ask his crush out on a date." (03:15)

---

3. From Academia to Startup: Why Simulations Matter

• Park’s research background in social computing, human-computer interaction, and foundation models highlighted the lack of tools to anticipate emergent phenomena in large-scale social systems (03:45).
• Historical problem: Real-world field testing of social platforms is costly and risky.
• Early work: "Social Simulacra" at Stanford (2022), used GPT-3 to create simulated communities (e.g., travel discussion subreddit). Saw early glimpses of authentic emergent group behavior (05:30).
• The academic to startup shift:
  • Research is "breadth-first search," exploration and hypothesis.
  • A company is "depth-first search"—focus, resources, execution.
  • Realization came after clear inbound interest from social scientists and Fortune 500 companies (10:10).

---

4. AI Models for Human Simulation: Where Are We?

• GPT-3 enabled initial simulations, but had limitations; more recent models provide "good foundational level" fidelity for simulating humans, but not all the way:
  • LLMs are optimized for rational, objective outputs, but real humans "are irrational... have a lot of subjective values, preferences and taste." (09:25)
  • To go further, "We do need the next frontier that is more geared towards actually modeling people’s diversity." (09:29)
• Simile evaluates/validates accuracy: "Simulations of thousand people of the US population... we can actually predict people’s behaviors 85% as accurately as people replicate their own." (11:50)

---

5. Simile’s Product: Applied AI Simulating Markets and Societies

• Customer Example: CVS (12:58)
  • Simile partners with corporate insights departments (e.g., CVS’s SVP of Human Insights).
  • Simile helps clients understand and test at scale—beyond what’s possible with traditional field experiments or surveys.
  • Simile creates agent simulations grounded in real data, collected in partnership with vendors like Gallup.
  • End product is SaaS: Customers can pose questions/scenarios to these simulated populations.

---

6. The Data: Why Not Just Use LLMs?

• LLMs are “good” but often reflect what people say, not what they do—addressing the real-world “say-do gap.”
  • "One of the big questions here is the question around say do gap. There are things that people say and then there are things that people actually do." (16:37)
  • Simile closes this by grounding simulations with real behavioral data (RCTs, interviews, stories-of-life) combined with more attitudinal survey data. (17:53)
  • Focus on extracting "long tail" information: life stories, key decisions, nuanced behaviors. (19:14)
  • Use both interviews and surveys to maximize understanding while minimizing the time required.

---

7. Model Building Strategy

• Simile builds its own models when diversity of values/tastes needs high fidelity and relies on existing frontier LLMs for other tasks (20:37)
  • "We want a model that's as human as possible." (20:37)
  • Simile’s models are more akin to a 'GPU of intelligence'—optimized for modeling variation in human values/preferences, not just superhuman reasoning.

---

8. Use Cases Beyond Every-Day Market Research

• Start with concept testing, messaging, product launches—clients use simulation to quickly iterate across thousands of concepts and subpopulations (22:03).
• Unique value: Multidimensional and multi-agent simulations (earnings call reactions, downstream impacts, complex interactions).
• Simulation allows for representative, scalable, and accessible research—addressing the major bottleneck of accessing relevant real-world populations. (24:05)

---

9. Simulation Accuracy, Limitations, and Convergence

• North Star metric: Population-level accuracy (measured by total variation distance between simulated and real distributions). Threshold for decision-use: t-vid ≤ 0.5. (26:47)
• Multi-agent/temporal simulations may accumulate errors (“daisy chaining”).
• Two simulation categories:
  • Convergent: Large-scale trends and network effects (e.g., hubs always form in social networks) are robust to some error.
  • Divergent: Path-dependent, binary, or rare outcomes (e.g., who wins an election, the origin of financial crises) can be more sensitive and require probabilistic treatment and repeated runs. (28:22)
  • "A huge part... is then to show when it diverges, to show the diversity of possible outcomes so that people can actually look, understand the mechanism..." (29:44)
• Simile is pioneering standards for simulation reliability—analogous to how p-values shaped inferential statistics. (31:07)

---

10. Societal and Scientific Potential: Economics, Policy & More

• Park envisions “Nobel Prize” scale impact:
  • From economics to climate change, central banking, democracy, and fundamental human questions. (33:02)
  • Reference to agent-based models from the past (e.g., Thomas Schelling) but now orders-of-magnitude more realistic due to agent fidelity.
  • Examples: Simulating bank runs, democracy collapse, policy impact, climate change coordination, monetary origins.
  • "What about simulation that takes actually $100 million to run once... but when we run it, it solves one of the fundamental questions of our society?" (34:44)
• Politics could be transformed—imagine simulating downstream impacts of policy, not just claims. (35:43)
• The "Hubble Telescope Moment": Simulation as a new measurement tool for the social sciences. "Simulation can be that for human society." (36:14)
  • Potential to "unlock our understanding of humanity and social sciences" and "make our society be a better place." (36:28)
• Host agrees: There is a chance "the field of economics as we know it may actually become solved by simulation...[or] everything that deals with human behavior..." (37:40)

---

Memorable Quotes & Moments

• On the startup vs academic research:
  • "Research is an amazing vehicle... for breadth first search. Company is a machine for depth first search." (10:10 - Joon Sung Park)
• On the integrity of simulation:
  • "There are things that people say and then there are things that people actually do. And the gap there is real." (16:37 - Park)
• On scaling idea/testing:
  • "What does it look like for us to test instantly thousand different ideas across thousand different subpopulations?" (22:03 - Park)
• On the field’s future:
  • "There's a lot of focus on natural sciences. But how can simulation really unlock our understanding of humanity and social sciences?" (36:28 - Park)
• On the transformative vision:
  • "Simulation can be that for human society." (36:28 - Park)
• On potential impact:
  • “If we do this right, there’s a Nobel Prize to be won there.” (33:02 - Park)

---

Key Timestamps

| Timestamp | Topic/Quote | |-----------|--------------------------------------------------------------------------------------------| | 00:00 | Inspiration from science fiction; AGI + simulation as societal pillars | | 01:13 | Smallville project introduction | | 02:34 | Emergent behaviors in Smallville (Valentine’s Day party) | | 03:45 | Research background; limitations of real-world social experiments | | 05:30 | "Social Simulacra" and early simulated communities at Stanford | | 09:25 | Current LLM models vs. real human diversity/irrationality | | 11:50 | Simile's US population simulation: "85% as accurately as people replicate their own" | | 12:58 | Customer journey example: CVS collaborating with Simile | | 16:37 | Why real-world behavioral data is needed, not just LLMs | | 20:37 | In-house modeling philosophy: “As human as possible” | | 22:03 | Simulated product/concept testing at scale | | 24:05 | Why do this vs. classic A/B or Facebook ad testing? | | 26:47 | Simulation accuracy, "North Star" metrics, and limitations | | 28:22 | Convergent vs. divergent simulations and error propagation | | 31:07 | The future of simulation reliability and scientific standards | | 33:02 | Nobel-level impact, economics, macro policy, agent-based models for society | | 36:14 | Science fiction vision: simulation as the Hubble of the social sciences | | 37:40 | “Economics... and kind of everything that deals with human behavior... may ... be solved” |

---

Episode Tone

• Inquisitive and ambitious: The conversation reflects deep curiosity and optimism about the power of AI and simulation to answer hard questions about markets, behavior, and society.
• Visionary but practical: Park is both realistic about model limitations and candid about the need for further research and standards, but ultimately driven by a belief in transformative societal impact.

---

Summary Takeaways

• Simile is at the forefront of simulating societies at scale, blending academic rigor, real behavioral data, and cutting-edge AI.
• Current models are good, but richer, more diverse simulations demand more tailored approaches.
• Applications span from faster, broader market research to potentially transforming economics, policy, and social sciences.
• Simulation could become the “Hubble Telescope” for understanding and guiding society—a new pillar for the age of AI.

Transcript

[A]

I am somebody who is quite inspired by science fiction. And when you read science fiction that covers societies that have progressed far enough in its technological maturity, you always see two pillars. You have some version of AGI and you have some version of simulations that really help guide the society. I do see an opportunity today to really take the first crack at building the simulation. I would not have said that even five years ago, but that is a conviction that we have built up over the years as we are going deep into this research.

[B]

Today. We're delighted to have Jun, founder and CEO of Simile. Simile is building an applied AI lab simulating human behavior and societies. And I'm very excited to have you here to discuss what you're building.

[A]

Same here. Thank you for having me.

[B]

Okay, take me back to April 2023, Stanford, California. Specifically Smallville, Stanford, California. What was that?

[A]

So Smallville was a project that we were running at Stanford, where the idea was that we made this observation that large language models can now encode a lot of human behavior that is embedded in its training data from the web and social media and so forth. That if you sort of probe at the right angle, you can actually get a lot of micro behaviors out of these models. So given a very specific demonstration or description of a situation, what would person X do? And it would actually generate really interesting behaviors. We found that to be so interesting, and we found that to be the ingredient that we had been waiting for for creating really complex agentic behaviors. So smallvid actually was an experiment where we decided that if we push this as far as possible, what would a society that is created by these agents look like? So we basically created generative agents that is paired with generative AI model with memory, planning and reflection to basically create this lived experience of agents living in the small town. So Smallville was basically a game town of 25 agents living in it. Individual agents had a description of Persona, but they would actually wake up in the morning, do their routines, go to work, actually have relationships sort of like people would, and they would actually have emergent phenomena like having parties and so forth. So that was the experiment that we ran.

[B]

What was the most surprising things to come out of the experiment?

[A]

So one of the surprising things was, so the experiment, the simulation itself actually sets place the day before a Valentine's Day. So you actually see these agents, One of the agents actually thinking, well, I run a cafe. So she's a cafe owner, her name's Isabella. She goes and thinks, it would be great if I can do a Valentine's Day party. Where we invite a lot of friends, customers. So you actually see her on the day before Valentine's Day, going around actually gathering materials for the party, actually telling our customers, hey, we're going to have this party, please come. And on the day of Valentine's, you actually see this immersion party that actually get formed with all these agents coming to the party, to the Biscuit Club.

[B]

Did anyone not get invited?

[A]

Well, some of the people did get invitation, but they forgot. That's one thing that did happen. Some of the agents did not explicitly get invited, but we had one agent who got the invite, Klaus, who decided to ask his crush out on a date. So he would actually bring in the date. They would actually have a party at this cafe. So quite surreal.

[B]

So how did you end up building Smallville in the first place? Were you studying kind of human psychology and social behavior? Or was this coming from the kind of customer back, or was it coming from the technology out?

[A]

So my particular team has been excited about simulations and we saw the vision of simulation failure early on. So my career as a researcher at Stanford really started back in 2020. That was the year when GPT3 was about to come out. It wasn't quite there yet, but it was just about to come out. We started to get its first demos. And my first year we wrote this paper called Opportunities and Risks of Foundation Model alongside many of the Stanford researchers. And it was led by one of my co founders, Percy Liang, who is now the head of the center for Foundation Model at Stanford. And when we were writing that, the part that I was really focused on was, well, here's a new class of models that we have not seen in the past, that these models that can be very generalizable in ways we didn't quite have in the past. And I got into thinking, well, if we can imagine the kind of interaction we can create with these models, what would that be? And many of my colleagues back then were surprised that these agents or these models can do classification or simple generation. And that was really incredible to see because these models didn't really know or wasn't really taught to do that. But the part that was surprising to me wasn't that these models can do that because from interaction perspective, we've known how to do this for a long time. The interesting part was, well, these models can actually encode human behavior. What does that mean? If we were to push this as far as possible? So part of the tradition I come from, research included what we call social computing, and social computing within human computer interaction really has to do with this idea of how can we build a better technological platform that would enable social interactions and collaboration? One of the most difficult challenges of building a social platform is not necessarily testing the UI UX of the system, but it's more about when you have tens of people, millions of people, and down the line, billions of people, how do all these people come together to create the immersion phenomenon that's both good and bad? And how can we design for a scale? And so far, we didn't really have a tool that would enable us to test for that. The only way we test it today is you basically field test it. You release your prototype, see what happens. And sometimes it actually comes at a real cost. Obviously it's high cost in terms of human hours and the time it takes. But at the same time, if you have a bad design, imagine you have a feed on social media that is more likely to propagate certain emotion that is negative. Then obviously that is something that we want to avoid. But this now gets tested in the field. So we wanted to see whether we can actually create a simulation that would actually let you test for this. So 2022, this was actually a year before generative agents. We worked on a paper called Social Simulatra, which actually really was the precursor to the agent paper that we ended up writing. The core thesis was imagine you're building a subreddit. You're a designer on a subreddit. You want to see what people might do in the subreddit, which is surprisingly hard task even for practice designers. And we basically decided, hey, we have this model seems unique. Let's use this model to create simulations of the entire subreddit. So you define the goal, you define the moderation strategies, and you populated with thousands of. Back then, we didn't call them agents, but we called them Personas, but populated with thousands of Personas. This is basically 22 version of mo book, which is quite interesting that it actually came back. And when we saw that, we actually got a lot of really important insights out of this. What are the good behaviors? We actually stimulated a community where the entire idea was for people to discuss with each other the places to sightsee in Pittsburgh. And all of a sudden you start to see this. Personas actually collaborate to actually discuss. Hey, XYZ places are amazing. Do you want to actually go to a trip together and actually plan those trips live in this simulated subreddit? So that's how we got excited. So we saw the vision and the excitement and the potential applications failure early on. But then the work that we had to do was then demonstrating how can we go beyond simple Personas to create complex agents that actually can think over time. Because we want to simulate the longitudinal aspect of our society and then actually validating that these simulations are actually accurate in practice.

[B]

Was there a point of model evolution at which you felt like, okay, we're there, the models are good enough for us to actually have a faithful representation of human society.

[A]

So GPT3, when it came out and Social Simulacra was built with GPT3 and it was very janky, it didn't do any instruction tuning, it did not follow your instructions. So just to have it to listen to you and do what you wanted to do, you had to do some weird tricks with prompting and so forth. But you could actually see the promise. The model actually have encoded a lot of human behavior and you could actually see the trajectory. And when we had the generative agents paper wasn't quite chatgpt, but we now had instruction tuning so we could actually build much more complex agents that can reason about its memory. That wasn't really possible when we did Social Simulacra. And since then, of course, the models have improved. So where we are today is the models at its foundational level have reached a point where we can actually imagine building these kind of applications. Now the part that actually I do think, however, that's quite interesting here today. If you look at many of the large language model companies, whether it's OpenAI Anthropic and many of the NeoLabs that are getting formed, the models they are creating are models that I would consider to be their North Star, to be something that is similar to let's build a super intelligent machine. These machines are meant to be rational and these machines are supposed to be really amazing at technique problems that have an objective answer.

[B]

So maybe that's not even the best simulation of true human society.

[A]

Then turns out people are irrational. We have a lot of subjective values, preferences and taste. So you actually start to see divergence in model size going up and the performance in its ability to predict and simulate human behavior. So we have sort of plateaued with current modeling paradigm our ability to really simulate humans. So it is sort of at the starting good foundational level. But to make it really amazing, we do need the next frontier that is more geared towards actually modeling people's diversity.

[B]

Very interesting. At what point did you realize that what you did with Smallville could become a company?

[A]

Right. So again, the promise of application was something that I was very much inspired by early on by Simulation with social simulacra and so forth. But the part that I realized over time is research and a company have very different function. Research is an amazing vehicle. If you want to basically do breadth first search. You are in a lab surrounded by really smart set of people and each of the researchers own a small piece of thesis and they go explore some of those thesis blossom into amazing research product. But we are not necessarily known for finishing our job. We're not usually the one to bring that research impact to the real world. Company is a machine for depth first search. You have a conviction on an area, you find a hill that you want to climb. This is the vehicle that let you put together resources and an amazing group of people to go after a singular vision without hesitation. And we got that conviction, I would say about half year after Generative Agents, after the original Generative Agents paper. We got so much inbound interest initially from actually social scientists who wanted to run their experiments and all the RCTs on our platform. Then very soon after, many of the Fortune 500 companies who saw this demo and their board members and CEOs who sometimes visit Stanford saw that and they start asking, well, we go run all these surveys and experiments and there's so many research questions about the market that we cannot answer today. Can we run that in simulation? That started to really intrigue me because that showed a clear line towards a real world impact for research, which is not always the case that we have that kind of opportunity. So that is when we decided we actually want to validate the simulations are accurate. So we went out and actually created simulations of thousand people of the US population. We demonstrated that using our architecture and the models, we can actually predict people's behaviors 85% as accurately as people replicate their own. When we saw that, we thought, okay, this is something that we feel comfortable providing to our users as a platform for simulating their really important decisions. So that's when the co founders, myself, Percy, as well as Michael Bernstein, was a researcher and my advisor at Stanford. Both of them were actually my advisors. So the three of us had been working together for five years and now at this point assuming six years, but that's when we got together to have the initial conversation of can this be a company?

[B]

Got it. Amazing. Maybe walk me through a customer engagement end to end today, like who's a canonical customer and which department? And they come to you, what are they asking you and what product do you or service do you deliver to them?

[A]

Right. So maybe an example that I can give to make this concrete so CVS has been partnering with Simile for the past, I would say nearly half a year and they've been an amazing partner. The way we initially got in touch with. So our main buyer at CVS is the lead is a senior VP who leads human insights and original story. There was he basically read my paper that validated the agent simulations and thought we have to bring this to CBS because today we are bottlenecked by the number of questions we can field test and we're also bottlenecked by truly the physics of human society. It's one thing to ask surveys and experiments, totally different thing. If down the line you actually want to simulate the entire market and actually map out all the second order impact of the decisions you suggest to your leadership. So, so he's been looking around for that solution and his cousin happened to know me and basically told our buyer SRI that the authors of the paper are actually looking to start something. So that's how we got connected. And in this particular engagement, usually the way this goes is our customers are very much used to working with polling companies or panel companies today. And there they go and basically ask these companies XYZ are the populations that we are interested in better understanding? Can we go run a research study of these topics? That initial stage looks very similar for Simile. So our buyers come and they tell us we want to better understand XYZ population. Then Simile goes out and we have, through our partnership with vendors, we have a strategic partnership now with Gallup for instance, who is a polling and panel company where we go out, work with our vendors to actually reach out to real humans. So these simulations are grounded in real data. But reach out to those people, collect data that we believe are efficient and generalizable about that person. So imagine you have 15 minutes. What are the magical questions you can answer or you can ask these people during that time we collect that data, use that data to create agents or simulations of these people that can basically be used to answer a large number of questions that goes way beyond the original domain. We load it onto our platform and it's basically a SaaS product. Our customers come and they can basically ask any questions about the group of people of their interest.

[B]

So interesting. It reminds me of in autonomous vehicles, you go and collect a bunch of data from the road and then you're able to augment it with simulation. Is this similar concept or are there big differences to what you're doing here?

[A]

It is similar concept in the sense that of course you with the self driving vehicles, you want to Create model that is based on real world physics. But you want to create a model that is generalizable beyond your training data. It needs to be generalizable in two different locations with different weather conditions. Very similar concept. Where what we want to create is we want to reach out to real people and for these people, want to understand something fundamental about these people in a way that we can encode into the model.

[B]

I would have thought that the large language models would be such a good representation of the whole world that you could almost narrow it down. You could tell Claude, you are a 34 year old woman living in a bicoastal metropolitan area and it would be able to have a faithful representation. So I'm actually surprised that you go out to Gallup. Maybe. Can you just explain why you have to go out and collect any real world data at all?

[A]

Yeah. One of the big questions here is the question around say do gap. There are things that people say and then there are people. There are things that people actually do. And the gap there is real. And a lot of the large language models are trained on attitudinal data. Fundamentally, it is the things that people have said online that does cover a large quantity of its training data. So one of the things that Simile's simulation platform does is actually closing that gap. So a lot of the data that we end up collecting by nature are behavioral. It also includes data that actually goes into literally questions like just tell me the story of your life. Turns out if we understand the person's story of your life, the kind of data you get from it is what we consider to be the long tail information about this person. It's not about what you've done in this particular moment. It's not about very broad questions like what's your view on politics? It's about where you grew up, what were some of the difficult decisions you have to make in life. And what's interesting about this data is it's an amazing way to build a translational layer between attitudes and behavior. So we combine these kind of data sets. But fundamentally that's the gap that we want to close.

[B]

What sort of behavioral data do you have?

[A]

So Simile does run a lot of experiments. So kind of models that we have trained. For instance, we have a huge repo of RCTs, so randomized controlled trials that were run in social scientific context that were run around pricing studies. So one of the models that we are training is basically the foundation model of human behavior in quite a literal sense. We have all the behavioral signal from RCTs can we actually encode that into the model so that the end outcome is a model that can basically predict the results of any RCTs at the same time, one of the conversations that we keep on having with our customers that we're very excited by is our customers then come in, see that potential and their mind goes to, wow, we have 90 million customers. Let's say here at CVS. How can we leverage this kind of data to create better simulations? So there's also conversation around, how can we in a responsible and ethical way leverage existing data that is also in house for our customers, then use that to create augmented version of Simulink's model So that of course is going to be more fine tuned, specific to the population of these customers. But that's the kind of data that we will be leveraging.

[B]

I see. And are you doing these interviews typically by voice? Is it a survey that you fill out? What's the modality?

[A]

So it's a huge breath. The quick answer here is it's both interviews are fantastic if you want to get the long tail information about people. So we actually do. In the original study that I conducted back in 2024, we literally ask question, tell me the story of your life. Now the way we do it is we are training our own model. So it's a reinforcement learning loop. But basically imagine the objective function here is how can you spend the minimum amount of time to get the maximum amount of visibility about this person. So that is one of the things that we do. So basically training an interviewer that is not really asking for factual information or an experience about a particular platform, but just what are the life story that people have that can be used to train our own model for these agents and then for the more factual or sort of more discrete choices, choice questions, surveys and so forth, these are also very efficient. These are time and data efficient because people can fill out many of the questions in short period of time. So for those, we actually do leverage them. For instance, if you want to just have a broad understanding of people's viewpoints on certain topics, certain policies and things like that.

[B]

You describe yourself as an applied AI lab. How do you think about where you want to build your own models versus where you want to rely on other existing models?

[A]

So in terms of building our own model, the core thesis here is there is an amazing model to be built that really encodes the diversity of people's values, preferences and taste in ways that simply a rational model cannot do. So one way I actually pose this. We're sort of building. So imagine the current, today's model are akin to the CPU of intelligence New data. It's a single model trained on amazing irrational data that is amazing at solving very complex objective questions. Simile's model is much more akin to developing something that is closer to the GPU of the intelligence unit. Where the idea here is we don't actually need a model that is superhuman at Simile. In fact, we want model that's as human as possible. But we want to make sure that these models at the sort of individual subunits can represent the real viewpoints of different cell populations. So where we see that gap, that's when we go develop our own model. But at the same time, we do leverage volunteer models, for instance, as a way to coordinate the research. Frontier models are amazing at coming up with a research plan. So that's where those models actually do get leveraged.

[B]

Very interesting. Are people typically coming to you with questions around new product launches, how they should be marketing their companies, pricing, all of the above.

[A]

So it is all of the above. Our customer journey usually does, however, start with very concrete use cases and problems they are trying to solve. Concept testing is a big one. It's also a very straightforward one. So they have a new concept, new product idea, new market message they want to test and they want to hear from their users what they would think about xyz. This is one way for them to quickly test those ideas. And then the promise they quickly see is, well, right now we're very much in the practice of testing five to 10 different ideas at most. But what does it look like for us to test instantly thousand different ideas across thousand different subpopulations? That's the initial vision they see. Then we really get into the nitty gritty details of, well, where does simulation go from here? They then pretty soon start asking, well, can this be used to do product testing? But not just simply submitting an image, but imagine basically asking these agents, go experience this product for 10 minutes and tell us about what you experienced, what you saw. So you're basically adding temporal dimension. Then you go into things like multi agent simulation. Some of our customers very routinely actually ask us to simulate their earnings call. This is actually a use case that both surprised me at first. But this is also surprisingly a common ask because of course the CEOs and board members always need to think about, hey, how are we going to design our earnings call? How would the audience react? So that is something that we also do, and this is very much multi Agent simulation.

[B]

You know, it seems like there's so many use cases that could potentially be tested once you have like a simulated almost customer population. Right. I'm curious the value of research and testing in sim versus just like let's say you have a new product concept that you want to test. Why not just go run a thousand Facebook ads and like you actually get the click through rates on this stuff. Isn't that real world data almost more useful than the simulated data on how people might behave that you then correct for with your own models?

[A]

So it's a great question and I think to some extent here the answer has to do with initially scale and then down the line, truly the new capability that comes because you can simulate the interactions. The scale question here is actually quite straightforward where yes, you can absolutely run Facebook ads and Facebook testing, but the kind of experiments that you can run in simulation is actual behavior simulation at scale. So you can basically pull in any number of users. Doesn't even have to be bounded by the number of population that's available on Facebook. And it's also much more representative because only certain groups of people will actually respond to the online experiments. But similarly, the model that we are creating, one of the key promises is that it is representative. We do the hard work of actually getting the representative set of people and then collecting the data that would actually represent them properly. So the scale representativeness is something that many of our users do not have easy access to. This is actually one of the common ask also that we do get or common sort of pain points that we have heard where the question that many of these people have isn't about like what questions do we ask these people but it's about in the first place, how can we get to the population that we're excited to talk to? That's a huge bottleneck. Then down the line you can actually really start to imagine. And this is something that our customers and some of the most forward looking customers are now going into, which is what are all the downstream implications of the decisions that you make? It's not just about whether imagine you have this particular product, do you like it or do you not like it? Would you pay for this? Not pay for this? It's not necessarily just that initial questions that we want to answer and finish, but we want to understand. Imagine you're a car company, you launched an electric vehicle in this market, maybe the electric vehicle does really, really well. So we can help you do concept testing around marketing and the product around the electric vehicle. But what does that do to the perception of let's say non electric vehicle, does that change the market perception? Then what does it mean for the rest of the product line? And how do you balance those kind of second order impact of your decision in a way that is more evidence based today? There's no way to test for this. You can run this in simulation. So really going beyond simply asking one question at a time, but then to think about what are the long term implications of your decisions is something that our customers are quite excited by.

[B]

I'd love to understand how you think about how predictive your model is in actually simulating real human behavior. I imagine you have a lot of evals on this. I guess what is your North Star metric? How do you guys do on that? And what do you think is the theoretical limit?

[A]

That's a great question. So theoretical limit and let me just start from there certainly does exist in the sense that humans are genuinely. There's a lot of randomness that if you ask me the same question, I'll actually answer the question slightly differently. So there is certainly that degree of randomness in human behavior. However, there's a lot of gains in performance that we can have even today in the way we are predicting people. So the measurement that we do is so at the level of population we measure the distribution of responses if it is more quantitative. So we actually measure total various distance which basically shows how close are the distributions of the ground truth versus the simulated information. And that is a metric that we run across all the use cases that our customers have. And we have certain threshold that we believe is good enough for decision making. So t video of let's say less than 0.5 we believe is actually quite strong evidence for making decision. So that is the North Star state that we want to hit for this class of use cases that are more quantitative. That's more question and answers. This also does cover RCTs which is many of the core use cases our customers have. Now there's actually a really interesting question to ask around. Well, what about multi agent simulation? What about all the downstream implications that we're going to be simulating? What does the evaluation of those look like?

[B]

Yeah. And then do daisy chain errors as you kind of. If this one is 85% accurate and then this agent is telling another agent something. And do you accumulate errors as you go as you go towards multi agents?

[A]

Exactly. And one of the core thesis here is we basically see two categories of simulations. One simulation is what I would consider to be simulations that converge. The other categories of simulations Are the simulations that diverge. And sometimes they actually coexist. And it's really about what research questions do you have, Questions that converge. Doesn't actually matter if you have a little bit of error. Now, the error cannot be obviously so dramatic that it sort of is completely detached from reality, but you actually are okay, even if the errors do compound over time, because the pull towards the convergence is strong enough that you'll actually understand where everything would fall. A good example here actually is if you simulate a network of people, then that network will always have a hub that gets formed. This is what sort of network scientists would call the scale free network, for instance. This is actually what powered Google too. One of the core observation of PageRank was doesn't matter how these networks actually get formulated. You actually see some web pages that get exponentially more links that are attached to it. This is a very fundamental behavior in humans that we also see in simulated networks. And that convergence always happens as long as you are replicating human behavior with certain threshold accuracy. Now, there are then questions that generally do diverge. It's like your classical questions like was World War I inevitable or was it not? And there it is sometimes difficult to run the same simulation over time and get the same exact outcome. Imagine you're running a. This is not something that necessarily simulate right now is going into. But imagine you're running a simulation of an election. Will the same person win the election every time? There are a lot of downstream implications of every single decision that does happen. So it does diverge there. The core evaluation is around confidence. So imagine you run the simulation 100 times. How many of those times do the results come out to be X? And how can we actually use that to basically create a bootstraps resampling to calculate the confidence around the simulations? Those are some of the questions that we do ask. And a huge part of this also of the power of simulation, is then to show when it diverges, to show the diversity of possible outcomes so that people can actually look, understand the cause or mechanism of how we got to those outcomes and prepare for those features. So those are some of the implications of divergence in simulations.

[B]

Are there any mathematical descriptions of why something would converge or diverge? I imagine if you have an average function, maybe you converge. And then if it's like you're splitting outcomes to a binary, then you might maybe diverge.

[A]

Yeah. So the intuition, I think is close. And technically this is also a research topic. So similarly is a Zeltinger company where we do go deep into this research Topic in the sense that I see simulation as a field as akin to developing your day one of inferential statistics. Inferential statistics. Scientists actually had to do a lot of discussion and research over time to decide that p less than 0.05 is actually evidence that is strong enough for science. Similarly is working on setting the same kind of threshold and standards for the rest of the field. So those are the intuition. I think that's exactly the right intuition in terms of actually how to make a robust mathematical equation around what's going to happen when it is a real research frontier for simulations.

[B]

Thank you for being nice about my vibe mathing. I'm curious. It seems like so there's a lot of Fortune 500 is coming to you. I'm wondering whether there are non existing corporate use cases that might. There are great mysteries of our society that might become solved. And for example, I'm wondering about economics central bank decisions oftentimes. I personally believe in macro nobody knows nothing and oftentimes a lot of the issues come about from human psychology. So to me macroeconomics is a function of simulating human behavior at scale. I'm thinking even in the venture capital use case, we often debate internally does value accrue to this company or not? You could run the simulation of all the different layers of the AI stack and and almost figure out where durability and value accrues. If you had a kind of perfect simulator of human behavior, there's so much more you could do than serving the Fortune 500. Do you agree with that? And then if so, are you serving governments the like?

[A]

Yeah. So it's interesting when we were still researching in this area, the way I actually got back then, my advisors Michael and Percy, excited about this was I basically told them look, we do this right, there's a Nobel Prize to be won there. And I truly believe that. And it's also not surprising in that your classical economics simulations, things like agent based models that really pioneered our understanding of back in the day, the kind of topics they studied was how does segregation happen? What are the causal mechanisms for segregation? So scholars like Thomas Schelling would actually build agent based models that are extremely simple and rudimentary, but that showed something deep about human macro behaviors. And he of course went on to win a Nobel Prize. I see the same opportunity here, but in an augmented way. Where back in the day the agent based models were very much deterministic in some sense. Where you basically in this simulation of let's say model of segregation from 30 years ago, individual Agent was simply red dot or blue dot. And every game iteration they would look around its corner, see how many of its neighbors are of the same color. And if that threshold goes below certain threshold, then they would decide to move to a new location. That was it. But now we can actually create real agents that replicate the full richness of individuals and run the same kind of simulations. So the kind of questions that we can ask that goes beyond simply the commercial use cases, for instance, in the context of macroeconomics. Actually the questions that I actually did get asked from economists were things like when does bank run happen? Or questions like climate change. One of the sort of core blocker of climate, like solving that issue is the collective action problem of many nations. Can we actually simulate that? Or what are the signals of a democracy that is about to collapse? Can we understand the origin story of the monetary system? These are the kind of simulations that I do believe ought to be the north star state of this field. And it is sort of interesting to imagine like what that would actually look like in practice. Right. Because these would involve very large scale simulations with many agents interacting with each other. I do see a future where today this is certainly not the case today. A simulation is quick and fast to run. But what about simulation that takes actually $100 million to run once and could take many months to run, but when we run it, it solves one of the fundamental questions of our society that I do think is genuinely a very exciting possibility for this field.

[B]

I agree. I'm even thinking like politics, for example, could be forever changed. Today everyone has an agenda of how they say some policy change will affect, will impact things. Why don't we just run the simulation

[A]

and understand all the downstream implications and not just what's going to happen this year, but what does it mean in the next five to ten years?

[B]

Exactly. Fascinating. I was going to close by asking you what makes you excited about the future? Is it what we just talked about or is it something else?

[A]

I am somebody who is quite inspired by science fiction. And when you read science fiction that covers societies that have progressed far enough in its technological maturity, you always see two pillars. You have some version of AGI and you have some version of simulations that really help guide the society. I do see an opportunity today to really take the first crack at building the simulation. I would not have said that even five years ago, but that is a conviction that we have built up over the years as we're going deep into this research. And what's exciting is there's a clear use case today that can serve our users. But then there's a lot of innovation that is yet to come that I do think will build up to actually building simulator that's akin to discern of human society. And one of the things that one of my co founders, Percy, sometimes say is you look at the greatest scientific innovation, they often start from an amazing measurement. Hubble telescope really change the trajectory of how we understand the universe. Simulation can be that for human society. So the thing that does excite me, there's a lot of focus on natural sciences. But how can social, how can simulation really unlock our understanding of humanity and social sciences? And how can we actually use it to make our society be a better place? That's exciting.

[B]

I remember reading somebody was excited about, you know, there's a small but, you know, breathtaking chance that the field of economics as we know it may actually become solved by simulation. And I'd extend that not just to be economics, but kind of everything that deals with human behavior and social sciences, which ultimately is everything around us. Truly wonderful. Thank you so much for joining today and sharing the story of both Smallville and what you're now up to at semily. I really enjoyed the conversation.

[A]

Same here. Thank you for having me.

[B]

Sa.