Sean Carroll’s Mindscape Episode 304 | James Evans on Innovation, Consolidation, and the Science of Science

Date: February 10, 2025
Host: Sean Carroll
Guest: James Evans, Professor of Sociology and Director of the Knowledge Lab at the University of Chicago

Overview

This wide-ranging episode delves into the “science of science”—examining how research actually happens, the dynamics that shape creativity and consolidation, and how collective intelligence (among humans and increasingly, AIs) drives progress and innovation. James Evans shares empirical insights about how choices, structures, and demographics affect scientific discovery, how new tools like large language models are transforming the landscape, and what it all suggests about the future of knowledge-making. Together, Evans and Carroll weave history, philosophy, sociology, and cutting-edge science into a conversation about fostering innovation while wrestling with ossification.

Key Discussion Points and Insights

1. The Challenges and Choices of Doing Science

(03:04 – 06:32)

• Science is often difficult not just because of technical hurdles, but due to deeper challenges: What problems to pursue, how to balance innovativeness with credibility, and which collaborators to choose.
• Modern science is shaped by collaborative, interconnected networks. “The world is very different than in Dirac’s time … even within theorists, it’s so easy to communicate, collaborate … with people who don’t know what you know.” (Sean Carroll, 04:57)

Memorable Quote:

“Science is not just one person. There’s lots of scientists doing things. So how do these scientists come together in groups? … What kind of groups make the best decisions?”
— Sean Carroll (05:44)

2. James Evans’ Interdisciplinary Approach and Career Path

(06:32 – 10:28)

• Evans describes himself as a computational scientist, a “meta-scientist” who studies the mechanisms and patterns behind scientific progress.
• Building new academic homes (e.g., computational social science, data science) has, for Evans, “created an instability that allows me to create new things, which is fun.” (Evans, 09:00)

Memorable Quote:

“If you just do conventional early on, you will likely do conventional forever. But if you have a portfolio … some of which you can purpose to present conventional leadership, then that’s …”
— James Evans (09:53)

3. Data-Driven Science of Science

(10:28 – 13:30)

• The field now leverages massive data to understand how science works. The “coin of the realm” for scientists is credit, leading to highly trackable contributions—unlike many other domains.
• Modern computational approaches, including AI, enable the study of knowledge production at a scale, making it an “unprecedented phase transition” in understanding progress.

Memorable Quote:

“Because scientists are incentivized to leave droppings ... it provides a rich landscape for thinking about collective knowledge and individual contributions.”
— James Evans (11:19)

4. How Scientists Choose What to Study: Innovation vs. Imitation

(13:30 – 20:55)

• The tension between working in established areas (reliable, but perhaps less groundbreaking) and pursuing quirky or innovative topics (riskier, but possibly disruptive) is central.
• Evans uses large-scale data and AI models to map the structure of knowledge, enabling identification of surprising or improbable ideas as they emerge.
• Groundbreaking Grant Announcement: Evans’ team received $20M to build “chronologically trained language models” that can evaluate how surprising a new discovery or patent is, and highlight subsequent adjacencies that become probable.

Notable Segment:

“Innovation is something that you can only study with large-scale data ... you need the full distribution of expectations that normal scientists would have so that we can understand when and why they’re surprised ...”
— James Evans (16:15)

5. Surprise and the Theory of Abduction

(22:07 – 27:09)

• Historical context: Peirce’s philosophy of abduction posited that the signal for progress is surprise—when observation/experiment violates expectations.
• True breakthroughs often emerge from interactions across fields—outsiders bring unexpected solutions to the problems insiders recognize as surprising.
• Evans: “Those … abductive discoveries are mergers … conversations between insiders and outsiders. … That’s where disruptive advance occurs.” (25:51)

Memorable Quote:

“You have a surprise in your field and you look … for resources that come from the most disconnected fields from yours. … That’s where disruptive advance occurs.”
— James Evans (25:50)

6. Bandwagons, Bubbles, and the Ecology of Innovation

(28:25 – 36:45)

• Too much disruption leads to chaos (no memory, no accumulation); too much conservatism leads to ossification.
• In some nations (e.g., China), innovation may appear high at individual level, but institutionally everyone may follow trends—limiting topic diversity.
• The persistence of old-guard influence (“priests” of the fields) leads to preference for established work; demography deeply impacts idea churn.

Key Stat:

“Everyone’s favorite paper was published on average a year before their first paper. ... About year 10, scientists start policing their field … start criticizing other papers … young people’s papers.”
— James Evans (35:56)

Notable Quote:

“If you have a high proportion of old people in your field … the field hits rigor mortis and no new ideas come in and new ideas are getting shot down …”
— James Evans (36:47)

7. Incentives, Prizes, and the Ageing of Innovation

(37:54 – 38:59)

• Funding and major prizes increasingly go to older scientists and established ideas, reinforcing conservatism.
• Disruptive work is often undervalued in prestigious prizes: “The things that are the most disruptive … are not the same things on average that are getting prizes, the Nobel Prizes, etc.” (Evans, 38:52)

8. Personalities, Values, and Hidden Biases in Science

(38:59 – 46:23)

• Disagreement in science arises not just from data but personality and cultural zeitgeist.
• Deep preferences—familiarity, beauty, simplicity—influence what is accepted as valid knowledge.
• Evans’ research shows that these hidden epistemic standards can drive or inhibit shifts toward new paradigms.

Notable Quote:

“Let’s just imagine … some of the decisions to pursue or not pursue areas were not rational … Now we can build ... large network models … to run science fictions.”
— James Evans (45:05)

9. Counterfactuals and ‘Science Fictions’ to Expand Possibilities

(46:23 – 49:51)

• Using AI models and simulations, Evans’ lab creates “conferences that never existed” and explores what knowledge might emerge in alternate histories.
• Cross-disciplinary citations (flowing “against the grain”) are strong predictors of disruptive, creative work—often sparked by social proximity like being at the same institution (or even married).

10. Team Science: Size, Structure, and Innovation

(49:51 – 55:40)

• Larger fields and teams tend to produce more, but not more disruptive work. Size drives incrementalism; small, flatter teams foster risk and fusion of ideas.
• Most fields do not have “enough” innovation relative to what would maximize discovery.
• Small teams are nimbler, exploring forgotten or outlying corners of the intellectual landscape.

Memorable Quote:

“Big teams … produce papers really quickly … They take huge popular ideas of yesterday and momentum invest. … Small teams … dig deeper into the past, dig deeper across fields … more predictive of advances likely to be important in 5-10 years.”
— James Evans (52:18)

11. Failure, AI, and the Shifting Nature of Understanding

(55:40 – 63:04)

• High rates of innovation require high rates of failure: “If you want to increase the likelihood of disruptive success, you have to increase your failure rate.” (Evans, 56:00)
• AI’s rise enables modeling of complex, self-dissimilar systems—potentially generating new, non-human-understandable forms of knowledge (as with AlphaFold2).
• Increasingly, science may involve interpreting the “hermeneutics” of machine-generated artifacts that are “unsatisfying as a scientist … but that’s what we know.” (Evans, 62:07)

12. Can AIs Be Creative? Diversity in AI and Human Knowledge Ecosystems

(63:04 – 72:30)

• AIs currently narrow scientific idea scope, but with deliberate architecture—a community of diverse, “rival” AIs—they could foster real creativity and speculation.
• The key is not to collapse diversity: “We need to create an ecology of AIs that ... enhance the population genetics from which future innovation becomes possible.” (Evans, 71:13)
• If all boundaries are erased, true intellectual capital is lost to mush; fields need membranes as resources for cross-pollination.

Notable Quotes:

“Fields are absolutely critical in this connected age. They need to hold their own standards … If you get to the point where everything is interdisciplinary ... [you] destroy the very ... epistemic assets that ... become building blocks in new knowledge.”
— James Evans (71:56)

13. Teaching the ‘End of the World’: Foresight and Agency

(72:30 – End)

• Evans discusses his “End of the World” course with Daniel Holtz at UChicago, teaching a long-view perspective on human survival, risk, and future thinking.
• Standard scientific methods aren’t always suited for reasoning about singular, low-probability catastrophic events—so narrative, speculation, and science fiction become important epistemic tools.
• The course provided students with “a sense of agency and possibility and also a long view … We don’t think about this unfolding set of future possibilities …” (Evans, 76:38)

Host’s Final Word:

“There’s a tricky balance, right, because you want to emphasize the very real worry that disastrous things could happen, with the lesson that ... there’s still things we can do about it, right? ... Be alarmed, but don’t despair.”
— Sean Carroll (76:51)

Selected Memorable Quotes (with Timestamps)

• “The world is very different than in Dirac’s time. … There are big collaborations … but even theorists … can change what they’re doing.”
  — Sean Carroll (04:57)
• “If you just do conventional early on, you will likely do conventional forever.”
  — James Evans (09:53)
• “Innovation is something that you can only study with large-scale data … you need the full distribution of expectations that ... scientists would have so that we can understand when and why they're surprised.”
  — James Evans (16:15)
• “You have a surprise in your field and ... look … for resources that come from the most disconnected fields from yours. … That’s where disruptive advance occurs.”
  — James Evans (25:50)
• “If you have a high proportion of old people in your field … the field hits rigor mortis and no new ideas come in and new ideas are getting shot down …”
  — James Evans (36:47)
• “Big teams … tend to … momentum invest. … Small teams … are much more predictive of [lasting] advances.”
  — James Evans (52:18)
• “If you want to increase the likelihood of disruptive success, you have to increase your failure rate.”
  — James Evans (56:00)
• “We need to create an ecology of AIs that … enhance the population genetics from which future innovation becomes possible.”
  — James Evans (71:13)
• “If everything is interdisciplinary … you destroy the very … assets that … become building blocks in new knowledge.”
  — James Evans (71:56)
• “Be alarmed, but don’t despair.”
  — Sean Carroll (76:51)

Noteworthy Segment Timestamps

• Introduction & framing: 01:29–06:32
• Evans biography & meta-science: 06:32–10:28
• How data shapes the science of science: 10:28–13:30
• How research topics are chosen (innovation vs. consolidation): 13:30–20:55
• Surprise, abduction, and collective intelligence: 22:07–27:09
• Bandwagons, ossification, and demography’s impact: 28:25–36:47
• Institutional conservatism in prizes and funding: 37:54–38:59
• Personality, values, zeitgeist, and biases in science: 38:59–46:23
• Counterfactuals, alternate conferences, & cross-field flows: 46:23–49:51
• Team size, structure, and innovative output: 49:51–55:40
• AI, interpretability, and the new epistemological landscape: 55:40–66:59
• Creating creativity in AIs and human knowledge systems: 66:59–72:30
• The “End of the World” course and long-view thinking: 72:30–End

Conclusion

This episode provides a compelling and empirically rich exploration of how science actually evolves—how its social structures, incentives, and cultures both fuel and stifle novelty. Evans advocates for deliberately designed ecologies, both of people and of AIs, to balance stability and surprise, preserving diversity as a resource. The conversation closes with a call for humility, foresight, and optimism as we face global existential risks—and as we seek to build scientific processes for the worlds to come.