38. Running Better Experiments at Work

Summary

---

Podcast Summary: At Work with The Ready

Episode 38 – “Running Better Experiments at Work”

Hosts: Rodney Evans & Sam Spurlin
Date: December 1, 2025

---

Episode Overview

Rodney and Sam tackle the essential but often misunderstood topic of experimentation in the workplace. Rather than arguing for why experimentation is important—an idea they assume listeners are sold on—they focus on how to actually run experiments that work within organizations. They share hands-on insights about designing experiments, managing expectations around failure, and the often-overlooked cultural and structural barriers. The tone is candid and witty, with a balance of practical advice and philosophical musings on organizational complexity, leadership resistance, and the emotional realities of learning through failure.

---

Key Discussion Points & Insights

1. Why Experimentation Matters in Modern Work

• Complexity Demands Experimentation: True organizational change requires accepting that most systems are too complex for perfect planning. The only path forward is “an iterative experimental approach” ([05:47], Sam).
• Science vs. Business Mindset: Despite organizations understanding the value of experimentation in fields like drug development, they resist applying the same logic internally, hoping for certainty where there is none ([08:34], Rodney).

2. Barriers to Effective Experimentation

• Cultural Resistance: Fear of failure and performative cultures (those that pay lip service but avoid real risks) keep experimentation from thriving ([04:35], Rodney).
• The “Messy Middle”: Experiments that are neither lightweight nor full-steam organizational initiatives tend to stagnate—they lack sufficient buy-in without being small enough for rapid learning ([18:43], Sam).
• Authority and Resourcing: “Usually what really good experimentation is missing... it's authority and resourcing” ([10:30], Rodney). Lack of clear ownership and time, funding, or permission often dooms ideas to the "side of desk" graveyard.

3. The Right Scaffold for Experiments

• Disciplined Structure: For experimentation to work, teams need “just enough scaffolding”—enough clarity, resources, and authority, but not so much bureaucracy that it stifles initiative ([09:49], Rodney).
• Performance Marketing Example: The Ready’s own disciplined experiment in performance marketing succeeded because of explicit roles, investment, timeframes, and “a bunch of people...shutting up and letting that happen” ([12:40], Rodney).

4. Experimentation Mindset and Posture

• Posture, Not Mere Process: Success depends not just on the experimenters’ approach but on the posture of the whole team—trusting, patient, non-interfering ([15:48], Sam).

  "You need everybody around experiments to also have that posture. Otherwise, you've got this mismatch..." ([16:32], Sam)

• Most Experiments Should Fail: True learning only happens when failure is acceptable and expected.

  “If 90% of your experiments are successful, they're not really experiments.” ([17:40], Rodney)

5. Sharing and Scaling Learning

• Transparency is Key: Local experiments are most valuable when their learnings are made visible across the organization. However, achieving useful knowledge management remains a huge challenge ([23:20], Rodney; [24:20], Sam).
• Context Matters: Failed experiments in one area can still offer valuable lessons or positive results elsewhere. The trap is assuming a single result applies universally ([24:20], Sam).

6. Emotional and Practical Realities of Experimentation

• Messiness, Sunk Cost, and Hard Calls: Deciding when to “kill your darlings” is emotionally difficult; most experiments won’t have a clear win/lose outcome, and discernment is a messy, subjective process ([32:12]–[34:40], Rodney).
• Limiting Work in Progress: Running fewer experiments at once supports better focus, learning, and decision-making.

  “I'd rather actually limit experimentation in progress, have sort of a fixed number of them running at any one time and finish one before we start the next one.” ([35:05], Rodney)

---

Notable Quotes & Memorable Moments

• Opening Banter:

  “Undermining scientific method. How dare you, sir? The scientific method—the foundation of our modern world.” ([00:00], Rodney)

• On the dangers of over-controlling experiments:

  “We want speed, we want experimentation, but then also we want...a lot of control and a lot of say in a lot of things that don't matter. And you can't actually have all of that at once.” ([15:18], Rodney)

• On killing your darlings:

  “Prepare to kind of hate this because no experiment gets launched because everybody thinks it’s a stupid idea... most fail, and you’re likely going to have to kill your darlings.” ([47:58], Rodney)

• On disappointment and learning:

  “Maybe the glib thing... is the true measure of an experiment is not whether or not it worked but whether we learned from it. Which... is true... also, is hard.” ([48:11], Sam)

---

Timestamps for Key Segments

• 00:00–03:37: Thematic check-in—personal experiments from Rodney (journaling with ChatGPT and astrology) and Sam (analog alarm clock)
• 03:36–08:34: The case for experimentation in complex/adaptive systems; parallels with drug development
• 09:35–13:45: Where organizations tend to fail: bureaucratic vs. chaotic approaches, missing authority and resources; performance marketing case study
• 14:42–18:43: The “messy middle” of experiments—why midsize, half-committed tests often stall; importance of skin in the game
• 22:20–25:22: The challenge of scaling learning and knowledge management from experiments
• 27:14–28:14: Fighting traditional business school mindsets; experiments aren’t “random mutation” but context-sensitive application
• 32:12–34:40: Emotional difficulty of experimentation, sunk cost fallacy, judgment, and decision-making messiness
• 35:05–36:02: Limiting work in progress and finishing experiments before launching new ones
• 37:37–39:14: Practical tips for creating more space and cultivating a learning mindset
• 39:14–44:22: Experiment template walkthrough—a practical tool for structuring and clarifying proposed experiments
• 45:04–47:58: Only design experiments you personally can steward; avoid “solutioneering” for others

---

Practical Tips & Takeaways

How to Cultivate Better Experiments at Work

• Make Space: Block out dedicated time for learning and experimentation, not just performance ([37:37], Sam).
• Use an Experiment Template: Define tension/opportunity, background, proposal, options considered, facts, assumptions, constraints, risks, impacts, and decision-makers ([39:14], Rodney).
• Clarify Authority: Ensure it's clear who can approve, resource, and execute the experiment ([44:22], Rodney).
• Only Suggest Experiments You Own: Don’t propose projects for others without their buy-in—focus on ideas within your span of control ([45:04], Sam).
• Expect—and Prepare for—Failure: Disappointment is inevitable. Build in emotional resilience and open retrospection ([47:58], Rodney).
• Value Learning Above Outcomes: The most important outcome is insight, not just success or failure ([48:11], Sam).

---

Wrap-Up

Rodney and Sam close by reinforcing that building an experimentation muscle is as much about culture and mindset as it is about process. The "real work" is emotional, practical, and ongoing—requiring courage, humility, and patience.

“All right, Sam. I think we’ve given them what we can on experimentation. Now they just have to go do stuff.” ([48:47], Rodney)

---

Have an experiment to share or a work pattern you can’t crack?
Contact: [e-mail]

---

Engineering: Taylor Marvin
Production: Jack Van Amberg
Created by: The Ready

---

Transcript

[A]

Undermining scientific method. How dare you, sir? Well, scientific method, the foundation of our modern world.

[B]

Honestly, honestly, it's like this is how we discovered literally everything that was interesting that ever happened. Hey everybody. Welcome back to our work with Ready, I'm Rodney Evans and the man on the mic is Sam Sperlin.

[A]

Hello.

[B]

AI continues to rewrite the rules of work in this moment. We feel like the future of work is here though. And now the question isn't whether or not you have a choice about adaptation, but how you're going to design work from this moment forward for what's coming.

[A]

Said another way, work design is no longer optional and the teams that treat it like a side project are actively being left behind. And the ones that treat it as essential will keep up with the pace of change.

[B]

So today we're going to dig into experimentation. I saw a post the other day from Sam Altman and he was talking about like the kind of company you want to build. And honestly, not to be a fucking know it all, but like it's the same shit we've been saying for 15 years and having clients be like, are you sure? And now, now, now, now, now we're going to really do it right guys, we're going to really do it because we have no choice. So today we're going to talk about experimentation, which is one of the foundational ideas that is so poorly executed in most systems. But first we're going to check in.

[A]

We are going to check in and we're going to keep it thematic. Rodney, what is a personal experiment you've done recently or are thinking about doing.

[B]

The one that springs immediately to mind? Everybody out there who liked Adam Grant's post debunking Astrology can just close your ears, just earmuffs for the next little minute. Also, because I definitely want to argue with people about this, not because I think astrology is science, but because I think it can be useful and I think it's annoying when people are just butt heads about it. But fair, I digress. I have been in a months long conversation with ChatGPT that has the backdrop of my astrological chart, but actually I just like feed it content more like a journal day to day and talk about sort of the weather of my life in an ongoing way and I have found it so useful I can hardly overestimate the ways in which it has helped me like process things, understand things, not react to things that I normally would. It's just been a great experiment for a few months and I'm finding it much Stickier than journaling and much more useful than a lot of other things I've tried. Cool.

[A]

Love it.

[B]

What about you?

[A]

I bought an analog alarm clock.

[B]

Mmm. Why?

[A]

Because previously I used my HomePod, which is a fine alarm clock, except it has this very deep flaw, which is you can use your voice to turn it off. And. And to use your voice, you can remain horizontal in your bed. And it was becoming a little bit too easy for that to happen. So now I have to walk across the room and turn off an alarm clock. And my little trick on this is that my normal alarm goes off at the normal time and the physical analog alarm clock goes off one minute later. So if I don't want to hear it because it's horrible, I have to get up and preemptively turn it off. And if I don't do that, then I will hear it. It'll make my wife mad. That's the kind of incentive I need to get up in the morning.

[B]

You're like a teenager. I don't remember the last time I woke up to an alarm. I just wake up. That's so.

[A]

That's.

[B]

I can't even imagine what that would be like. It would be so nice to still be asleep. Okay, so let us get into this pattern. Here's how I see it. Out in the world, companies say they want experimentation. Not all of them, but some of them. A lot of leaders, I think, understand that experimentation is the key to innovation. But what they don't have is the foundational sort of learning mindset culturally to do it. So even in companies that say they're going to, like, fail fast or move fast and break things, we still often see an underlying fear and performative culture that prevents true experimentation from taking place. The sort of bastardized version of experimentation is something that's either overly planned and sort of no fail, which isn't really an experiment or something that's really chaotic, where people have just, like, gone rogue and in a fairly unstructured way, are trying. Shit.

[A]

Yep. Yep.

[B]

In either case, when those experiments fail, and they generally will, leaders often take that as proof that experimentation wasn't the move to begin with, and the pattern sort of repeats itself. So we're not today going to, as we have before, try to convince anyone that experimentation is the key to change. We're just gonna assume that everybody listening to that podcast is already Gucci with that, and instead we're gonna talk about what is required for experimentation to function.

[A]

Yeah.

[B]

Yeah.

[A]

Okay.

[B]

Okay, great.

[A]

Let's get into it. I was going to start with, like a brief. Like, why is experimentation so central to this? But you've made the case that we already. Everybody know.

[B]

No, it's okay. You can. You still can. Maybe there's a new. I mean, we do have a lot of new listeners. Maybe there's somebody who's like, experimentation. What?

[A]

Yeah, and we can do it really, really tight, I think. And this. I feel like sometimes when we talk about experimentation or people are assuming that somehow we're just like these fancy weirdos who like, want to be different. And like, we. We're not going to use the established change management theory of the last a hundred years. Like, we just need to be special and different. And I mean, maybe, but no, it's.

[B]

I mean, yes.

[A]

And it's about this idea of complexity and whether or not you believe an organization is a complex adaptive system. If you do, then the experimentation follows that because a complex system is inherently unknowable as a complicated system, meaning that you can't just plan the perfect approach and. And then execute it and know what is going to happen. So if you agree that you're dealing with a complex system, the only way to have productive interactions with it is through an iterative experimental approach. So it's a. We're coming at this from a very pragmatic point of view, not like as a bunch of fancy duchesses and dukes.

[B]

Yeah, I mean, I also want to be that. But it's interesting that you say this, and I'm glad you started here, even though I said we weren't going to, because I worked for a while with a large pharmaceutical company in my time at the ready, and I worked with, like, a therapeutic area, and they were doing drug development. And it's the closest that I've ever been to that kind of R and D. And it's so interesting that in that kind of field, it is so obvious that when you're trying to create a new drug, the human body is a complex system, and you're gonna have to do a lot, a lot of stuff to figure out how the drug is going to impact the human body. Like, it's so obviously like in the posture, in the mindset, in the DNA of people who are doing something like drug development to have that understanding, to have the understanding that experimentation with ever increasing fidelity and ever increasing risk is how you get a new drug to market. And yet in a company which presumably is exponentially more complex than a single human body, we're like, what if our thing, we could just perfectly plan what all of the consequences and second and third order effects would be. And then preemptively mitigate those and then just roll it out. And I'm like, how does anybody not see the parallel between these things? And go like, you would never do that with a drug. You would never make a two year roadmap of how a drug is going to get into the population and then just go step by step without doing an experiment. I mean you could, but you'd be like, you'd kill a lot of people and you'd get like real sued bad. So it's just so interesting to me how even though science has this very well hewn position on how you create something that is safe and useful, so many companies just refuse to engage in that way.

[A]

Yeah, yeah. It is really interesting to think about it. I think the inscrutability of the human body and similar sort of biological systems I think allows us to have that mentality more easily. Whereas in an organization, because they are collections of humans and we have created a lot of human sized ways to interact with it, we think that we can control it a lot more than we actually can. And not to make this an AI episode, but I think this will actually be an interesting thing to follow with AI as it is experimented with more and more in organizations. Where does it break up our kind of human sized conception of organizations in a way that actually makes organizations more inscrutable and therefore more amenable to a true kind of iterative experimental approach?

[B]

Right. I think that's so true.

[A]

All right, we did complexity 101. Yes.

[B]

Yeah, it's not a watch, it's the weather. Carry on.

[A]

Yeah, no, we're done. We're good. So.

[B]

Okay, we're done.

[A]

Okay, yeah, we're done with that part. Now we're going to talk about why and how it is. So it seems to be so difficult within organizations. And it seems like we talked in your, in the pattern, you kind of talked about both ends of a continuum that aren't great.

[B]

Yeah. So I'm going to talk about actually what I find to be missing from both. So from both the overly bureaucratic and the overly chaotic side, which is the third way here, is just enough scaffolding for experimentation. So on the bureaucratic side, what you have is people who, before the experiment ever even kicks off, they want to know what the key results are going to be, how they're going to track it, it, when it's going to succeed, what is going to happen and on what timeline. And the most important thing is steps in a plan, ownership and tracking. That's not what I'm talking about when I say scaffolding. Usually what really good experimentation is missing. If I had to say two things, it's authority and resourcing. That's usually where I see experimentation fall down. So somebody has an idea for something that they could test, it becomes something that is either like side of death or unfunded or unclarified. Or what happens, I think, with a lot of good ideas is it's unclear whether the individual with the hypothesis has the authority to actually run the experiment. So they're like, I think that we should change our budgeting process. And it's like, you know, the system fights back. The system is always going to fight back. So unless you are the finance person in charge of the budgeting process, like, you're immediately going to run into a wall with that one. And so I'm saying this because I think that when we want to run a really great experiment, we have to have a really clear space and a really clear lab to do it. And I'm going to give an example from the ready, because you can pile onto this one. And it's a place that I think we have done really disciplined experimentation. And it's with performance marketing. So we have a member who has a lot of experience in marketing. For a variety of reasons. We realized that it's you guys, you listeners, you're it. We realized that, like, basically we are talking to ourselves like we're in a bit of an echo chamber. And people who listen to us and consume our content have been around for 10 years. And also we're just talking to you guys. We want to talk to more people, not because we don't love you, but because there are a lot more people out there in the world. And so we were like, how do we essentially broaden our market? And the experimentation around this has been hyper, hyper disciplined. So it started with a clear proposal, a clear timeline, a clear investment, clear role charters from the people involved. Clear, like in three months we would expect to see this. And we're going to start tracking the data at this time to see the delta between what we believe and what's actually happening. And real clarity with me of, like, you know, Rodney, as the steward of this company, like, we need you to do these things and then also to, like, willfully ignore these other things because you're just gonna put sand in the gears and be annoying. And it has worked because we gave it enough oxygen and enough time and enough space and enough clarity to know what we were Testing, to run real tests and to see real results and to use those to steer. Now, I'm not going to talk about the outcome because first of all, that's still tbd, and second of all, it doesn't really matter. The idea here, though, is, like, a fair amount of work went into clarifying the petri dish, and one person with the support of a team saying, this is what would be needed to run these tests well enough to get to learn. And then a bunch of people, frankly, shutting up and letting that happen. That almost never happens in companies that I see.

[A]

Yeah.

[B]

And it doesn't happen for six months. It happens for, like, six days. And then somebody's like, so where are we with that? Can we get a status. Should we get a status meeting on the calendar?

[A]

Yeah. Yeah. Well, I think the way you describe that, what just kind of clarified for me is what one could listen to what you just said to me. Like, isn't that, like. It's like a ton of upfront work? Isn't that what we're trying to, like, push up against? It was a ton of upfront work to create the space in which a lot of very organic and trusting and quick and autonomous work can happen. And I think a lot of times that is flipped in an organization. Like, we don't really ever define where we're going to do this or what the constraints are. And instead we are all in the mix, arguing over all the ad copy. It's like, well, that's right. That's not actually a good use of our time. And not really. We're not really experimenting with anything there other than, like, how much we can test each other's patience.

[B]

That's. I think that's exactly right. And to that point, it is. I would say it is more work upfront than how this kind of thing, which is really any new thing, normally gets launched to create that space. You're right. And once it's running, like, someone who is external to the ready looked at that work probably three months in and was like, you guys have done, like, 16 months of work in three months. It's, like, pretty astonishing in terms of the data gathered, the number of tests run, et cetera, et cetera. And this is why, like, when people say they want speed, it's because we weren't futzing with pixels and ad copy and arguing about palettes. And, like, we, like, let the experts do their thing and, like, let it run. And this is. This is where the stuff stumbles in, in organizations. And so we hear we want Speed. We hear we want experimentation, but then also we want like a lot of control and a lot of information and a lot of say in a lot of things that don't matter. And you can't actually have all of that at once.

[A]

Totally. This is why I find myself frequently talking about experimentation as a posture within an organization, which when I say that word, it's basically pointing to the idea that it's not just the moves you described, but it's how all of us interact with that around it. So the idea that we're not expecting the folks who are doing this experiment to show up a week later with everything figured out, the understanding that we don't have to understand everything that's happening within this experiment and that there are, we. We trust that there are checkpoints built in and when we need to weigh in on something, we will weigh in on it. And this won't be a runaway process that just runs forever. I think.

[B]

Right.

[A]

You need to everybody around experiments to also have that posture. Otherwise you've got this mismatch where the people who are actively running the experiments are like, yeah, you want me to be doing this experiment, but really you just want me to tell you the answer. And that's right. Is a recipe for not really learning anything.

[B]

Well, and further to that, really what you want is me to tell you it worked, right?

[A]

Yeah, the answer. And we found it because it worked. Yeah. Not the answer because like it didn't work. And we're pointing to something else.

[B]

And this is one of the fundamental problems with experimentation in a lot of companies is that my super hot take is most experiments should fail.

[A]

Hot take zone. Right.

[B]

Like, I think you want to run a lot of them and I think you want to kill them quickly. And what that means is, you know, we've been saying for years it's an Aaron dignity. It still slaps like, you know, radical change at a non radical scale. But this is how you get that. It's like do really weird shit, but do it really tight and not for a very long time. If 90% of your experiments are successful, they're not really experiments.

[A]

Yeah, I completely agree with you. And as I was preparing for this episode, I thought about something in a different way that I want to run by you and see if it tracks to your experience as well. Because I do think I agree with you. But the thing that I landed on when I was thinking deeper about this is I see two many experiments that are kind of in the messy middle size wise. And I think it's much More interesting to do very small micro experiments that are light and easy to spin up and learn from and have very small blast radiuses or doing something that requires greater coordination, greater cohesion. Maybe a little bit of planning up front like the performance marketing one that you talked about that is involving lots of roles and lots of teams and. And though I see those as kind of different ends of a continuum and both are cool and interesting. And it's the stuff in the middle that I think sometimes ends up being the most challenging and the most difficult to actually execute and learn from.

[B]

Mm. What's an example of something in the middle?

[A]

Something in the middle is like a couple of teams kind of half assing something together about their workflow or about how they're working together where it requires kind of more thought up front about how does this actually work and interface with other parts of the organization. And we're kind of just trusting that we'll just try something and we'll learn from it as we go. Which I think sometimes works. I think what it comes down to is you don't have. People don't have enough skin in the game. I think the middle one is when there's no, not enough skin in the game and we're just like kind of half assing a thing. Whereas this really small stuff, you don't need to have a ton of skin in the game. Like you can buy an alarm clock and see if it helps you wake up easier.

[B]

Sure.

[A]

And the stuff that requires much more skin in the game we've created the structure for. It's that middle. That middle amount of commitment that is the problem.

[B]

Yeah, that's interesting. I mean I think that like my reframe on this because I think a lot of times that like messy middle is like we're going to make a thing that we think is missing. I think that often is the experiment that happens in the messy middle that nobody actually uses. It's like we're going to document our onboarding workflow. I often think that the messy middle of experiments is we see something missing and we're going to make it. And I think a more interesting reframe on that is can you go and validate what the problem is with the missing thing as the experiment?

[A]

Is it actually missing?

[B]

Is it actually missing? I think too many experiments at the team level result in documentation that nobody gives a shit about. Right.

[A]

If we gave a shit about it, in most cases it would already exist. Yeah.

[B]

Because if it was a real problem, like it would have been documented and also usually I mean, documentation doesn't solve that many problems in a complex system, if we're being honest.

[A]

We like to think it does.

[B]

Yeah, we like to think it does. But like, yeah, what does it really do? So, yeah, I think that's really interesting. I do think the altitude of experimentation is a tricky one. It's a tricky nut to crack. And there are also like some interesting things I've heard recently. Like, I just saw someone, I was at the Charter conference last week, shout out to my buddies at charterworks and someone on stage was saying that in her company, which I believe was an AI native company, but in her company there's basically a marketplace of agents and any employee can create an agent and they effectively get deployed and upvoted very like organically within the system. And now that to me is like a very contained way of doing experimentation because basically I can build an agent to address my problem that I think is worth addressing. And then if you also think it's useful, you can use it. And if you don't, it doesn't matter because I still have the thing that's doing what I want. And it doesn't really require team coordination. There's a social component of it, so that you're seeing which things are solving organizational problems, blah, blah, blah. That's a great use case for experimentation. And what I loved about it is actually the open source nature of like. And then anybody can redeploy these and we can see like what really slaps that feels like a really clear, like small things that add up to big things, but the gnarlier things, like, you know, we want to change the OPERM with our board. It's like, you probably want to like do that really well one time and see what happens rather than trying to architect something for a year.

[A]

Totally. And that gets me thinking about the opportunity of scaled experimentation. And by that I mean I don't mean huge experiments. I mean lots of experiments in huge organizations. And I think you just described an example of how experimentation needs to be paired with transparency. You know, marketplaces are tools, are mechanisms for transparency. And I think the huge opportunity in a lot of organizations that I haven't necessarily seen done super well lots of times is how does the learning from local experiments scale up into other parts of the organization? How frequently so hard are we learning from what others have done in their experiments for our own inspiration or for our own tweaks? That that is, I think, a huge opportunity that is really hard to do.

[B]

It's so hard to do. You know, there's no one and done answer here. Certainly I think having trained GPTs is a really helpful way of doing this because people can pull the information that they actually want rather than trying to consume an entire slack feed of demos or wins or whatever. But all of that being said, it's still really hard. Like, I think that the sort of like knowledge, I hate to say knowledge management, but like the knowledge availability part of experimentation in a complex system is something that I see so few organizations really able to crack. I think it's incredibly difficult because it's not just is the information available and pullable, it's like, do people know that? Do they know what's there? Do they know when to ask? Do they know how to apply it? Do they have time in their day? Is it something. They're incredible. It's just like it's a whole ball of string. That again really relates to having learning being part of your culture, which is pretty rare.

[A]

Totally. Well, plus the added complexity of just because one team or one part of the organization experimented with this thing and it didn't work. The takeaway, the simple, the overly simplistic takeaway is like, okay, won't try that anywhere else, do that anymore. But like that is absolutely not the takeaway. If it's like a different department or a different function that does a complete. Has a completely different purpose than yours, maybe, yeah, it wasn't a good fit for their contacts, but it might be for yours. And yeah, it's such, it's such an interesting thing. And I, I wonder how and if it can really be cracked in large organizations. Because I think this is one of the, this is one of the benefits of being a huge organization. If you were able to do this really well because you are just inherently able to test more things more rapidly and if you can, if that is connected to learning more and learning quickly, then you are at such an advantage. But I think it's in most legacy organizations which are almost by definition older, they were not built on this DNA and have not built it for themselves yet.

[B]

Yeah, I think that's right. You know, like a lot of other things that we've talked about and sort of myth busted on this show. I feel like this is also one of those things that's really easy to sort of like mock. Like, like I definitely have had the interaction with a trad leader who's very like MBA ish, who I think finds experimentation unserious. And to your point about being like fancy weirdos, I've definitely had the interaction where I think Someone was like, in an attempt to defend their position, trying to undermine this idea by discrediting my understanding of how work is and, like, how real business goes. And like, first of all, who fucking cares?

[A]

Undermining scientific method. How dare you, sir? Well, scientific. The foundation of our modern world.

[B]

Honestly, Honestly, it's like this is how we discovered literally everything that was interesting that ever happened. But to your point about what organizations are built on, you know, management science, which is what the Harvard Business School was founded on, the first principle is there is a right answer. The first principles are through analysis and research, you can be correct. And so we really are. When we have these conversations with leaders who have been socialized and educated in traditional and especially in elite institutions. We are fighting against 130 years of history at this point and trying to unlearn the idea that a right answer is uncoverable with enough analysis and debate and into the mindset that we're not sure what the right answer is. And all we can do is tests.

[A]

Totally. And I think maybe a way, a way that I have sometimes taken a couple of steps in their direction and maybe break down some walls is that sometimes I think that there is a misunderstanding that what we are saying is just try random shit. We're not talking about random mutation here. We are talking about practices and ways of working and processes that we have seen work in other contexts. And there's good foundational, like people have thought about them, they have used them in other contexts. And we're saying, try this thing that has worked in other places in your context, not try this super random thing that makes no goddamn sense and maybe it'll work here. Like that's. We're experimenting with whether an idea works in this context, not the, the idea itself in most cases. Not saying that you can't come up with something totally random and new, but that's generally not what's what the vast majority of experiments are.

[B]

Yeah, I think that's right. What have we not hit?

[A]

I think one thing that I wanted to say, and Jack may cut this if it doesn't really go along with the flow of the conversation, but I think it is useful to also think about the system's reaction to an experiment as data about the experiment that is, that is useful data.

[B]

Say more.

[A]

So I, I think, you know, let's take something like, something really simple. We change the operating rhythm and people are asked to show up in a different way. The people who are not actually involved in the meetings. And there is some aspect of them being like, I could Picture both ends of this really positive or really negative, like, who are you to tell us to meet in a different way or that our meetings are bad or. Or the positive version of that is, like, what? Like, what's happening over there? Like, what, what? Like, I'm noticing that people who are involved in this thing are, like, happier and having a better time. I think sometimes we limit our view of what an experiment has accomplished to the people who are in the experiment itself. So we're just asking the people in the algorithm, like, hey, do you like your algorithm better instead of, like, seeing, like, what else it changes around in the system? I mean, I think you could even think about it in more of an org design way, like, all right, we changed the OPER with an experiment over here, and suddenly there's, like, these conversations about strategy that have never really happened before over in this other place. Like, what is that telling us? That's interesting. And maybe we follow our nose that way so that our next experiment has nothing to do with algorithm. It pointed us toward this other thing that we wanted to dig into as well.

[B]

Yeah, that makes a lot of sense. Yeah, it does make sense. I mean, I think the other thing that is worth saying about the systemic tendency of experimentation is they tend to accumulate inertia. My experience as a steward of a company with experimentation, and we've run a lot of them in the last couple of years in a much more disciplined way than we used to. We used to be really fucking YOLO about experiments, and we often didn't get much from them because they were poorly scaffolded and underinvested and side of desk, and it was like, I don't know, is this person gonna, like, figure out a new line of business while they're fully deployed to a client just on, like, Sunday mornings?

[A]

Yeah, the. The YOLO nature of it was like, there was a very, very small chance that it was gonna work amazingly well and be obvious to everybody. And we pat ourselves on the back and the vast majority would, like, continue forever or fizzle out or nothing would really happen.

[B]

And, yeah, Yeah, I mean, I can count on, like, less than one hand the number of experiments from a bunch of years that I would say really, like, turned into something material over time. I say this because that was also a lot easier, actually. The more chaotic approach to experimentation, we're like, we're not super crisp on the hypothesis we're testing on when we expect to see results, on what we would like those results to be in the delta on when we're going to convey those and how we're going to measure and when we're going to steer and when we're going to shut it down and when we're going to. When we don't do that work, it's a lot easier just to be like, you know, there was an idea on the board that somebody, like, opens an ice cream store. Did Joe ever get around to doing that? No. Oh, well, like, we didn't spend any. It was kind of like nothing ventured, nothing gained. But the opposite of that, which was like, well, we didn't get anything, but it didn't cost us anything. Who cares? Let's do it again.

[A]

Which actually has a huge impact because what was invested was time and even just, like, mental capacity.

[B]

Totally.

[A]

Just, like, super rare in an organization. Like, unaccounted for time and attention is maybe the rarest resource in an organization.

[B]

Totally. But emotionally, it was easier because it was like, yes, yes.

[A]

Low risk and no skin in the game.

[B]

Yeah, no skin Again, back to the no skin in the game. Right. Like, and so now over the last couple of years of, like, taking bigger swings and doing bigger things, I will say, as someone who has seeded a lot of those things, or at the very least, helped to protect the space for them, like, it's not easy because first of all, sunk cost fallacy is real around experimentation. My experience so far is there are very few things that are, like, clearly ready to be killed. Like, especially for an optimist, there's almost always some kernel of something interesting, learned or uncovered in an experiment that you're like, should we just re scope this and keep going in this direction? That's 45 degrees off. It's very uncommon in my experience for it to be like, 0 out of 10 points awarded. Shut it down right now. That never happens. And so prepare for the messiness that, like, it's not gonna feel black or white and you're gonna have to make a call.

[A]

Totally.

[B]

One, two. The opposite of that is also true where when you feel like you're seeing signals and, like, you want to double down, that also takes courage because also in early experimentation, it's really rare to see undeniable proof that it's working right.

[A]

At that point, it almost stops becoming an experiment. It's just like, oh, okay, like, we do this now, right?

[B]

It's like, oh, we. We hit oil. So, you know, now we're just like, on the geyser. Like, it's. It does not go. I mean, I'm sure it goes like that. One in a hundred, but, like, My experience is that it's rare that it goes that way. And so I'm saying this because when you're a person like I am, who holds some authority and some discernment over these things, I just want to say out loud that, like, the messiness of it is real. The sunk cost bias is real, the searching for signal is real. It's really hard, actually to do this well, and that doesn't mean that it's not the right thing to do. But counter to what I said before, you're not in a lab doing very specific trials with a team of scientists who are getting very concrete data when you're experimenting on a complex system.

[A]

And.

[B]

And it does take a level of judgment and taste and opinion when you're deciding the fate of an experiment, that feels pretty uncomfortable. Like, it feels uncomfortable for me, and I think I have more tolerance for this stuff than most people.

[A]

How do you think, to the extent that you think you have gotten better at this over time? Like, what. What has allowed you to get better at it?

[B]

I think a couple of things. One is not doing too many things at once. So when you're using all the dry powder across a lot, a lot, a lot of bets in hopes that one of them pops first of all, then you're trying to do the kind of discernment game in too many places and not just, like, as an individual, but probably as some sort of, like, stewarding team. I'd rather actually limit experimentation in progress, have sort of a fixed number of them running at any one time and finish one before we start the next one. Right.

[A]

Because you can use that as a forcing mechanism to decide, like, we could keep this one going. But I see my backlog and do I want to keep this one going even over trying this other thing? And it, like, it forces you to get a little bit clearer other than just like, yeah, why not keep it going? Like, maybe it'll work.

[B]

Work. That's right. It's so easy to just be like, well, what if we do the other one alongside? But, like, there's only so much context switching and money and energy in any system. And so the limitation of the number that we're gonna do and also being like, if we want to do that when one of these has to stop is. Is actually really helpful. I think the other thing is, like, we have an H2 even over for the ready, which is now, I can't remember what the exact words are, but it's basically patience with experiments in flight, even over new experiments, and the fact that As a team, we agree. We chose that even over knowing what experiments were in flight. Has also helped me in the moments where I get a little fiddly, to be like, no, no, no, no, no fiddling. We are going to be patient with what is in flight until we reach undetermined conclusion, rather than launching a bunch of new shit because we're feeling away. I think those are the things doing it as a team sport. Limiting whip.

[A]

Yeah, I love that. I love that. I was just gonna say, you know, to point to our even over episode. What makes that a good even over is I could totally. I could picture a scenario where we would want the opposite, where we actually want to spin up a bunch of stuff and, like, try a bunch of new things and not double down on things we've already been doing.

[B]

Totally nice. I think that's exactly right. Okay, Sam. I feel like we've covered the territory of experimentation mostly what doesn't work. Let's give people some ideas. Why don't you start us off?

[A]

I will happily do that. So my first thought is broadly around how do you cultivate more space in an organization? I think that is often what gets in the way of orgs feeling like they can do some more experimentation on kind of the internal ways of working. Because we've used this idea of the learning mindset over and over. And it's really, really, really hard to be in a learning mindset if your environment is expecting you to be in a performance mindset or in a performance context. So to the extent that space can be made, things can be stopped to create a little bit more space to talk about and run and design experiments. I'm a huge advocate of that, which is often why the first batch of experiments a team or an organization or a department might do is around, how do we create more space? What do we stop? What do we do differently? How do we reserve a chunk of every two weeks for us to deliberately shift into a more of a learning mindset?

[B]

Yeah, I love that.

[A]

What do you got?

[B]

This is way more tactical. But one of the habits that has really taken hold over the decade at the ready is using an experiment template. And we've used this with a bajillion clients over the years. It kind of works every time. Like, it's kind of awesome in every single context.

[A]

And the reason it's always better than not using it. It doesn't always solve every problem, but it's always better than not using it, dude.

[B]

A hundred percent. And, like, I'll regularly get something shipped to me that has like a bunch of other stuff up top. And I'm like, blah, blah, blah, where's the template? Because the template is where I'm going to determine whether the thinking around this passes muster. So here's what's on the template. Define the opportunity or the tension. So what are we solving for? What do you need? Be specific. You only want one tension per experiment proposal. So don't be like, here's 50 things that are wrong. What's the actual thing that you're solving for? Relevant background. Give a little context about the situation. Proposal. What are you actually proposing to address the tension? We love this. Then other options considered but not recommended. Here's what is so rad about this field is that everyone who has ever sat in a meeting of any kind, who's heard a proposal of any kind, the immediate response is did you think about X? Here's your opportunity to be like, yeah, nerds, I did think about X and I wrote it down here so you don't have to ask now on the off chance that you have an idea in that meeting that is not captured here. Fantastic. But my experience is nine times out of ten, they in fact did think about X before I had a chance to be annoying about it. That's the content of the experiment that's being proposed below. That is additional context that I find incredibly valuable. So there's facts, what do we know, data points, constraints, et cetera. There's assumptions. What assumptions are we making? There's constraints. What constraints must be considered? What's non negotiable? That could be like we only have $10,000 for this, or this has to be done by the end of the year. Risks. What are the risks if a key assumption is wrong? So calling back to the other field, dependencies, impact to the system, other roles or people, what are some possible consequences of the proposal? And decision makers. Which person, role or group has the authority to make this decision? Okay, those are the fields of it. And there's prompting questions for each one that I didn't talk about. But what I find this gets you without actually it takes way, way less time than starting from a blank sheet of paper to try to explain something is it really makes explicit what we know and what we don't know. And that is incredibly useful because we're basically trying to get a group of people to say this is safe to try, even though we don't know a bunch of stuff. And that muscle in and of itself is so weak in most organizations and usually the conversation is how could we Know more rather than knowing that there are these gaps and assumptions and risks and whatever. Can we do it anyway so that we don't spend more time trying to know more, we can immediately start trying to learn more?

[A]

Yeah, yeah. I've experienced the difference between sitting in a decision meeting where there has been a proposal filled out ahead of time that we've all read and where there hasn't been. And boy howdy, is it such a better conversation when that proposal template has been thought through and filled out and people have been able to read it ahead of time and maybe even already drop some comments in. The quality of the conversation is so good.

[B]

Yeah, it makes my teeth itch when a client just sends me a document that's like 12 pages. It's like, we want to talk about this at the off site. And I'm like, oh my God, yeah, how? Like, how are we going to do that? So, yeah, I love a template now. One other pro tip about using it is decision rights. And like, authority is just sort of worked in most places. So our little template assumes that, you know, who needs to make the decision. I get that. That's like easier said than done in most places. And my sort of hack when I'm creating a proposal is if authority isn't clear, I try to get the people involved to one, will actually have to work on the experiment to make it happen or a representative of that. Two, the person or people who will have to fund it or like have an eye on the monies or the resource allocation or whatever. And three, someone who's going to be involved in the longer term maintenance. So if I'm like, we should make a course about AI, it's like, you should probably have the person who's going to have to make the landing page and create the workflow and do like think about the whole thing, not just the inception of the idea, in order to get consent, even if they're not all involved at the very beginning. Because what you don't want to do is run an experiment that turns out to be really, really rad and then have whoever's maintaining that work, whether it's OPS or technology or HR or whoever, be like, yeah, cool, we don't have any capacity to do this, so why did you make it? Totally.

[A]

Or you made some decisions early on that didn't seem like a big deal to you because you aren't an expert in this domain. And actually we've kind of borked ourselves by not making this alternative thing. Yeah, totally. I love, I love it.

[B]

What you Got okay, I've got one.

[A]

More, much more tactical than my, than my first one. So this comes from the observation that every time I work with a team who is thinking about kind of ways of working, experimentation, op model experimentation, experimentation, focus on the organization itself. Far and away the way things go off the rails is that everybody has experiments for other people to do. So everybody is coming up with experiments left and right and they're never about like what you actually have any sort of influence or control.

[B]

I love it.

[A]

So I always, I've made the rule that in these workshops you're not allowed to make an experiment for anyone else without being in conversation with.

[B]

Great.

[A]

Yeah, really what, really what that, that means though is just that like start with where you are and if you're a senior leader, you have more kind of freedom of movement and greater influence to propose experiments because you do influence more. But even as an individual contributor on a team, there are plenty of experiments that you can steward and own. Even if it's purely just the individual. How I show up to things or how I do my role and all of that is valuable. And building the muscle of identifying the levers that you can experiment with that you actually can reasonably reach, it will serve you and your team and your organization so much better than kind of just blue sky solutioneering experiments for everybody else.

[B]

Solutioneering is a new word that I like a lot. It feels like a combination of mountaineering and making solutions. Solutioneering, folks, you heard it here. Okay, my last idea that's not really an idea at all, but is like the warning label on experimentation is if you're doing it right and you're having the moments of reflection and you're having the cadence of check ins and retrospection and you're really looking clear eyed at the data to see if this thing has legs, you're going to have a lot of failure. And also what is going to come with that is probably disappointment. And I'm saying this as a PSA because most of us, especially in professional contexts, try to avoid disappointment at all costs. Maybe even more than like failure, we try to avoid disappointment. And so prepare to kind of hate this because no experiment gets launched because everybody thinks it's a stupid idea. Everybody launches an experiment with a twinkle in their eye and when it doesn't work out, it's a bummer and there's, you can't avoid that. And it's not the reason not to do it and it's not the reason to massage it into something that is inevitably going to work. Because, again, that's not an experiment. But prepare yourself and your team that's launching it for the reality that you are likely going to have to kill your darlings and you are likely at some time going to be confronted with your past self, who had a lot of belief and optimism around this thing and is now like, I gotta take it out back and shoot it. It sucks. Like, that part is really hard. And so have the kind of systems, conversations, support, whatever around that allows that to happen. Because if you're avoiding that at all costs, you're never going to do anything interesting. Totally.

[A]

Maybe the glib thing that I was going to say is like, well. And that's why the true measure of an experiment is not whether or not it worked, but whether we learned from it. Which, yes, yes, is true. We love to say that also is hard. We do love to say it.

[B]

I love to say it, too.

[A]

It's one of those things where, like, you can say the words and agree with the sentiment and when the thing that you conceptualized and thought was going to be great ends up not being that. And yes, we learned from it. Yeah, of course. And we'll capture those lessons and we'll share them. Like, it still sucks. And I think over time, you can start to divorce your identity from, like, does this experiment work or not? And really focus on, like, what is it teaching us and what are we going to do next? But you have to be in a particularly, I think, receptive environment for that to actually be cultivated and grow over time, which is not always the case.

[B]

Yeah, that's right. All right, Sam. I think that we've given them what we can on experimentation. Now they just have to go do stuff.

[A]

We've done our job. It's out of our hands now.

[B]

We deserve. Love it. It.

[A]

All right, so we are always looking for new topics for the show. So if you have an organizational pattern that you're having trouble changing, or for this episode in particular, if you have an experiment that you ran or participated in in your organization that either went really well or really poorly, you just want to tell us about it. I would love to hear those as well. And you can send them to podcast podcasttheready.com Nice.

[B]

This show is engineered by Taylor Marvin and produced by our vacationing friend Jack Van Amberg, who I kidnapped last weekend in Los Angeles. More on that later. At Work with the READY is created by the ready, where we help organizations around the world change the way they work. Thank you so much for listening.