Using Pokémon Go to Map the World

Summary

---

Podcast Summary: What's Your Problem?

Episode: Using Pokémon Go to Map the World
Host: Jacob Goldstein
Guest: Brian McClendon (CTO, Niantic Spatial)
Date: May 28, 2026

---

Overview

In this episode, Jacob Goldstein interviews Brian McClendon, the Chief Technology Officer at Niantic Spatial—a company seeking to advance mapping technology from conventional 2D maps to dynamic, ever-updating 3D maps. Spun out from the creators of Pokémon Go, Niantic Spatial leverages both game data and consumer contributions to create detailed maps that robots and humans can use to better understand and interact with the real world. The discussion covers the evolution of digital maps, challenges in 3D mapping, integrating AI with physical reality, and the broader applications for navigation, robotics, and augmented reality.

---

Key Discussion Points & Insights

1. The Evolution of Digital Mapping

• Early Digital Maps & Google Earth:
  • In 2004, digital map data was already in use, but it was largely inaccurate and outdated. Google’s Street View project began as an ambitious undertaking, placing cameras on cars to systematically film the world, starting with cities and then expanding globally ([04:44], [05:36]).
  • Quote:
    “A picture is worth a thousand words and probably megabytes of actual useful data. And there hadn’t been any systematic sort of extraction of the value from those pictures because that’s not how maps had been made in the past.” — Brian McClendon ([06:41])
• Street View’s True Purpose:
  • Beyond providing cool images, Google used Street View to create more accurate foundational map data ([08:11]).
  • Apple Maps’ Initial Struggles:
    • Apple started with purchased map data—lacking ground-truth field data, making it less accurate at launch ([08:22]-[08:47]).

2. Lessons from Google & Real-World Use Cases

• Tech Giants Pioneering Mapping:
  • Google and Apple poured vast resources into mapping, not originally part of their core missions but strategic due to smartphone adoption and user demand ([09:00]-[09:28]).
• Unexpected Impact:
  • After Hurricane Katrina (August 2005), helicopter pilots used Google Earth for rescue operations, navigating via satellite photos rather than now-submerged streets ([10:48]-[11:42]).
  • Quote:
    "There were literally hundreds of rescues done using Google Earth by pilots a few months after we’d released it. And that was a big surprise for us." — Brian McClendon ([11:38])

3. Pokémon Go’s Contribution to Mapping

• Data from Players:
  • Pokémon Go players initially took AR photos privately, but the company later incentivized users to record short videos of pokestops, consciously contributing mapping data ([12:15]-[13:06]).
  • Purpose:
    • These contributions enable more sophisticated AR experiences, where in-game elements appear in consistent, precise locations for all users ([13:11]-[13:59]).
    • Quote:
      “We wanted to create more AR gaming experiences...what we wanted to do was to be able to augment the pokestops in the outdoors. And so to do that accurately, you actually need to have a very accurate map of those pokestops, not just where they are with a blue dot, but actually the 3D model of the benches around it and the statue itself.” — Brian McClendon ([13:11])

4. From Gaming to Robotics and Enterprise Mapping

• Transition to Broader Applications:
  • Niantic Spatial began encouraging application developers to scan any real-world location, enabling business and robotic uses beyond gaming ([14:44]-[15:27]).
• Current Capabilities:
  • Using consumer phones and drones, Niantic Spatial reconstructs entire city-scale 3D maps. San Francisco has already been mapped extensively with lightweight drones and photogrammetry ([18:26]-[19:14]).
  • Combining Data Sources:
    • Merging ground-level phone and 360 camera data with drone and satellite imagery to build robust, real-time updating maps ([19:43]-[20:21]).
  • Robot Localization:
    • The goal: enable robots (e.g., delivery bots) to instantly localize themselves in a city, even after a ‘blindfold’ reset ([20:21]-[20:32]).
    • Quote:
      “That’s absolutely the goal, yes.” — Brian McClendon ([20:32])

5. Technical Challenges & AI Solutions

• Nature as a Technical Obstacle:
  • Trees are a major challenge due to their fractal detail and variability—“frankly, they are my nemesis” ([20:38]-[21:13]).
  • Solution:
    • Training AI to ignore unstable visual data (like leaves in motion), focusing on consistent background features ([21:38]).
• Robotic Navigation Issues:
  • Delivery robots can lose localization due to reboots, loss of visual input, or GPS inaccuracy—especially problematic in urban canyons with tall buildings where GPS errors are significant ([22:16]-[23:20]).

6. Industry Landscape & Custom Mapping

• Not Just Google:
  • Other companies try to build “digital twins,” but Niantic Spatial focuses on high-resolution, customizable mapping—enabling specific environments (like hospitals) to be digitally recreated for navigation and robotic use ([23:46]-[24:32]).
    • Hospitals Example: Corridors and rooms change often, challenging both humans and robots in wayfinding.

7. Scanniverse & User-Generated 3D Maps

• Scanniverse App:
  • Users can scan objects or areas with their phones, creating detailed, photorealistic 3D models easily shared or uploaded ([25:23]-[26:06]).
  • Enterprise Use:
    • Scanniverse now supports large-scale scans (factories, construction sites) for digital twins and robotic training ([26:29]-[27:06]).
  • Quote:
    “If they want to recreate their factory...they can do that with our enterprise version and then allow for modeling and allow for robotic training within their site using that tool.” — Brian McClendon ([26:29])

8. World Models: AI’s Next Mapping Challenge

• Beyond Language Models:
  • “World models” aim for AI to understand and predict the physical world using video and 3D data ([27:43]-[28:22]).
  • Connecting language AIs with 3D semantic models will allow them to reason about the real world as fluently as they now process language ([29:24]-[29:52]).
  • Fundamental Innovation:
    • “Trying to do a fully language embedded 3D semantic map of the world so that language models and robots and others can understand it is what we’re working on.” — Brian McClendon ([30:37])

9. Risks, Challenges, and Potential Impact

• Pitfalls:
  • The difficulty of mapping every meter of the world in 3D at high fidelity and low cost—old methods would have cost billions or trillions ([31:17]-[31:53]).
• Exciting Outcome:
  • AI combined with comprehensive 3D maps could yield profound insights about infrastructure, city dynamics, and enable smarter, more efficient urban planning and robotics ([32:12]-[33:43]).
  • Quote:
    “If you can give it enough information about the world, it can start to answer really interesting questions about the city, about how the city works, about which intersections are too busy, and about flow of people and things that could make the city more efficient because you have a more complete vision of it, especially in three dimensions.” — Brian McClendon ([32:37])

---

Notable Quotes & Memorable Moments

• On Ambition:
  "If he wants to do it, we'll give it a shot...the support that Google had for organizing the world's information was incredibly strong."
  — Brian McClendon ([06:07])

• On Tree Mapping:
  “Trees are my nemesis in my career.”
  — Brian McClendon ([20:38])

• On Privacy:
  "The inside of people's homes. You have to have a private space."
  — Brian McClendon, when asked what should not be mapped ([39:59])

• On Science Fiction’s Influence:
  “Neal Stephenson wrote Snow Crash and described a version of Google Earth before we built it. Both John and I had read that book and feel at least partially inspired by that.”
  — Brian McClendon ([42:26])

---

Timestamps for Key Segments

• The leap from 2D maps to Street View: [04:44]–[08:11]
• Apple Maps vs. Google’s data fidelity: [08:20]–[09:00]
• Hurricane Katrina’s unexpected hack: [10:31]–[11:42]
• Pokémon Go as a mapping goldmine: [12:15]–[14:44]
• Scaling up to city-wide 3D maps: [18:26]–[20:21]
• Technical hurdles—nature, trees, AI: [20:36]–[21:38]
• Robotic navigation—problems and promises: [22:05]–[23:20]
• Custom mapping for hospitals/businesses: [24:22]–[25:02]
• Scanniverse and the crowdsourced 3D world: [25:23]–[27:06]
• World models as the next leap in AI: [27:43]–[30:58]
• Lightning Round and McClendon's inspirations: [36:16]–[42:56]

---

Lightning Round Highlights

• Google Earth Easter Eggs:
  The center of Google Earth on Windows zooms into McClendon’s childhood bedroom ([36:16]).
• Gaming Tips:
  McClendon’s Pac-Man mastery, competitive scores, and the differences between Pac-Man and Ms. Pac-Man ([37:17]-[38:19]).
• Favorite Map:
  The data-rich map of Napoleon’s Russia campaign praised by Edward Tufte ([39:16]).
• Personal Cataloging Habit:
  Since 1993, McClendon has tracked every book, movie, and major event of his life in spreadsheets—now serving as a personal memory aid ([40:23]).
• Top Sci-fi Picks:
  Movie: Contact; Book: Three Body Problem ([41:36]).

---

Conclusion

This episode delivers a fascinating look at how gaming, user participation, and AI technologies are revolutionizing mapmaking. As Niantic Spatial expands its 3D mapping efforts from “Pokéstops” to entire cities, its work highlights how essential accurate spatial understanding is for the coming age of ubiquitous robotics, smarter urban infrastructure, and immersive augmented reality. The episode combines historical storytelling, technical insight, and a glimpse at how science fiction shapes real-world innovation.

---

For those interested in technology, mapping, robotics, or augmented reality, this episode compellingly illustrates how digital tools and crowdsourcing are converging to re-map—and reshape—the world around us.

Transcript

[Jacob Goldstein]

This is an iHeart podcast.

[Amazon Health AI Announcer]

Guaranteed Human Amazon Health AI presents painful

[Amazon Health AI Narrator]

thoughts why did I search the Internet for answers to my cold sore problem? Now I'm stuck down a rabbit hole filled with images of alarmingly graphic sores in various stages of ooze. I can clear my search history, but I can never unsee that.

[Amazon Health AI Announcer]

Don't go down the rabbit hole. Amazon Health AI gets you the right care fast. Healthcare just got less painful.

[Jacob Goldstein]

I'm not like a coffee connoisseur, but recently I tried coffee from a company called Perc and I loved it. And I'm not just saying that because this is an ad for Perc though. This is an ad for Perc. I really did think the coffee was delicious. The bag I'm drinking at the moment was grown in Peru. Perk sources coffee from all over the world. They have lots of different kinds of coffee to choose from and and they color code their bags. Blue bags are more mild coffee and pink bags are more wild coffee. That Peruvian coffee I'm drinking now is wild and if you're on the fence, I recommend trying wild. The other morning I had my first sip and I thought of that Will Ferrell line in Old School where he hits the beer bong and then he says once it hits your lips, it's so good. Find the coffee that matches your vibe and get 15% off your next order with promo code problem [e-mail] that's P E R C coffee.com promo code problem small businesses are the pulse of every community. They bring people together, create opportunities, and drive growth. Chase for Business helps business owners like you with personalized guidance and convenient digital tools all in one place. With that guidance and your determination, you can take your business farther and help build a brighter future for your community. Learn [e-mail] business chase for business make more of what's Yours the Chase Mobile app is available for select mobile devices. Message and data rates may apply. JPMorgan Chase Bank NA Member FDIC Copyright 2026 JP Morgan Chase Co. Pushkin Jacob I'm Jacob Goldstein and this is what's yous Problem? Two quick things before we start the show today. One, we're gonna take a little break of a few weeks after this week's episode, but we'll be back soon. And two, I wanna say thanks for listening. I really appreciate it. I wanna keep making a show that you like, so please feel free to email us Problemushkin FM. You can also find me on LinkedIn or on X formerly known as Twitter JacobGoldstein okay, that's the housekeeping. Now, here's today's show. My guest today is Brian McLendon. He's the chief technology officer at a company called Niantic Spatial. And his problem is how do you make maps better? Specifically, how do you take the two dimensional maps that we have today and turn them into three dimensional maps that change over time to keep up with changes in the real world? Maps like this would be very useful for robots. For one thing, Brian's company is in fact doing a pilot with a robot delivery company. Those little robots that drive down the sidewalk to bring you a pizza or whatever. But the maps could also be useful for people. For example, the company is working on a project with the US Coast Guard to build better flight simulators for helicopter pilots. And the kind of maps that the company is building could also be useful for augmented reality. And in fact, Niantic Spatial spun out of the company that made Pokemon Go, the augmented reality game. Brian's been working on maps for a long time. In the early 2000s, he worked at a mapping startup that got acquired by Google in 2004. And Google turned the project he was working on into Google Earth. And after that, Brian kept working on mapping at Google until 2015. In our conversation, Brian and I discussed the problems he's working on now trying to build better maps, including why trees are his nemesis. Also, we talked about what it will take to teach AI to understand the physical world. But to start, Brian told me the biggest lesson that he learned about mapping during his time at Google.

[Brian McClendon]

The most interesting thing is that map data in 2004 was already digital. We had suppliers that you could buy map data from, and MapQuest was out there showing you a little map on the screen, on your website, on their website. But what we discovered when we started building Google Maps, we used that same data and we showed it to our users and launched in 2005, we came up with this idea of Street View, where we'd go take pictures of a lot of places around a city and publish them.

[Jacob Goldstein]

Can I say, by the way, because I remember when that happened and when I heard of that and when I first heard of it, I was like, surely they're not going to drive cars with cameras on every street and take pictures. Surely that's too much. But no, that's what it was like. So wildly ambitious, just on a logistical level.

[Brian McClendon]

Absolutely. It was a crazy project. You know, Larry Page gets credit for coming up with the idea and then following through and supporting us as we built it.

[Jacob Goldstein]

What did you say when someone. Was it Larry Page who said to you, hey, let's do this. Who told you about it?

[Brian McClendon]

You know, Larry had actually taken a video camera and driven around the Stanford campus, you know, sort of systematically and said, you know, why can't we do this for the whole world?

[Jacob Goldstein]

What did you say? When he said, why can't we do this for the whole world? What did you say?

[Brian McClendon]

I said, he's Larry Page and it's his money man. If he wants to do it, we'll give it a shot. And honestly, that's, you know, the support that Google had for organizing the world's information was incredibly strong. I mean, Larry was serious about this. And we initially built a very expensive data collection van that drove around a few cities. Then we started to drive other cars that were cheaper, but still okay. And then that's when we launched the service. And then we eventually said, we need to do this for the entire United States. And the reason we did this was that when we launched the first cities, the first thing that people said was they looked at the pictures on the screen and then they looked at their map data, and they started complaining about the map data. How can you get this wrong? I have a picture right here that shows a street sign and a street number, and you have it completely in a different location. So a picture is worth a thousand words and probably megabytes of actual useful data. And there hadn't been any systematic sort of extraction of the value from those pictures because that's not how maps had been made in the past. And so the big insight we ever had was that map data to that moment was not good.

[Jacob Goldstein]

And by taking pictures for the first time, people could, in a scaled way, look at a photo right next to right on top of the map data and see, like, oh, the map is wrong. And this photo proves it. Like, it was the moment when people understood how wrong maps in general were.

[Brian McClendon]

Exactly. And people always kind of knew this because they always got frustrated every now and then when they'd use the maps and they would fail. But now we had this much larger complaint body, and we knew that the data that we'd been buying from third parties wasn't good. And so we decided to make our own maps. And. And the biggest basis for that was the Street View pictures themselves. And we drove all of the U.S. canada, Mexico in 2008, and by 2013, we had driven 50 countries.

[Jacob Goldstein]

So the point of Street View was not just for the, like, hey, cool, you can look at pictures. It's to get Better data as an input to the sort of more traditional maps.

[Brian McClendon]

Absolutely.

[Jacob Goldstein]

I never knew that. Is that why Apple Maps sucked when it first came out? Cause they didn't have that.

[Brian McClendon]

I'm so glad you said it. It keeps me from saying it.

[Jacob Goldstein]

I remember when it happened. They fired the guy. Remember they fired the guy. But is that why? Because they just were buying the normal map data and everybody was comparing it to Google and you guys had literally driven cars on every street in America to figure it out.

[Brian McClendon]

That's exactly right. And they eventually got there. Apple has driven many millions of miles themselves. But if you look at worldwide, I would say that Google is still probably better.

[Jacob Goldstein]

I don't have a dog in that fight. It is interesting how both Google and Apple had these were just throwing off crazy amounts of money. Right. Each for their own special reasons. But we're reinvesting it in an interesting way that like ex ante I would not have guessed. Right. Like this sort of search giant and the fancy phone maker becoming kind of going to the vanguard of mapping is not obvious.

[Brian McClendon]

So I think when Steve Jobs added Google Maps to the iPhone very, very quickly, he realized it was strategic. And he knew that because there was a GPS on the phone and because he now had a pan and zoom capable touchscreen, that he had an amazing new method for helping people understand the world. And so getting that map data on there was very important. And Android and iOS both had it very quickly. But the moment you do that, you have the same problem. That Street View showed was that people walk around, they look at their map and they look up at the street around them and it's wrong. And so they knew that, you know, the biggest complaint they would be getting was this factual failure from their users about their maps.

[Jacob Goldstein]

Yeah, well, a map that's wrong is worse than no map. Arguably, yes. I mean, were there surprises to you about sort of what developed, what didn't develop, how people used Google Maps? Things you thought might happen that didn't happen?

[Brian McClendon]

I think one of the things that we probably speculated about, but if you realize that Google Earth and Maps launched in February And June of 2005, in August, Katrina happened.

[Jacob Goldstein]

And Katrina was Hurricane Katrina in New Orleans.

[Brian McClendon]

Yeah, yeah, it was an incredible hurricane. Did huge amounts of damage and the city of New Orleans was flooded completely. And this turned into a challenge for everybody involved, both for just escaping the city and going to the right places. But then it became about rescuing people on rooftops. And the problem now was the streets were gone and they couldn't figure out where Things were. And so it turned out that the helicopter pilots at the time used Google Earth to find out where the address was on the picture and then use the picture to navigate their helicopter to find the people waiting on the roof. And there were literally hundreds of rescues done using Google Earth by pilots a few months after we'd released it. And that was a big surprise for us.

[Jacob Goldstein]

It is extremely interesting to me that the company right now spun out of the company that made Pokemon Go and that there was this kind of kernel of an idea which was like, oh, hey, people playing Pokemon Go have taken a billion photos in the world. Like, truly a billion photos. This is an interesting data set. Maybe we can do something with it. I mean, if I understand right, that was before you got there. But like, how did that play out? Did they know when they were launching Pokemon Go that like, oh, we're going to get this data? Did somebody realize it?

[Brian McClendon]

So, interestingly, I think John Hanke definitely had a very long term vision. He was thinking about not just the game itself, but about games in the real world. Now, the photos that were taken initially by Pokemon Go players were taking a picture of a Pokemon at a location. Those photos stayed private. What happened was that John, seeing the future, said, can we create a game mode where players consciously go out and record videos for us of pokestops and help us map these locations by recording videos? So it was a conscious gameplay activity where you would get in game rewards if you would upload a 30 second video to the game. And that was where that data came from. So it was consciously contributed by the players rather than sort of background collected.

[Jacob Goldstein]

And what was the point of that from the point of view of the company?

[Brian McClendon]

So we wanted to create more AR gaming experiences. And Pokemon Go already used augmented reality to create those photos that you were talking about. Put a Pokemon sitting on a chair and take a picture of it. So it was already using augmented reality. What we wanted to do was to be able to augment the pokestops in the outdoors. And so to do that accurately, you actually need to have a very accurate map of those pokestops, not just where they are with a blue dot, but actually the 3D model of the benches around it and the statue itself. And so creating that 3D model and then being able to point your phone at it and say, I know exactly where you're standing relative to that statue. Now, the Pokemon that was placed by the last person is exactly the same place than it is for you.

[Jacob Goldstein]

So you need a really sophisticated model of the physical world. For that to work, otherwise it'll be like floating in space a foot from where it's supposed to be.

[Brian McClendon]

That is exactly the problem we spent several years solving. And we only launched the product that used it, I think, in 2023. And that was part of Pokemon Go, and it's called Pokemon Go Playgrounds. And it's the ability for players to leave Pokemon at locations in interesting formations and interesting combinations and then have other players discover them or add to them.

[Jacob Goldstein]

And at what point did somebody think, oh, we can do something wholly other than gaming here? We can build a new kind of map that is more legible to a robot.

[Brian McClendon]

I think that in some people's minds that was the thought from day one. I think John would argue in his mind he was, he was really thinking in that direction. But I think from a product perspective and a business perspective, being able to allow developers of applications to go scan their own places, not pokestops, but any location around the world, and then create these experiences, that was our first step. I think the robotics aspect came into play more recently, probably in the last 18 to 24 months, as we realize that robots struggle with finding themselves in the real world. And so they need a form of maps.

[Jacob Goldstein]

Don't we all? We'll be back in just a minute.

[Amazon Health AI Announcer]

Amazon Health AI presents painful thoughts.

[Amazon Health AI Narrator]

Why did I search the Internet for answers to my cold sore problem? Now I'm stuck down a rabbit hole filled with images of alarmingly graphic source in various stages of ooze. I can clear my search history, but I can never unsee that.

[Amazon Health AI Announcer]

Don't go down the rabbit hole. Amazon Health AI gets you the right care fast. Healthcare just got less painful.

[Jacob Goldstein]

I'm not like a coffee connoisseur, but recently I tried coffee from a company called Percy and I loved it. And I'm not just saying that because this is an ad for Perc, though. This is an ad for perc. I really did think the coffee was delicious. The bag I'm drinking at the moment was grown in Peru. PERC sources coffee from all over the world. They have lots of different kinds of coffee to choose from and they color code their bags. Blue bags are more mild coffee and pink bags are more wild coffee. That Peruvian coffee I'm drinking now is wild. And if you're on the fence, I recommend trying wild. The other morning I had my first sip and I thought of that Will Ferrell line in old school where he hits the beer bong and then he says, once it hits your lips, it's so good. Find the coffee that matches your vibe and get 15% off your next order with promo code [e-mail] that's P E R C coffee.com promo code problem running a small business takes everything you've got. But with Chase for Business, you're not alone. They bring together local support and a broad range of resources to more than 7 million customers with a deep understanding of your day to day needs. They provide products and guidance built to help you thrive right now. Earn $500 when you open a new Chase Business Complete Checking Account for New Business Checking customers with qualifying Activities Offer Expires June 18, 2026 Chase Business Complete Checking has the flexible tools you need to accept payments, make deposits and manage your finances with confidence. Learn [e-mail] PodcastBizOffer Chase make more of what's yours Fees may apply to Chase Business Complete checking accounts. The $500 offer is available for new business checking accounts with qualifying activities through June 18, 2026. Eligibility and qualification requirements must be met. Additional restrictions may apply. Please speak with a business banker for more information. JPMorgan Chase Bank NA member FDIC so let's talk about where you are now, like what's in the world now that you're making.

[Brian McClendon]

I think the experience of working with the data from the phones of the players taught us a lot about how to reconstruct and create these precise maps not with super fancy survey tools but with just consumer devices. And what we got good at was using low quality data, which I mean it's still very good but it's not professional grade anything and being able to turn that into professional grade data. And so I think that was the skill that we're bringing into the enterprise applications that you hear now. And we can do it not just at the scale of a statue or a single location, but we can do it at city scale. We can collect an entire city and help you find yourself.

[Jacob Goldstein]

Have you collected an entire city?

[Brian McClendon]

Yes.

[Jacob Goldstein]

What city or cities?

[Brian McClendon]

We have all of San Francisco in pieces of other cities and they're usually collected from drone data. Basically commissioning and saying we wanted a drone based camera. Follow a path, take a picture every several meters and take it from multiple angles and then use photogrammetry to reconstruct that city at a level of detail beyond what Google can do it.

[Jacob Goldstein]

And when you say drone you mean like a flying drone? Like a little quadcopter or whatever?

[Brian McClendon]

Exactly. A very lightweight, very limited, under 250 grams. So the smallest ones.

[Jacob Goldstein]

And so what are you doing now with your 3D robot legible map of San Francisco.

[Brian McClendon]

We are combining all of that interesting data collected on the ground with phones and with 360 cameras, with this drone data, and also with satellite data and proving out that we can help robots and humans find themselves, localize themselves anywhere in the city without having to pre map it specifically at that location.

[Jacob Goldstein]

So you mean like a robot? It's just like you blindfold the robot, for lack of a better word, and then take the blindfold off and it knows right away where it is and which way it's looking.

[Brian McClendon]

That's absolutely the goal, yes.

[Jacob Goldstein]

What do you have to do to get there?

[Brian McClendon]

I think the challenge is that the world is a complicated and varied place, and trees are my nemesis in my career. And they are absolutely.

[Jacob Goldstein]

I'm pro tree. For the record, I'm pro tree.

[Brian McClendon]

I understand they are visually amazing, but they are, in fact, fractally based. Right. They have very high detail. And worse than that, their leaves, they flutter in the wind. It's deeply inconvenient. And so when you take multiple pictures of a tree, no two pictures have the tree in the same spot when

[Jacob Goldstein]

they change rapidly with time.

[Brian McClendon]

Absolutely. So another problem, the old way of doing this was that satellites and airplanes would take pictures of cities in the winter because the leaves were off and that gave them more information. But if you're trying to build a model of the city and localize, you actually need to see them in all their states, not just the leaf off state.

[Jacob Goldstein]

So is the short answer to how you solve that problem AI? How do you solve the tree problem?

[Brian McClendon]

You try to look beyond the trees. Trees other than the tree trunk. There's not a lot of stability there. But there are many things behind the tree or next to the tree but that are useful. And being able to effectively train an AI to ignore the tree is usually your best bet, because there's enough remaining features in view that you can find enough to connect yourself to that location.

[Jacob Goldstein]

I know you just made a deal with a delivery robot company. Is that the first use case? What do you think the first use cases are going to be?

[Brian McClendon]

Definitely is the first use case. They struggle with, effectively, the blindfold problem, that sometimes they don't have enough context to know where they are. And they need to rediscover their location. And they need to do so accurately enough that either the autonomous controller or some human who drops in to look at where it is can know where they are in the city, because city blocks tend to look the same. Unless you have that information, how does

[Jacob Goldstein]

the robot get lost in the first place. Like, how does the robot find itself in that situation?

[Brian McClendon]

There's two ways. One is, I mean, they hit a reboot and they literally lose connection if they are covered, if for any reason enough of the world is blocked from them for long enough, they lose control. GPS by itself is rarely good enough because especially in cities, the GPS has reflections off of buildings, and you can be a city block away, and the GPS will happily tell you that you're there.

[Jacob Goldstein]

So it's insufficiently precise, particularly in cities. Yes.

[Brian McClendon]

And big cities with big, tall buildings are absolutely the hardest challenge, other than tunnels and parking garages.

[Jacob Goldstein]

But that's good for you. That weakness is your opportunity.

[Brian McClendon]

Exactly.

[Jacob Goldstein]

Tell me about the broader. I don't know if industry is the right word, but the broader context. Who else is working on things like you're working on?

[Brian McClendon]

I mean, I think we've had companies that have worked on building digital twins of the world, either in the micro or the macro, since before Google. And Google certainly is the biggest wholesale attempt at this. But there's a need for more resolution and also to have it from suppliers other than Google. Because one of the areas that I think is hard is that Google's all about recreating the public world, but they don't really have any way for a company to do their own location and add it to the real world.

[Jacob Goldstein]

Oh, interesting hospital. Did I read hospital as an example? We were talking about that when we were doing the prep of, like, how hard it is to find your way around a hospital.

[Brian McClendon]

It is, and I just experienced that yesterday at Stanford. I think the challenge is building a map. And with the hospital, there's actually a lot of change that happens in the corridors of a hospital at any given time. The big corridors are stable, but most of the actual patient areas are constantly changing their visual appearance. And so getting a. Getting robots that are able to navigate that space accurately and reliably is certainly one of our potential use cases.

[Jacob Goldstein]

So there's an app called Scanniverse. That's you, right? Yes, the app Scanniverse. So I downloaded that this week, and astonishingly, there are. Astonishingly, to me, within 100 yards of my house, there are two things. Two objects that have been scanned in 3D and uploaded to this app.

[Brian McClendon]

Yes.

[Jacob Goldstein]

Which is interesting. Cool. Like, I had no idea what's going on there. What is it? And why are people on my block taking pictures of things?

[Brian McClendon]

Scanniverse is a great app, and we actually acquired the one person who wrote it. Brilliant, brilliant programmer. And turned it into this product that allows capture of 3D objects. And what you're seeing there is that a lot of users love to be able to go out and scan their area, their location or a statue or whatever, and recreate it in 3D visually, almost completely photorealistically, and then share that with the rest of the world. And so you're experiencing sort of the map that Scanniverse puts up at the front of that.

[Jacob Goldstein]

So it's in the same way that people like to do whatever geocaching and geo guessing and whatever. It's some version of kind of sharing the real world in a digital way.

[Brian McClendon]

Exactly.

[Jacob Goldstein]

Is it big? I infer from the fact that there are two on my block or one on my block and one around the corner that lots of people are using it. Is that true? Do I live in a weirdly dense area?

[Brian McClendon]

There are a lot of people using. We have hundreds of thousands of users, and they are passionate about this ability. And we have used. We've recently, very recently added the enterprise capabilities to that app. People want to create a digital twin of a tiny little place. They can do it on their phone and upload it like you saw. If they want to recreate their factory or their business hundreds of square meters, they can do that with our enterprise version and then allow for modeling and allow for robotic training within their site using that tool.

[Jacob Goldstein]

So you can use your phone and walk around your factory, say your little small factory or your construction site or whatever, and the 3D rendering it generates is robot legible. Then you can train your robot on the thing you made with your iPhone.

[Brian McClendon]

That's correct.

[Jacob Goldstein]

So there's this phrase you hear in AI world model, and it's sort of a complement to a language model. Right. Instead of AI that's trained on language, it's AI in some way that's trained on the physical world. Tell me about that and how your work fits in with that.

[Brian McClendon]

So the world models have been mostly focused to date on solving a sort of very specific problem, which is how to translate videos, which they're mostly programmed with, into understanding of the world that's good enough that it can predict the future.

[Jacob Goldstein]

Yeah. And when you say predict the future, you don't mean any, like, weird oracle thing, right? You mean like, predict the future as well as a human can that. Like, if fork starts to fall off the table, it's going to fall all the way to the ground.

[Brian McClendon]

That's exactly right. So if something is initiated, how would it play out with the objects? And if you've seen enough forks fall in videos. And if you've seen enough objects fall, then you can start to predict it.

[Jacob Goldstein]

And we were so used to, now AI being so good at language, it's easy to forget that it has no understanding of the physical world. I mean, except in language. Right.

[Brian McClendon]

It's a weird. Yeah, it can think in its brain. I know what the rules of Newton's law are. But turning that into pixels on the screen is actually pretty hard. But if you train enough video models on it, then it can figure it out. And so I think the connecting the language model to the physical world is in some ways there's two ways to do it. One is to have the model understand the physics video. The other is to turn the world itself into words or into meaning. And this is, you know, typically called semantics. And there's all sorts of, you know, technologies that try to label information about a picture. And if you train enough of models on the labeling of the pictures, you can then label any, but any picture of the real world.

[Jacob Goldstein]

This was kind of the, the first big breakthrough of neural networks, right? I mean, it sounds like that.

[Brian McClendon]

Cats. Cats and dogs. Cats and dogs, yeah.

[Jacob Goldstein]

And imagenet, right? Yeah.

[Brian McClendon]

So if you give the information about the embeddings it's called, and you do that not just in cat and dog dimension, but you do it in hundreds of dimensions of objects.

[Jacob Goldstein]

Right.

[Brian McClendon]

You can give these language models the equivalent of a language understanding of the real world and then they can do amazing things with it.

[Jacob Goldstein]

Huh. And so how does all that relate to the work you're doing?

[Brian McClendon]

We believe that to do that successfully, you need to start with 3D data. Because the ability to label something in three dimensions is much more accurate and argue with much more information than if all you have is a 2D picture of it. Because in 2D pictures there's always obscuration of data and things are behind things and you can't see the whole picture. But in 3D you eventually see everything that is seeable.

[Jacob Goldstein]

I mean, is that the fun not to be reductive, but kind of to be reductive is that the fundamental thing you are doing that is novel is making a three dimensional map in a world where maps have always been two dimensions before?

[Brian McClendon]

I would say yes, there are 3D visual maps, there are rare one off 3D maps, but trying to do a fully language embedded 3D semantic map of the world so that language models and robots and others can understand it is what we're working on.

[Jacob Goldstein]

So if we zoom out like, well, two things, I guess there's two things we'll do. The sad one and the happy one. Like what might go wrong for you? Like what has to go. Yeah, what might go wrong for you over the next years? It seems like you're at this moment when you have this interesting technology. You're sort of proving it out. It's not widely used yet. What are the sort of pitfalls?

[Brian McClendon]

I mean, it was hard mapping the world in 2D for humans. Mapping the world in 3D for robots requires a level of detail and precision that nobody's ever done before. And the only way to get it done is to apply AI to the problem, to figure out how to make it more accurate more quickly and more cheaply than has been done in the past. Because carefully reconstructing the world is every square meter of it. The old way is billions, if not trillion dollars. And this can't be that. Nobody's going to pay that.

[Jacob Goldstein]

Yeah, billions maybe, but trillions definitely not. And so if you get it right, give me the big exciting outcome, you know what I mean? If it goes well, what's the world look like? How is the world better?

[Brian McClendon]

I think that as we're seeing, the AIs have been able to dig into human knowledge and ingest all of the text of all of our scientific papers and all of our blog posts and all of Instagram. And they've synthesized this, in many cases, into some very interesting information that answers at least personally relevant questions today. And if you talk to some mathematicians, they're doing pretty well in math as well. But the questions about the real world are simply not there yet. But the advantage of an AI is it doesn't stop and it doesn't stumble on too much data. And if you can give it enough information about the world, it can start to answer really interesting questions about the city, about how the city works, about which intersections are too busy, and about flow of people and things that could make the city more efficient because you have a more complete vision of it, especially in three dimensions. The 2D maps for a street are fine, but the moment you go down to the subways in New York, some of those are three, four levels deep and some of them are extremely popular, and others are ghost towns. You know, I think there's opportunities to improve the infrastructure that we already have built without having to like rebuild it from scratch.

[Jacob Goldstein]

We'll be back in a minute with the lightning round.

[Amazon Health AI Announcer]

Amazon Health AI presents painful thoughts.

[Amazon Health AI Narrator]

Why did I search the Internet for answers to my cold sore problem? Now I'm stuck down a rabbit hole filled with images of alarmingly graphic source in various stages of ooze. I can clear my search history, but I can never unsee that.

[Amazon Health AI Announcer]

Don't go down the rabbit hole. Amazon Health AI gets you the right care fast. Healthcare just got less painful.

[Jacob Goldstein]

I'm not like a coffee connoisseur, but recently I tried coffee from a company called Perc and I loved it. And I'm not just saying that because this is an ad for Perc though. This is an ad for Perc. I really did think the coffee was delicious. Perk has lots of different coffees to choose from and they color code their bags. The blue bags are mild coffee, the pink bags are wild coffee. You can find the coffee that matches your vibe and get 15% off your next order with promo code [e-mail] that's P E R C coffee.com promo code problem Being a small business owner isn't just a career, it's a calling. Chase for Business knows how much heart and effort go into building something of your own. That's why they make your business growth their priority. The team at Chase takes the time to understand your mission, where you are now, and where you want to go. Their broad range of solutions is designed with you in mind so you can bring your ideas to life. From banking to payment acceptance to credit cards, you can conveniently manage all your business finances all in one place with their digital tools looking for tips and advice, their online resources are always available to give you the solutions you need to help your business thrive. See how your business can get stronger and go farther with Chase for Business. Learn [e-mail] business chase for business Make More of what's Yours the Chase Mobile app is available for select mobile devices. Message and data rates may apply JPMorgan Chase Bank NA Member FDIC Copyright 2026 JPMorgan Chase Co. We're going to finish with a lightning round. Tell me about the Meadowbrook Apartments in Lawrence, Kansas.

[Brian McClendon]

I grew up in Meadowbrook. It was a fine apartment complex. I moved there when I was four and moved out when I was 18. So the center of Google Earth on the Windows PC in particular will zoom you into the bedroom of the apartment of my apartment building.

[Jacob Goldstein]

Still?

[Brian McClendon]

Still?

[Jacob Goldstein]

Yeah. That's a robust Easter egg.

[Brian McClendon]

There's another one in that category, which is that a friend of mine who I went to University of Kansas with, I hired him in 2005 to Port Google Earth to the Macintosh, and he secretly changed the center of Google Earth on the Mac to be Chanute, Kansas, where He's from. But he did it to the main intersection there in Chanute. And the city of Chanute loved it so much, they actually built a mural on the street at that location. So if you're ever on a Mac, give it a try and you'll see what it says.

[Jacob Goldstein]

What's one tip for becoming a Pac man champion?

[Brian McClendon]

The old, old machines had patterns. And when I was the very first Pac man machine, if you knew the five key pattern and Abu, I was able to break a million. They're really professional people. I think the high score, the infinite Perfect score is 3.3 million, but I was able to get a million. But these days I still play Ms. Pac man at home. I have a home machine, and it's really about reaction. And if you play it on fast mode, it's all about speed.

[Jacob Goldstein]

Do I recall that Ms. Pac man was more deterministic than Pac Man? Is that right?

[Brian McClendon]

It's the other way around. When they launched Ms. Pac man, they surprised people because they had extra logic and they actually randomized it. So literally no pattern really works, even from the beginning.

[Jacob Goldstein]

Oh, okay. So it's a kind of more elegant game, maybe at some level, or you could play it for longer. Anyways. How are your skills?

[Brian McClendon]

My high score is 279,000. My friend's high score is 275,000. And that's where we're at.

[Jacob Goldstein]

Are you currently training to win any other video game championships?

[Brian McClendon]

No, my only games. I played a lot of video games in the early 80s, and so Missile Command, Ms. Pac Man, Pac man and Galaga, I think were my main games.

[Jacob Goldstein]

I liked the big ball on Missile Command, the arcade Missile Command. Remember, it had that giant ball, almost

[Brian McClendon]

a bowling ball, and it had momentum. But the only games I really play now are the New York Times word games, and I still play Pokemon Go.

[Jacob Goldstein]

You have any favorite maps from history or whatever? They're maps you love or like genius cartographers.

[Brian McClendon]

There's one map that's more of a sort of a data science presentation. There was a Tufti was a guy who wrote some really great books about data visualization. And he highlighted this map, which has always stuck with me. And it was a single sheet map that showed Napoleon's march to Russia, including the size of the army as it progressed. And then it decimated itself in Russia and then shrunk and shrunk and shrunk as it returned because they were going through winter with no food. And just the amount of information in that map beyond the map itself is Incredible.

[Jacob Goldstein]

Anything you think shouldn't be mapped.

[Brian McClendon]

The inside of people's homes. You have to have a private space.

[Jacob Goldstein]

What was the last time you got lost?

[Brian McClendon]

That's a good question. Never for very long, you know, I don't remember that. Blue Dot's pretty powerful.

[Jacob Goldstein]

Tell me about cataloging your life.

[Brian McClendon]

So I started doing it in 1993. I started recording books I read movies I watched and various other events in my life. And I actually back propagated and re remembered movies I had seen before 1993. So I have a list of every book, every movie, as well as many other life events, all in a spreadsheet. And it's. As I grow older, it becomes my memory because it's not super detailed, but in many cases just the event itself existing reminds me of what I was doing that year, that month, that day.

[Jacob Goldstein]

Do you look back on it the way people might look at old photos or something? How do you use it or engage with it?

[Brian McClendon]

I do. I looked back at what movies was I watching in the last semester before I graduated high school? You know, how did I spend my time that year and what was available and what's going on?

[Jacob Goldstein]

And do you have qualitative notes? Do you have ranking? Like, are there interesting columns in the spreadsheet?

[Brian McClendon]

I've gone back through and I have my top 44. Let's see, what do I have my top 40 for books and I have my top 200 for movies and I have them somewhat ranked within that.

[Jacob Goldstein]

Top movie, top book. Let's see.

[Brian McClendon]

I'll still say Contact for reasons. On the movie I just find it very compelling. And on the book I love Scalzi and Hugh Howey. So I almost want to say silo, but I'll still say three Body problem.

[Jacob Goldstein]

So both science fiction.

[Brian McClendon]

Oh, I am unabashedly, almost entirely science fiction fiction.

[Jacob Goldstein]

It's really. I guess it shouldn't be surprising, but it's persistently notable to me how influential science fiction is in the world, the world we live in. People built it in some significant degree because they were inspired by science fiction in really specific ways.

[Brian McClendon]

Absolutely. Carl Sagan wrote Contact and it was hugely influential that way. But Neal Stephenson wrote Snow Crash and described a version of Google Earth before we built it. And both John and I had read that book and feel at least partially inspired by that. There were other things that inspired us too, but that very specific description of a visualizable holographic 3D model of the planet that you could zoom into was very powerful.

[Jacob Goldstein]

Thank you so much for your time. It was a pleasure. Very interesting to talk with you.

[Brian McClendon]

Thank you.

[Jacob Goldstein]

Brian McClendon is the chief Technology Officer at Niantic Spatial. Our show today was produced by Gabriel Hunter Chang. It was edited by Lydia Jean Cott and engineered by Hans Dale. She. I'm Jacob Goldstein and we'll be back in a couple weeks with more episodes of what's yous Problem? Thanks for listening. I'm not like a coffee connoisseur, but recently I tried coffee from a company called Perc and I loved it. And I'm not just saying that because this is an ad for Perc, though. This is an ad for Perc. I really did think the coffee was delicious. The bag I'm drinking at the moment was grown in Peru. Perc sources coffee from all over the world. They have lots of different kinds of coffee to choose from and and they color code their bags. Blue bags are more mild coffee and pink bags are more wild coffee. That Peruvian coffee I'm drinking now is wild. And if you're on the fence, I recommend trying wild. The other morning I had my first sip and I thought of that Will Ferrell line in Old School where he hits the beer bong and then he says once it hits your lips, it's so good. Find the coffee that matches your vibe and get 15% off your next order with promo code [e-mail] that's P E R C coffee.com promo code problem.

[Inner Balance Oestra Advertiser]

If you're feeling off fatigue, mood changes, skin shifts, yet your labs say everything's normal, you're not alone. Meet Oestra from Inner Balance. The first all in one prescription strength bioidentical hormone cream that's natural and effective and only takes one drop, 10 seconds a day. Oestra replaces five to six products women typically use to treat symptoms and is third party tested to ensure the highest quality. Visit innerbalance.com today to start feeling like yourself again. That's innerbalance.com you ever wonder how far

[Electric For All Advertiser]

an EV can take you on one charge? Well, most people drive about 40 miles a day, which means you can do all daily stuff no problem. Go to work, grab the kids at school, get the the groceries and still have enough charge to visit your in laws in the next county. But they don't need to know that. And the best part, you won't have to buy gas at all. The way forward is electric. Explore EVS that fit your [e-mail]

[Jacob Goldstein]

this is an iHeart podcast. Guaranteed human.