← Blockspace: AI & Bitcoin
Blockspace: AI & Bitcoin
Inside TeraWulf's Lake Mariner Data Center Construction: "They're Working Around the Clock"
00:29:44
Loading summary
Transcript31 lines
- [00:00]
A
What's up, y'?
- [00:01]
B
All?
- [00:01]
A
Welcome back to the Blockspace podcast, brought to you by CleanSpark. Today we have a bonus episode for y' all that was a segment on our Thursday livestream with Rich Miller, the founder of Data Center Richness, where he covers AI data centers and the energy and data center landscape across the U.S. rich Miller is a veteran data center reporter with 25 years under his belt. So this was a fun conversation where we cover some of the current trends, what he's seen change over the years, and specifically in this episode, he gives us an inside look at Terra Wolf's Lake Mariner build out after coming off a tour of the facility recently this year. Hope you all enjoy.
- [00:42]
C
Rich, hey, I can hear you. Welcome to the show.
- [00:45]
B
Awesome. It is great to be here.
- [00:47]
A
Thanks for joining, man. Really, really appreciate it. Before we hop into it, Rich, just give people a quick background and what you do at Data Center Richness.
- [00:56]
B
Sure. I've been covering the data center sector full time now for about 25 years through a series of publications that I launched. Data Center Richness is the newest. We look at just about all aspects of the data center industry, particularly in terms of handling the scale that's going on and frankly, building better data centers, because that's a big topic now. But do podcast, YouTube and also on substack.
- [01:25]
C
We're basically in the same line of work now, Rich. It's amazing. We've been covering bitcoin miners really since it became an American industry, but they all pivoted to AI. And so my first question for you is you've been covering data centers for decades and it's absolutely the most important industry in the America economy right now. When did you identify this, like crypto to AI shift? When did it pop up on your radar and what did you, like, what did you think when you initially started watching this trend?
- [01:58]
B
So the interesting thing is I started paying close attention to bitcoin mining operations back in the early days of sort of Bitcoin enthusiasm. 2014, 2015, there was a lot of ideas that crypto miners would be data center. Customers pretty quickly figured out that that was not going to be the case because data centers are built for enterprises and cloud with a lot of redundancy, a lot of cost overhead to stay online all the time. Crypto doesn't need that. And we're always looking for the lowest cost approach and they wanted a lot of power, which would turn out to be important. So crypto built its own style of facilities and the data center sector plowed ahead with enterprise and cloud, the moment when everything changed was in 2022 when two things happened. The first was that in the state of Virginia, a little area in Northern Virginia called Data Center Alley in Ashburn ran out of electrical power. This was important because that is ground zero for the data center industry. It's where all the networks meet. It's where the most cloud and data center capacity was deployed. Because historically, the way that data centers have always chosen where they were going to locate is by following the network. They wanted to get close to the big Internet intersections in the major business markets and everybody build data centers right around the interconnection exchange. When they ran out of power in Northern Virginia, suddenly people couldn't deploy all the capacity they wanted. And that shifted to a new strategy, follow the power. So that was the first thing that happened. And of course, then in November 2022, ChatGPT appeared on the scene and we, you know, it became pretty clear that AI was going to become a really significant use case. And by, you know, mid 2023, it was also clear that they would, these customers would want very large chunks of capacity. And so what kind of happened is that the data center sector moved in the direction of what the crypto miners were doing in terms of suddenly wanting to follow the power, find places that didn't necessarily have to be in a core network market, but had lots of land and particularly access to a lot of power. And there were a lot of crypto mining companies that had that at that time that were thinking about the. At first I think they talked about as hpc, but then the AI shift. So it's really been fascinating to watch and I've learned about some of these companies watching some of you guys podcasts because there were a lot of new players suddenly appearing on the scene who very quickly have become consequential for AI deployments.
- [04:57]
A
Rich, quick question that, and for some context, I don't know what year this was, but when he's talking our listeners about Data Center Alley, Virginia Data centers account for anywhere from 25% of the entire state's electrical capacity at a given time. But going back to the need to chase where the power is, did anything change about the networking where they were now able to move further out the curve to middle America, places like Texas, Ohio, Pennsylvania. Did anything change about the actual networking of these data centers that allowed them to do that, or was it just purely necessity driven?
- [05:36]
B
It was purely necessity driven that the thing they needed first was the power, because you can always drag the network behind you and that's what a lot of the big guys, if you're Google, if you're aws, if you're Microsoft, you've got deep relationships with network providers. Most of them have dark fiber operations. So they can work pretty quickly to get fiber to a place where you wouldn't necessarily have the kind of connections that you have in Northern Virginia. That doesn't mean you'd have the same level of connectivity there, particularly in terms of the number of networks you can reach. But the other thing is, you know, some AI workloads need really low latency. Others seem to do okay with, you know, some of the reasoning models in particular they're going to take a little bit to answer your question. So the kind of latency that you would need for you know, 911 services or emergency services and things like that, or financial services, not the same.
- [06:39]
C
Rich, I'm curious about like the DNA of the different companies. Now you have traditional data center builders and traditional tech and hyperscalers and you've got the, I would say historically a bit more wild west bitcoin crypto miners. And I'm curious because it seems like both industries are going to have to come together to figure out how to talk to the other really. Well, from the perspective of a bitcoin crypto miner, is there something in the DNA of how those companies operate that you see that is different, that is an advantage in this new world of like rapid build out speed to power?
- [07:20]
B
I think there's a couple of things I'd highlight. One is that data center operators are very used to dealing with enterprise clients, large clouds. So they've always have focused for a long time on really kind of being pretty corporate and working towards strong credit ratings. And I think that's, that's an area that's been important for them and their clientele that's a little bit different as opposed to crypto in most cases. I think the thing that's been important is that there's all sorts of players now whose interests are aligned about bringing more AI capacity online. And some of those players I think have played a really important role in working with some of the crypto mining specialists who wanted to make that crypto to AI shift and had the power. Nvidia very clearly has worked closely with a couple of them to help them understand what the requirements are going to be and help them to kind of get up to speed. And Dell I think has played an important role in that as well. And I think when everybody's sort of interests are aligned around getting getting that capacity online. There's been some cooperation to help help some of these folks who are new to it move up the value chain a little bit more quickly. And I think, you know, the other thing is that there's a lot of differences between the facilities that I think there's been a gap to bridge there. I talked to Wes Cummings from Applied Digital on one of our podcasts, and he talked at length about this. There was a real learning curve to go from what they were doing with crypto and to deal with the kind of gear that Nvidia might be bringing in. But there are some corollaries, too. Lots of bitcoin folks are very comfortable with liquid cooling, have seen it before, probably to a higher degree than some of the people in the data center sector have, because it's been on the horizon for a long time. But it adds a whole maintenance and overhead thing that a lot of people have resisted. So it's really Nvidia that's shoving the entire sector into the liquid cooling sort of frame of mind,
- [09:45]
A
speaking to those challenges. Rich I was initially very skeptical in 2023 when these Bitcoin miners started talking about making this pivot, because a handful of these companies have, quite frankly, never really been very proficient in mining bitcoin. I mean, they're always mining it at a Part of the reason why they could sustain their operations was during bitcoin bull runs. They could sell their equity into the open market and then use that to cover their costs and expand much more rapidly than, let's say, a more financially prudent private miner would do. But as you noted in your articles, I think the biggest thing that they had an advantage here was that they were already sitting on the power agreements. And so then it was just a matter of what is the best model for them going forward. And it seems like the convergence on the powered shell model that the majority of them are going after believe out of the biggest one iron is the only exception is and they're going fully integrated. But you got a chance to look at one of these constructions at Terror Wolf's Lake Mariner facility recently. And so I would love to open this part of the discussion. If you could just tell our listeners what were some of your biggest takeaways from touring that site. It must have been pretty interesting.
- [10:56]
B
It really was, because I've toured a lot of data center campuses over the years, and the first thing is that this is an entirely different kind of scale than we normally see, and it was an extraordinarily busy construction zone too you know, you're used to seeing construction activity, but the thing about this is, and they've been public about their agreements with FluidStack, they're building two huge data centers for them at the same time. There is all sorts of simultaneous construction work going on on these data centers which are 330,000 square feet. It's probably the size of like, you know, three Walmarts each building. And so there's a constant, this constant frenzy of motion with backhoes and little trucks carrying construction equipment. There's huge cranes on site. There's the, the steel superstructures for the are, are up and they're framing out the buildings and it's a lot of activity and to the, to the point where one of the biggest challenges I think they have right now is directing traffic on the site as they're trying to move all the parts and equipment into place at, at top speed. And the thing is they're going 24 hours a day, they working around the clock, which is, you know, pretty extraordinary. And I think they said they've got, you know, apart from there's about 300 staff between Terra Wolf and Fluidstack, but then there's between and in any given day, between 1200 and 1600, you know, contractors, construction workers who are on site and working very quickly to bring all the components, the power, the cooling. You have whole UPS and generator skids that come in shrink wrapped and are next to the data center ready to go. So it's a really interesting job site
- [13:04]
A
that when reading your article of the tour, that's what really blew my mind was the scale of the contractors on site. And two follow up questions. So they have contractors working night shifts and I guess that's part of why this build out has been expedited. You mentioned that it's kind of breakneck speed. It's faster than you would expect for something like this. And the second question, those contractors looking at the job creation angle, which is often hit on by these data center operators, those contractors almost certainly aren't local. Right. Like they're coming from around the country based on who they contracted for each specific construction job. I would imagine there's a lot of
- [13:44]
B
that and that's a really big issue for the data center sector in general. It's no secret that community relations is a real challenge right now for the data center industry. People have mixed feelings about AI. There's a lot of attention right now to the resources that these large data centers use and they're way more efficient than they used to be. But at the same time, what all of these are doing is, is taking an enormous amount of economic activity and concentrating it at a single point in a community and in the network. So particularly there's a lot of power used. And that's a sensitive issue at a time when everybody's concerned about energy affordability. So the trade off, there's a couple of trade offs. Real benefits of having data centers in the community. One has been the tax impact which is often huge because of the amount of not even the building so much as the gear that goes in to these facilities is so expensive that like the sales and use tax on those things. You know, I think in Northern Virginia that those places it supports all sorts of things for the community because they get that tax break. The other thing is that it brings a lot of workers into the community. Now the data centers themselves are, are highly automated so they don't create a huge amount of jobs. So they're a little bit of an odd bird from an economic development standpoint. But from the, during construction there are these huge armies of construction workers that come. The hyperscalers in particular are working hard to work first with the local community and try to get folks who are in state, but they bring in a lot of people from out of state. And, and one of the huge problems for this whole industry right now is there are not enough construction workers, not nearly enough to keep pace with all of the projects that are out there. There's a number of things that are delaying projects, but that's the ability to get folks on site is a real challenge. So you're correct in that a lot of them will come from out of the area which then creates, you know, other challenges about, you know, you know, some places they'll come in with the size of the campus build outs now they're there for longer. It's not just, well, we build this one thing, then we go away. If people are building, you know, Facebook might or Meta might build nine buildings on a campus and as they do them one after the other, sometimes those people will be there for three to five years. So it's, that's a big bet factor. But getting qualified people on site is a challenge. And so you people, you see companies pull from all over based on the, the relationships that they have in the trades.
- [16:39]
C
We talk about like speed to power because that's I Terra Wolf's tagline. That's a lot of industry players. Tagline TERRA WOLF COO I said, I believe, said they believe they're quote the fastest ongoing data center build in the US right now. And I'm kind of curious, like, why and what factors I look at, like, zooming out really far. The Lake Mariner site, if I understand it's a brownfield site, so it's like redevelopment. And that seems to be pretty key to all this because then you have power, you've got transmission, you probably have city and infrastructure nearby. What like, are it, like, what's the landscape of brownfield sites in the US if you could even like, speak to that and like, are those all bought up and everybody called dibs on them or are there just. Or different plays to make? When we're talking about like site selection
- [17:32]
B
in the U.S. i think there's different plays to make. I think Terra Wolf is a really interesting example of a company that said, you know, we're going to work with the infrastructure of the old economy to build the new economy on top of it. Lake Mariner is along the shores of Lake Ontario, not too far from Buffalo. It was home to a coal plant for many years. That was the leading employer in the community and it had a substation there. And of course, because the power was going out from there, it has all the transmission lines connecting it to the grid. Now, you know, talking to the Terror Wolf folks, that's an opportunity for them. It's not like there's no work to do to make the transition. They had to do, you know, studies, interconnection studies with the grid. But what they told me was that that might take a year or two. Whereas if you're trying to get power from a utility and get a new connection, like in Virginia right now, that's seven years. Other markets are, you know, two to three years is, is standard Texas. They're having sort of contests to like, see who can get to the front of the line to, to connect. So any of the ability to use a brownfield site that already has either some kind of energy infrastructure on site and more importantly that has transmission lines to it. So old power plants, you know, Terra Wolf is also working with a couple of aluminum smelting sites, which of course have tremendous power draws. And so they've got a lot of infrastructure coming to their sites. I do think a lot of the opportunities are probably either in process or being scarfed up right now. But there's also a lot of creativity in people trying to think outside the box. Because what's happened with AI and this has played out very well for the crypto to AI players, is that all the old assumptions have kind of gone out the window to try and bring new capacity online. And the data center sector has probably never been more willing to think about new ways to get things done.
- [19:49]
A
I've got one, Charlie. I want to hit on the public relations nightmare angle really quickly one more time. I'm curious. I know that you talked to the Terra Wolf team, and so I'm going to kind of ask this in two ways. What did they say were the primary concerns for these AI builds? And did you get a sense of the community's reaction to this? Like, did you have any conversations with locals about their thoughts on the data center?
- [20:17]
B
I didn't have any discussions with the locals while we were there because it was kind of a quick in and out. But I've done, you know, I've read a lot about the community and what's going on. So the, the sort of backdrop on that is Terra Wolf came in first to do crypto mining on the site. That was. You can see the sort of evolution of their infrastructure as you go across this large campus. And early on the concern was about noise, that neighbors were concerned about the noise from the, from the mining rigs. I think as some of that capacity has been shifted over to AI uses, that's not so much a concern right now. The big concern is water. You know, they say repeatedly that they use a closed loop system, which is one where you fill it once with water and then you just recirculate it and use an outside cooling loop and a heat exchanger so that you're not constantly drawing water and evaporating it, which was a strategy that data centers used to use to optimize for their lowest carbon impact. That was the best way to do that, was to use water. Now there's a lot of pushback on water usage. They've got another facility nearby in Lake Cayuga, near Ithaca, where there's a lot of controversies and a lot of pushback about that. And of course, New York State, there's enough discussion about it that the state legislature has a proposal to do a moratorium on data centers. I'm not sure how long it is. They're typically about a year to 18 months. And the governor might have a decision on that if that makes it to her desk. So there's, there is a lot of community pushback, but what the Terror Wolf team told me was, was, look, because they had a lot of, you know, public information sessions and lots of times these are really difficult discussions. The people who are coming to a town hall are concerned about what this huge project is going to mean for Your community, what, what's going to change, particularly in rural areas. I think there's a lot of concern that a facility like this could change the character of their community. And so there's a lot of tough questions. And the Terror Wolf team just said, look, when the questions get tough, you have to be thick skinned and you have to listen more carefully when the conversation gets difficult because those are the things ultimately if you are going to be a part of the community for a long time, that you can't just steamroll the local population. You have to hear them, you have to hear their concerns. And historically for Terror Wolf is just in one scenario. The data center industry as a whole over the years has really tried to just, I'm not going to say steamroll is the right word, but they prefer to work with local officials rather than with the public and that has had repercussions. Now there's a reckoning that's come. There's some polls showing that 50 to 70% of Americans aren't interested in having a data center in the community. But meanwhile everybody's using data centers all day long for just about everything. YouTube, Netflix, your online banking, your social media. Ironically, the most of the community groups that are opposing data centers are using Facebook to organize and do it. And you know, so it's like they're using AI to data centers to fight AI data centers. But, but I mean I think the thing that I thought was interesting about the Tower Wolf team is they were like, look, when the conversation gets tough you have to listen harder and you have to convince people that you're going to be a part of the community, that you have a vested interest in being there and that people are being heard. And you know, that's, that's not an easy discussion, but that's, that's what's going on right now.
- [24:44]
C
I spent a while in oil and gas. I'm familiar of like the criticism of people who are also using it. I kind of know that. But, but what's interesting about the data center pushback is it's pretty apolitical. It's cross party, cross socioeconomic. It is a, it as you said, it does feel like it's a reckoning coming from the industry. Okay. I kind of want to like zoom out as we kind of wrap this up. And you again you say 25 years in the data, data center infrastructure world, you've seen it change. And Colin, I guess relatively new careers in the non bitcoin data center world. What are we missing? Like what is the, like what is like the. What's like the bitcoin former bitcoin crypto folks missing or trends in the traditional data center industry. Maybe it's like constraints or bottlenecks we're not aware of. What are blind spots from those of us former bitcoin miners now piling into the AIHPC game.
- [25:52]
B
I think the first thing that a lot of these companies really are going to need to think about in coming years as they scale up these huge operations is talent. We talked about the difficulty of finding people and there's a limited supply of folks who can operate AI hyperscale data centers. A lot of those folks are already working in the industry. All of the companies that have traditionally been working in data centers are hiring like mad. So I think that's one of the things that's going to come up a lot as an issue probably is trying to find skilled staff who can understand not just the infrastructure you need to run a crypto mining campus, but the additional layers that are involved with dealing with AI customers. The other thing that I think is really important that I think a number of the key players have done really well at is sorting out the financing to scale and the way that some of these players, I'm thinking Terra, Wolf and Iron and Hut 8 have been able to enlist their customers who have enormous financial strength to support them in ways that can help them put deals together to get the capacity deployed. There's a lot of, I think, creative thinking going on there and I think that's one of the ways where we're, you know, there's innovation that's moving both ways here. But I do think that the data center industry is becoming stratified in a fascinating way because there's the crypto to AI players, most of whom are doing the powered Land or powered shell model. Then there's the Neo clouds who are actually leasing out the GPU space, which apparently now includes SpaceX and that's a whole other asset class within there. Then you have your traditional corporate colocation centers and then there's the folks who have Google and Meta. Microsoft are still building their own data centers as well. So it's a fascinating landscape right now, but it's pretty wild because to your point, two or three years ago when I looked at these companies expressing HPC and AI aspirations, I'm like, yeah, that's not likely to happen because there were big differences, but the gap has been narrowed pretty quickly when everybody's aligned in getting more capacity online.
- [28:36]
C
Yeah, it's almost never been a better time to be a nerdy data center guy. So you spent 25 years putting in the work, and now you could be the expert. And every dinner party, you get to know every single thing about what's going on.
- [28:49]
B
Well, it's wild, because now I don't have to explain what a data center is anymore. Everybody knows. So it used to be you say, yeah, I write about data centers. They're like, well, can you come fix my printer?
- [29:04]
A
I won't speak for Charlie, but I feel similarly because I would usually just tell people I'm a tech reporter, and then maybe if they knew what bitcoin is, I would say, like, oh, I do bitcoin or bitcoin mining stuff. And half of the time, people are like, oh, okay, that's really interesting. In the other half, that they're just eyes glazed over, have no idea what you're talking about.
- [29:23]
C
But now it's AI and people care
- [29:24]
A
about it, and they either want to skin you alive when you mention it or they are really interested in what you have to say.
- [29:32]
B
You never know.
- [29:34]
A
Well, hey, Rich, thank you so much for joining, man. We'll have to do this again sometime. Keep up the great coverage and appreciate you hopping on.
- [29:41]
B
Yeah, absolutely. This has been a blast, and I really appreciate the invite.