← Currents
Currents
Ep333: Rethinking Forecasts in an Uncertain Climate
00:29:41
Rob Cirincione of Sunairio explains why traditional weather models miss the extreme events that drive grid risk and how high‑resolution, ensemble‑based forecasting can better predict impacts on power markets.
Loading summary
Transcript36 lines
- [00:00]
A
Foreign.
- [00:05]
B
Welcome to Current to Norton Rose Fulbright podcast. Today we're recording with Rob Sorencion, CEO of Scenario. Rob joins us to discuss the dangers that we're facing and the power grid as a result of weather related extreme events and a lot of the outdated weather forecasting models that most of us are relying on, at least according to Rob. So. So Rob, thanks for joining us today and hopefully enlightening us.
- [00:29]
A
Thanks, Todd.
- [00:30]
B
All right, so first, since not everybody might know what Scenario is all about, maybe you can give a quick introduction of yourself and also tell us in a couple sentences worth what Scenario is all about.
- [00:44]
A
Sure. First of all, thanks a lot for the invite. Nice to be here. Starting with my own background, I am trained as an engineer but never really worked as an engineer. Spent the bulk of my career working as a power and gas trader. Spent a long time working for Constellation and then also worked for a shop called Boston Energy Trading and Marketing, which at the time was a division of NRG and was later spun out to Mitsubishi International. And sort of throughout that career I traded pretty much everything, power and gas and the Eastern Interconnect and Ercomet and then left that several years ago to found this business Scenario. The reason I did that, quite simply, is that I felt like my job got really, really hard as a trader. And we'll talk more about that. But broadly, what we do at Scenario is generate high resolution, high fidelity, both historical and forecast data of weather, power generation, assets and our markets.
- [01:48]
B
Okay, so why does somebody who's a, an energy trader decide to become sophisticated weatherman?
- [01:54]
A
Right? Because pretty much everything that's interesting in, in the power sector, in power markets is, is driven by weather and weather volatility. That, that's kind of like, like point one. It's what you find very quickly when you become a, you start working in the power sector. And specifically for me, you know, working as a, working as a trader, you spend half your day staring at weather. You, you are sitting at a, you know, sitting at a desk watching the weather models come out. You are staring at historical weather data, trying to understand what happened or put together, you know, models of, of what could happen in the future. It's, it's really just sort of, you know, the primary, primary driver of the most important reliability and economic outcomes in, in the power sector.
- [02:38]
B
So I assume like most trading, that the way to outperform the market is to have better information and interpret the information either quicker or better than your competition. How do you do that? All the weather information I'm assuming is, is for the most part derived from things that come from the government. You know, how do you one, is that true and how do you use the weather information then to get an edge? Is it because you can translate it more quickly? Is it because you can translate it better or, or a combination of the.
- [03:14]
A
Both. Yeah. So that's a really, actually that's a key word, translating in translation. And so our key value proposition, the differentiator, you know, at scenario is that we are hyper focused on translating weather and climate risk to grid risk. And that is not trivial that. And you're right, which is that when you said that, well, I assume, you know, most information, you know, starts with sort of data from the government. And that's correct, although it's changing now, right? It's changing now with, with AI weather forecasts. But for most of, you know, for most of the time until, until a year or two ago, and still most of the information that, that we're getting starts with data that's publicly available from government sources, government forecasts, public. But there's a huge gap in terms of taking sort of a, you know, raw data or raw model output and translating that into grid outcomes. And that's what's, that's what's so difficult and takes, you know, such expertise and that's really where the, where the gap and the opportunity is that we fill.
- [04:27]
B
And so how do you do that differently than the guy sitting three buildings down from you?
- [04:35]
A
Well, so, so first of all, I think we do a number of different things because we have different perspectives in large part because of the, you know, the way we're, the way we were founded, right. We were founded by somebody who was in the trenches, you know, managing this risk every day and who understands what all the little, not little, but, you know, all of the gaps, all, all the nuances are and we have very, very specific needs in the power sector. So for example. Right, right. In, in the power sector we need to do things like we need to forecast the generation of a wind farm or all of the wind farms in a power sector, you know, like ERCOT or PGM or something like that. And we need to do that at least hour by hour. The reason we need to do that is we care about supply, demand balances and regional power grids 24 hours a day. The grid needs to, needs to be in balance at all times, minute by minute. Right. If it's not, then you're not delivering reliable electricity at the, you know, correct frequency. And, and we can risk, you know, all sorts of outages so the grid apps has to be in balance at all times. So we don't care about just, you know, what's the high temperature on a certain day. What's, you know, what's, what's the low going to be here. We care about it all we care about 24 7, what, what that forecast looks like now right there. You can't do that. You can't forecast the hourly generation of a wind farm from the raw forecasts that, for example, that are published by Noah or Noah's counterpart in Europe, the ecmwf. The reason you can't. Couple reasons. So first they don't issue forecasts that cover every hour. After a few, even, you know, after a day or two, they start issuing like every three hours or every six hours. Well, that's not good enough because you might have a situation, you know, within that, you know, three hours where things get squirrely. That's, that's problem number one. Problem number two is they don't do it at the right resolution. So for a lot of things that we care about, increasingly we do care about things like wind generation and solar generation, right. We've built a ton of capacity of renewables in this country over the past few years. And the weather that drives renewables like wind and solar is highly, highly variable over short distances. So the wind, the wind speeds at a location, you know, a specific location might be very different. Just you know, a couple hundred meters, half a kilometer. Especially if you're going over terrain, you're going up a ridge top to rebuild the turbines, right? You need to know the wind speeds exactly at those locations in order to have, you know, accurate, accurate forecast. The standard, standard forecast don't issue their data at a resolution that's sufficient enough to model these things. Then there's a third problem and the third one's really, really key and you kind of mentioned it earlier and that is that, you know, I said, well, broadly like whether's driving like all of the important outcomes, right? That, sure. Well then if you look at like really, you know, the lion's share of, of kind of the important outcomes that we care about, they are driven by extreme weather. It's the extremes that matter in, in the power sector, in power markets. And here's, here's a statistic, statistic for you. So in ercot, the Texas power market, right, a third of the annual market value, the annual real time market value accrues from less than 1% of the hours per year. So just 1% of the hours per year accounts for a third of that market value. And it's because, you know, just a few hours of the year spiking and going nuts due to some, you know, extreme weather, unusual combinations of weather. Those are the things we care about, right? And modeling, forecasting and getting those extreme conditions right is really hard. And it's something that the sort of like the standard weather forecast, they do not do well. And so I think for all of those reasons you can't just take the, you know, you can't just take the public forecasts or public data and expect to get a sophisticated, accurate, you know, nuanced view of, of, of power sector and power markets risk. There's so much you need to do to enhance that and that's what we do.
- [08:44]
B
On your ERCOT statistic does the penetration of energy storage and now there'll be some more gas fired assets eventually there, does that shave some of the variability off or is it the problem just getting worse and worse?
- [09:01]
A
Yeah, maybe not. And so I agree with you, right? If you said like, well, if you had, you had storage, that's kind of like a, you know, kind of peak shaving things, you kind of flatten things out. Maybe not. And in fact, when you look at markets, right, that have a lot of, lot of renewables and an increasingly, a lot of storage, you know, like a California and increasingly ercot, they're still very spiky. You move the spikes, right? They get moved to different hours. So what happens for example in ercot, and we're even seeing this in PJM now, as you build more solar, you're moving the, the peak price hours, sort of your riskiest hours during the day, for example, in like the summertime, you're moving it later, you're moving it later because what happens is, you know, in the afternoon. So historically maybe, you know, 5pm the hottest, maybe that's the hottest hour in July or something that we could kind.
- [09:53]
B
Of people come home and put on their air conditioner.
- [09:55]
A
Yeah, we used to, we used to expect that that hour, you know, would, would be sort of the, the peak demand and therefore the peak price hour. Right. Well, what happens when you have a lot of solar is, well, the sun's still out In July at 5:00', clock, right. So there's tons of free generation. So actually prices can still be quite low. However, you know, three hours later as the sun, sun starts setting, but it's still hot on a hot day. 8 o' clock could still be really hot outside the sun. The sun sets now, you can lose more generation than demand and you actually have to ramp up batteries or anything else that's dispatchable. And so you get these, you get these late evening ramps. And that's what we're getting in ercot. Now you can see these peak hours moving later in the day, these peak net demand hours we call them, they're the peak that influence peak prices. They move later in the day. We're seeing it in pjm, it happened last June with the heat wave in pjm. And honestly what's happening is you're moving kind of a, you know, what used to be a three, four hour kind of high priced peak and you're squishing it into one or two hours. And I think, you know, an interesting point that can be made here because we think about all this stuff because we've really, we go the whole way through, we, I mean we start with the weather, the weather and the weather tech and the weather forecast, but we do, we do asset level and regional grid modeling and price forecasting as well. And I think one of the things that's really interesting is if you look at a market like Texas where you're squishing all of these high prices into just a few, you know, a few, one or two hours a day. When we look at it, it could be the case that prices and price caps might actually might not be high enough. That's almost how like spiky, potentially you could have a market like ERCOT where you're moving all of that high price risk into just one or two hours. And in order for there to be sort of equilibrium and provide a signal for new investors to actually build stuff in the future, you would have to have extraordinarily high prices for just one hour or two hours. And I think, you know, in some cases we look at markets like ercot and say $5,000 price cap might not be high enough.
- [12:06]
B
Moving away from ercot. But just a general question, since a lot of audience does project finance and I definitely do, we've had other people on the podcast over the years who have said that a lot of the forecasts for production, for wind and solar have been overly bullish and definitely not correct. And I always thought, and I challenged them and would say, hey look, okay, that might be true a year ago. Then I would expect, you know, the market's gonna say hey look, this too bullish, so let's figure out what we did wrong. And it closer and closer and closer over time because people would realize there's a mistake and try to identify where it is since you're doing this, it sounds like, you know, down to hour, to hour by hour when, when we're predicting out the, the weather over periods of 110 year periods. One year periods, how accurate can we really get? And are the predictions, have the predictions in your view been too bullish? And what, what's the state of the prediction market today in terms of coming up with forecasts for the production tied to weather?
- [13:18]
A
Sure. So, so let me make sure I, and I think I understand. I just want to make sure that we're talking about the same thing. So generally you're talking about pre construction estimates of new project production over, you know, investment time horizons, 10, 15, 20 years.
- [13:34]
B
Yeah, usually people are looking at like a P50 for one year or 10 years.
- [13:39]
A
Okay, I don't know that we have enough time on this podcast for me to actually give you all my opinions. But let me, let me, let me, because yes, I have a lot on this one. But, but let me, let me summarize and see what I, Let me, let's see if I can, I can pick a few. So first, yes, in many cases they're, they're too high. And yes, the whole industry knows that. Personally, I think a lot of the reason they are too high just comes down to sort of maybe poorly aligned incentives. Projects get built by people who are incentivized to build projects and projects look better when they have higher production. So in theory there, you know, there, there are, there should be safeguards against that. There should be independent engineers who could safeguard that. But there are all these ways to kind of game the system and whatnot. And then you'd say, oh, but the lenders should really, you know, push back on these, you know, you know, projections and this or that. Well, the lenders don't really, don't, don't, they don't really bear the risk. Right. Because they're at the top of sort of the cash flow waterfall and they honestly don't kind of wear the risk of being wrong by a percent or two. But you know who does? The equity owner. Right. The ipp, the person who's going to own this project for the next 15 or 20 years. So one thing that we see, which is kind of weird, but it's sort of the state we're in. We will often talk to a prospective new customer, an itp, and they're interested in sort of our forecasts of their, their portfolio production over long time horizons. Because we can do, we can actually, we, we run our forecasts out from, from next hour all the way through next decade. So we can handle operational tasks all the way through project finance. And so, you know, what we'll talk to them is and we'll say, look, you know, the methods we use are generally, you know, not consistent for a number of reasons with, with what, you know, you may, you may have used in your, you know, your pre construction, you know, pro forma or something like that, like you. And, and they're, and they're going to look a little different. And more often than not, if we're talking to like the development side of that ipp, they'll say like, oh, well that's not going to work. You know, then we can't have, you know, that we need to be all, everything needs to be consistent, same methodology, everything, everything. Like, okay, fine, if we talk to the group that manages the fleet, right, that's in charge of like the actual risk day to day, they'll say, oh, that's great. I don't want to use that stuff. I just want to know, I want the most accurate projection of this. And I, yeah, it was, it was, it was built to, you know, that, that pro forma three years ago. But we know that's not right. We can see it, we can see it in, in the data. Like we need to know, you know, we want to understand how this fleet is actually going to, to perform and which is, you know, it's a little bit crazy when you think about it. It's almost like having two sets of books, but it's kind of, it would be nicer if things were a little bit more harmonized. But I think it comes down to.
- [16:28]
B
Incentives and how you read about in the headlines, you know, in the news that the climate is getting more and more unpredictable. I don't know if that's true. Is that true first of all?
- [16:40]
A
So, yeah, it's, it's definitely getting, it's definitely changing and getting and getting more difficult to anticipate.
- [16:45]
B
So they're more and more. And so are the models that people used five years ago, 10 years ago. There are massive changes underway here to reassess how we're making our predictions or what's going on to basically adjust for this variability.
- [17:00]
A
It's a great question. What I would hone in on this whole issue of how is, how is climate changing and how does it impact us in, in the power sector? The way to not think about that is in terms of annual averages like, oh, climate change means that, you know, temperatures are increasing. But again, go back to like what is interesting in the power markets. I said it's extremes. It's those extreme events. And in fact, that's what's the most interesting thing about climate change. That's where the impact is. Because if you shift these, you know, distributions, and that's what weather basically is. It's some basically, you know, random distribution of, you know, temperatures and wind speeds and everything else. You know, it could be cold, it could be hot. It's, it's very, it's quite, it's highly variable. But it follows some, some distribution. And if you shift that a little bit, what you do is you in some cases exponentially increase the frequency of extremes. Right. So you can go from a case where 100 degree day might be expected to be observed, you know, one or two times per year, to a situation where, you know, just a few years later, 100 degree day could be expected 10 times per year. That's a 10x, right? One to 10, that's 10x. And if that, if the issue is, if you're a utility, you're a grid operator. Right. And 100 degree days are a problem for you. Going from one problem day to 10 problem days, that's a big deal. That's a huge deal. And that isn't necessarily something that, where it's like, oh, we need a new, like, fundamental climate model. No, we know this. We're just not translating, we're not, we're not doing the downstream kind of analytics to translate these, you know, higher frequencies of extremes and say, what does that mean to reliability? What does that mean to profitability? That's where I think, you know, we've been pretty vocal about, you know, making the case that like, that that's where there should be a lot, that there's been a big gap and there should be a lot more attention.
- [18:59]
B
I know you got this scenario one, maybe tell us what it is, how it's new, how it's different and what, how your models are changing.
- [19:09]
A
Yeah, so it's basically what I've been describing, but we branded it Scenario 1. One stands for an omniscale next generation ensemble. We'll break that down. So first, the ensemble component. So in forecasting there are, it's kind of the standard way of thinking about forecasting is what we call a deterministic forecast. So you look into the future and there's one prediction for every point in time. Right. Your traditional weather forecast, you open up your phone, it says it's going to be 70 degrees tomorrow at 3pm Right. Problem is, we know there's a lot of uncertainty in that in that forecast and especially in this industry, we need to understand that range of uncertainty, that range of risk. Well, sort of the best practice approach for doing that right now is what's known as ensemble forecasting. So you don't just, just issue one forecast, you issue many forecasts. Right. And somehow you change, you change your assumptions a little bit and you can, and you can generate another forecast and try to capture the range of uncertainty, whether it's the range of uncertainty tomorrow or next month or next year. Right.
- [20:11]
B
And so a Monte Carlo simulation kind of idea, that's the.
- [20:14]
A
Yep, exactly. So a Monte Carlo is an example of an ensemble forecast. But there's many examples, right? You could say, oh you know what, I don't. And here's something we used to do before I used to do before I had scenario, I was a term trader. So I was looking out months and years into the future. Well, I didn't have forecast out there. So I would use history and I would say, well I've got like 20 years of history, that sample of 20. So it's like an ensemble forecast. Not great, but it is. And so yeah, so we have a very specific way of generating this ensemble. We happen to use a hybrid machine learning and a statistical approach. And so scenario one generates a thousand member onsite. We generate a thousand, which might sound a little ridiculous at first. Go back to the ERCOT statistic. In power markets you need to see the 99th percentile and beyond. We really, really care about the tails and they're really extreme. And so that's why we generate this thousand member ensemble so that you can see the tail events in the extreme. So that's the E, the N. I already mentioned next generation. It just means that this model is not based in traditional physics, it's primarily machine learning and statistical model. It's trained off of traditional data and physics based data but it doesn't generate it's output that way. And then the omniscale component. So yes, we made up that word but it basically describes the fact that this model generates forecasts from next hour to next decade. So we, we're moving across short, medium and long time horizons. And we do that for a very, very like, for a very commercial reason. Right. Which is that generally in business like our business needs don't respect the boundaries of like traditional like Noah's weather forecast, Noah issues a, you know, they've got a 48 hour forecast, they've got a 14, you know, 15 day forecast. Like that's great. I'm looking At a Balmo power contract that's 24 days long. What do I do with that? I'm looking at a PPA, right? That's, that'15 years long. I want a seamless solution. I want one solution of the future that I can put into my proformas or, you know, my irr, something like that. So that, that's what we're providing under the hood. We have to stitch together, you know, different methodologies to forecast next hour or next day, next decade. But we do all the work to make sure that it's physically realistic and consistent.
- [22:48]
B
So it sounds like you're using AI to come up with weather forecast or a model here. Is that basically what you're saying?
- [22:58]
A
So it's related and. Well, I guess first thing I would say is we don't use. So we don't use a lot of the AI methods that like a DeepMind or Nvidia is doing in their forecast. They rely a little bit. They are almost, I'm saying almost. DeepMind has had a number of models come out, so I really am not even aware of the, probably the way the most recent ones work. They rely on AI a lot more than we do. I would say, first of all, and I described our model as a hybrid and I think that's important to understand. We've done a lot of research into AI solutions for weather and grid forecasts and stuff like that, and there are a number of limitations that we see commercially. Number one is this, this issue of extremes. Like on their own, the AI is not going to do a great job in the extremes because you don't see them, you don't like, you can't see them in history. And so when I can't see something and can't train on it, it's really unreliable in terms of, in terms of predicting that and maybe not as much related to the weather, but especially as we get towards like grid forecasting and market outcomes. You know, another challenge with us, for us is that, you know, we're giving this to customers who are investing or, you know, risking a lot of money and they need, need to be confident to understand sort of how the whole, how the whole thing was put together and how it works. They can't say, for example, like, hey, boss, you know, I think we should, I think we should build this, you know, $300 million, you know, project in ERCOT. Why? Oh, well, the black box says it's, it's going to be great. It just doesn't work. We get this over and over from customers why? Why that outcome? Why? And it's a real, real challenge for AI. And so in this sort of application, like we need to bring in other methods to be able to trace back sort of like the why.
- [24:54]
B
All right, so let's wrap this up with kind of the natural question based on what you just said. Who's the target audiences, plural? I'm sure there's more than one for, for your product and, and how are they using it? And if people are interested in learning more, how do they get access to this information?
- [25:13]
A
Yeah, sure. Three primary target customer segments. So the first is traders spent a while talking about them obviously. You know, I think you can imagine how traders use, use forecasts. They're, they're using this to inform their decision models that are helping them monetize information in, in markets. And they do that both they can. Some, you know, some we do the whole sort of weather, energy grid markets continuum. Some of our customers just want the weather, some of our customers want sort of the energy and grid. Some of us want, you know, the whole way through like kind of the market, market forecast. That's fine. Second would be the ipps. And, but as I said, the IPP really right now what we're finding is the primary use case for the ipps is understanding and managing and hedging and hopefully improving the risk profile of an operating fleet. That's where we see the biggest sort of the biggest benefit. And maybe I'll make one comment there on the IPP thing. There's kind of another customer class that is related and we're seeing some new interest from, and it's corporates, right? Like Fortune 500 corporates who have done, who themselves have done like renewable PPAs. And what we're, what they're finding and we could have told them this, but what we're finding that they're finding out is that. So let's say you're a big box store, right? You're a big box store, you got tons of load. You have these stores all over the country, right? And then you know, you went out, you did all these renewable like you know, solar and wind PPAs. You're, you're basically a, you know, like a utility, right? Or like an integrated merchant. You're really no different. You've got a huge load and variable generation position. And what we're increasingly seeing is they are understanding that that is a very, very complicated position to manage and understand and they'd like help sort of understanding how to make better decisions and reduce that risk. Like hey, should I Next, should I buy this PPA or that ppa, what's going to fit better into that portfolio? And the third category, now that we're kind of talking about, you know, putting together a portfolio a little bit more like a planning decision would be reliability organizations. So the final kind of use case customer segment, the third one really that we have is for vertical utilities and grid planners. That I think is a, is, is just a, you know, a killer, you know, application of, of this technology for all the, all the reasons we've talked about. And it, it might, it probably wouldn't shock you. But you know, one of the things that I'm always kind of, kind of amazed about thinking back to the extremes is still today, if you look at the, the planning studies, like the long term planning studies that, that any sort of, you know, major vertical utility balancing authority or even like PJ go to PJM's, you know, most recent like long term planning study, go into their assumptions, go into the assumptions and they'll have their load forecast for the example and they'll say, well, here's our load forecast. Where are the weather assumptions? Well, we assumed weather from 1990 to 2020 and no, we didn't account for any sort of changing weather. We did not account for climate change. It's in the document. And I think that's also kind of just amazing in the time we're in and how critical, I mean the whole planning, whole planning study is to try to figure out can the grid withstand extreme events. And I think there's low hanging fruit that, you know, in terms of, you know, what they're, what they're not doing there. So we come in and we really like working with, you know, with the, with these organizations to kind of, you know, fill that in.
- [28:52]
B
All right, well after leaving us with that scary message there that we're going to have some brownouts, did.
- [28:58]
A
I didn't say that. I didn't say that.
- [29:02]
B
Thank you for joining us today and enlightening me a little bit on how this works.
- [29:07]
A
Awesome. Thanks a lot for having me, Todd.
- [29:12]
B
You can find us online at www.projectfinance.law or send us an email at currentsortonrosefullbright.com Please rate, review and subscribe on Apple Podcasts, Spotify or your preferred podcast app. Our show today was produced by Emily Rogers. Stay ahead of the Currents.