← Emerging Litigation Podcast
Emerging Litigation Podcast
Agentic AI on Trial: You Be The Judge Part 2 - Smart City Traffic Control
00:22:23
Loading summary
Transcript13 lines
- [00:00]
A
Foreign.
- [00:10]
B
Welcome to the Emerging Litigation Podcast. I'm your host, Tom Hagee. This is part two of our series of you Be the Judge Agentic AI on trial, AI liability and risks. In this episode, we're going to talk about traffic. AI is transforming urban traffic management worldwide from Los Angeles to London. If you've driven in either of those towns, you know, they could probably use it. Adaptive signal systems powered by AI can cut congestion, emissions and travel times. You know, you must like me, you often think when you're sitting at a traffic signal, a red light, and there are no cars coming in any other direction, you're sitting there, you know, contributing to pollution. And you're also wasting your time. But it's a good time to check your phone, no? So you know that that leads to emissions that aren't necessary. In Pittsburgh, Pennsylvania, where I grew up near there, its system reduced wait times. It's called their SirTech system. It reduced wait times by 40%. Who couldn't use that? And emissions by 21%. You know, that was a city that in the 60s, the lights would go out or lights would come on. The street lights would come on in the daytime. Those were smart street lights. But there was so much pollution that it simulated nighttime and tricked those streetlights. They would come on. Okay, that's not the case anymore. Pittsburgh is beautiful. Predictive modeling now forecasts congestion. I'm not talking about. I'm talking about traffic congestion. It now forecasts that with 85% accuracy. And integration with connected vehicles enables rerouting of those vehicles. But what happens when this, when this autonomy fails, when it just suddenly doesn't work and there is congestion, people are stopped unnecessarily. You're going to see how that can play out. In the scenario that our panel will provide and discuss. They're going to talk about who bears responsibility. Is it the city, is it the vendor, the policymaker, somebody else? So they're going to examine foreseeability, sovereign immunity, and auditability in public AI deployments. Again, this panel is Galena Dostkovsky, PhD. She's a business strategy advisor. Marina Kaganovich, she's an attorney and compliance advisor at Google. You've heard of them. And an actual judge, Judge Lisa Walsh of Florida's 11th Judicial Circuit. Also, I want to thank Katherine Ratigan, who is a partner with Robinson and Cole in Providence, Rhode Island. She pulled this all together. I've had the pleasure of working with her before. She's always working on some aspect of the law as it relates to emerging technologies. She's Also going to provide a more in depth introduction of our panel. And as always, the opinions expressed here are those of the individual panelists. They are not those of the organizations they represent. And with that, here's episode two of Agentic AI on Trial, AI Liability and Legal risks. AI. Hope you enjoy it.
- [03:08]
C
We have three leaders in legal compliance and technology. We first have Galena Datzkowski. She's an internationally recognized authority in compliance, information governance, AI data analytics. She has a PhD in computer science and really deep expertise in AI. She advises a lot of organizations on business strategy. She's on a lot of boards. She's just really interesting to hear her background and has a lot of great knowledge to share with this community. She's joined by Marina Kaganovit, who's an attorney and compliance advisor at Google. She also specializes in AI governance, cybersecurity, risk management and data privacy. She works with a lot of executive leadership teams to talk about how to secure cloud migration in a compliant way. This is an evolving regulatory environment. She's really knowledgeable in that. She does global cross functional programs for, you know, a lot of different organizations. As an advisor, she's part of the ARMA International Board. She's really a thought leader in this space as well. And then we're also joined by Judge Lisa Walsh, who is a circuit court judge in the 11th Judicial Circuit of Miami Dade County, Florida where she's an administrative judge of the Appellate division and she participates in some international arbitration division as well. She, I think an interesting fact is she's done over 100 jury trials, both criminal and civil. She has a lot of appellate decisions out there, lots of experience, and she kind of brings her judicial experience and insight into this discussion. So they're really a great group and they represent a pretty, a big. It's kind of interesting because they converge on law, technology and governance and justice. Really all their perspectives are really interesting to put together. And this you Be the Judge podcast.
- [05:00]
D
Okay, so let's look at a different scenario. In this podcast we've started with our you Be the Judge podcast on Agentic AI. And in this scenario we want to take an entirely different tack and get away from our medical issues and look at the use of agency in other areas. So in this scenario we have a major metropolitan area. The major metropolitan area implements an agentic AI system designed to autonomously manage all aspects of urban traffic flow. What does that mean? That means optimizing traffic light timings, dynamically rerouting public transport, advising drivers via navigation apps, and even coordinating emergency vehicle movements. So all of this is now happening through agency. The AI is constantly learning from real time data to minimize congestion and to minimize commute times. However, we have an unforeseen series of unfortunate events that happens to occur simultaneously. So we have a large public, large scale public event, multiple unexpected road closures and a severe weather anomaly all coming together to really make the life of the AI agents quite difficult. The AI autonomously adaptive logic, in its attempt to over optimize for local conditions without a global systemic understanding, creates lots of bottlenecks and directs traffic into increasingly congested areas. It's an independent decisions lead to a large scale gridlock that lasts multiple hours, causing significant delay, delays in emergency response and an increase in minor traffic accidents. So we have quite a mess on our hands obviously in this particular city. One of the consequences of this mess is that an ambulance cannot get to the hospital for hours and impacts the patient who is being taken. The patient does not survive the ride. So we have some accidents, we have some severe outcomes, we have a death. Since autonomous and human driven vehicles do share the road, given the number of convergent events, the system can't cope with the predictable and unpredictable decision making. The unpredictable decision making of course being done by the humans. And further, due to interconnectivity, not only were the operations of the smart city impacted, but so was navigational guidance which also tries to optimize for conditions and, and starts redirecting people. Now we are in a real mess and a jam and that's the scenario. And once we have an absolute mess and everybody's nerves afraid, let me ask our judge. So we have this mess, we have all of these consequences and what are the potential causes of the problem? Is it the trading data back to that trading data idea that we had in our first example with the medical system, Is it that the system is trained on typical data, not extreme scenarios? Other suggestions that you might have, what is the impact? And further, what should the agent do in such a case that the AI should have alerted the police, followed some kind of escalation, you know, closed all roads until further notice like we did on 9 11? What would have been a reasonable way to deal with the problem? And of course we'll get to liability soon, but let's do that first.
- [08:49]
E
All of this is real. We invented these scenarios and then keep coming across stories in the news of this type of bad outcome actually happening out there. So the issue is the adaptability of the particular system to a series of unfortunate events. All the training that you can input in all the data you can input in conditions on the grounds. If the aliens attack, you know, unexpectedly, they land in the middle of your metropolitan area highway. And that's never happened before. The system may be completely bamboozled on, on how to, you know, how to react. It's again, it's, it has the agentic capability, but it is not, it's not a human brain. It's not the team, you know, it's. It's, you know, this, that that's necessarily the problem. So as far as what the potential causes of the problem can be. Training data. Is the system not trained on how to, how to manage events when all roads are gridlocked? So that's kind of critical. You can have, and we've all experienced it. We all live in major metropolitan areas. I think, you know, the president is coming to town at the same time that there's closed bridges and there's a major accident on another artery. Everything happens at once and there's unexpected gridlock. And the solution to it is usually human interaction that the police are called out to a certain area to direct traffic. But if everything is automated and you don't have the humans on the ground that are called in to redirect traffic and it's rendered to be impossible, where will the accusations fly? So, so the first question would be if the developer is being called out to be the cause of the problem and they're the ones that are finding themselves in court, the creator, you know, how was the system created? Was it just unifying, you know, Waze and Google Maps and the Maps app and satellite data and all of that being poured in with certain prompts? Or is it, is it being trained on scenarios? Was there a robust training function for various scenarios that allowed the system to pivot and decide how it was that they were going, it was going to direct traffic to proceed to lighter arteries or, or call in for help. So if the system were trained on, quote, typical data, not an extreme scenario, and that's all it was trained to handle, that's when you, you end up with, you know, perhaps a, a waymo that just circles LAX for five hours and never lets its, its, its passenger out, which is something that, you know, happened recently, or the inability to handle, to have an alternative that it directs all of the traffic lights and all of the aspects of the urban system in question to react and handle. So as far as the question as to what the agent should have done, some of the things that courts might look at or that, you know, a plaintiff may, may argue is that there should be a safe, a safety valve in the system for calling in the police to direct traffic or some kind of an escalation procedure that interrupts the system to call in human interaction or some other fail safe to handle things when the gridlock reaches a certain area, the delays reach a certain area, situations like that. Or there's a way to close roads for emergency traffic, for example, as a fail safe to ensure that emergency vehicles can get through. Marina, do you have any ideas on what some of the other potential costs causes could be for a system to overload in that way?
- [12:51]
A
Yeah, I mean, I think, I think you're right. Like certainly looking at the way that it was trained would be really critical. And then as you mentioned, having some sort of escalation protocols. And we often talk about, just in general, we talk about governing AI and now with agent AI as well is looking at where are those choke points where it makes sense to have a human in the loop. And so in the scenario that we had devised, there is no human in the loop. It looks like the navigation and the traffic management is exclusively left to the AI agent's domain. And in our scenario there are some dire consequences. But I think you're right in the sense that certain human in the loop aspects can be programmed in like if certain conditions are met, then escalate. But there should also be this concept of even without that even without certain conditions being met, it's just that there's sufficient uncertainty and sufficient outcomes that are deemed to be negative in those scenarios. There should be some sort of escalation protocol as well. And I think that that's something that should all should primarily likely be with the deployer of the system and defining what that looks like and then determining whether the developer of the system that they're getting has, has arranged for those types of controls to be implemented. So either by default or ones where they can be flexible in adopting this, this AI for use and making sure that there's enough flexibility to have a way of identifying the types of scenarios where a human would be needed. And then if a human is needed, who that would be? I mean, you mentioned the police, but it might be other forms of emergency personnel, for instance.
- [14:39]
E
So what concerns me is if a system like this is rolled out and this also is not, it's not fictional anymore, it becomes an opportunity for, for example, a municipality to cut costs and, you know, and eliminate certain positions and certain jobs in urban planning or traffic management, for example. So the only way to ensure that there can be quick response if there's human interaction is incorporating that into rolling, rolling the product out. That it, you know, that if, if this is going to happen, this is who we call so that there needs to be an actual person there, there needs to be an actual department there. If that's the fail safe. If there's no autonomous fail safe that can be implemented, you know, autonomously that doesn't require human interaction. For example, maybe there could be yet another agent that is the urban planning agents, and they are the ones that exists, that separate thing exists to call the right person to bring in the police, to bring in traffic management, you know, and do so quickly and efficiently when prompted.
- [15:52]
A
Any thoughts on the liability there for the municipality? Like assume that they implemented an AI agent and you know, the vast, vast majority of the time it'll work well.
- [16:03]
C
Right.
- [16:03]
A
There are obviously certain edge cases where escalation might be needed. But if, as Galena mentioned, we have some dire consequences here. If a suit is brought, what can the agent, what can the municipality argue? And to what extent do you think responsibility would have been assessed against it?
- [16:22]
E
The, the unique aspect of municipal liability is that the rule rather than the exception is something called sovereign immunity. And for, you know, for decisions that are made structurally in a city, city planning, this is how we run our lights. This is how we do this or that. The city or the municipality is likely to argue that it is immune from, from, from any lawsuit. When the city makes an individual decision, an individual case, like not cutting a particular tree back that interrupted a line and caused a death, for example, then the city or its proxy, whatever that might be, may be liable. But that, that's always the, you know, the theme, the thread that runs through governments adopting autonomous systems or anything, frankly, is that it is, it is more likely than not to argue that it is immune from any kind of lawsuit. It's an executive function. It's made, it affects many people. It is not with respect to an individual. So like traditional, typical negligence, because this is how we hang our traffic lights or this is how we pave our roads in general. It always has that, that argument to make.
- [17:41]
D
Yeah, I wanted to come back, if you don't mind, and just comment on a couple of things you said about training data. It's almost imposs to train for every kind of scenario because they wouldn't be training data. A good example is 9, 11 and grounding every aircraft. Right. There was no such precedent before, so somebody made that decision. Now, it is not unprecedented for agentic AI to be at the point of taking all the Data and making a bold decision because that. Right, that's the whole creativity of current AI. On the other hand, would we want it to make it completely autonomously or to your point, with an escalation for approval? And by the way, in that, that point, I would say it is the developer's job to decide what constitutes a bold decision that is not just derived from the data in a regular fashion and where to escalate. I think that's important. And I think in this case the developer has a lot of responsibility on how they employ, deploy and program that. And of course, you're right, the municipality should check if that's available and make the right use of it. And certainly multiple agents talking to multiple agents is fine, a good idea. I'm wondering though, do you think in any way, either the drivers who are involved in all of these incidents because people do stupid things in bad situations and or potentially makers of autonomous vehicles that are also making decisions on the fly, they're basically agents in their own right. Do you think liability could be extended to a variety of sources at that point? Because really, the conglomeration of all of those is the cause of the problem, not just entirely the traffic system in its own right.
- [19:35]
E
It is likely to. I mean, different states deal with shared liability different ways, but more often than not, liability is shared among, you know, various potential tort fees. And then it will be ultimately, if this thing, if there's a case, if there's a lawsuit that's filed and it goes to a jury, the jury decides by what percentage. So that's very common. So it isn't. One should not look at who is liable as an either or situation. Often it's both. So where would liability for an individual driver? An individual driver is always subject to a duty of reasonable care. So if the roads are blocked and the driver decides that they're going to drive up the shoulder to try to, you know, get around the traffic, and then an emergency vehicle can't get through or they cause an accident, of course, then they are liable if they're stuck in the scrum. No gridlock, traffic, probably not an autonomous vehicle. If the autonomous vehicles just start circling a block aimlessly and mindlessly and it makes no sense because the system which controls them is confused if they're part of, say, one global taxi autonomous system, or then, yes, the developer may have a problem there because their system went haywire as soon as things looked strange or they didn't, or their autonomous vehicles did not recognize what was happening and they just crashed into a bunch of cars. So you know it, I hate to say it depends, but it always depends on who is the person who's in charge of the decision making, what went into the decision that they made? How did the end result relate to the decision that they made? Is this something that could have been preventable? Do they have a duty? Who do they have a duty to? Do they have a duty to the ultimate person who is stuck in traffic? Do they have a duty to the person that is riding in the emergency vehicle? And you, you kind of work your decision tree out from there. As far as the system developer that there, there's, there's obviously going to be scrutiny on the system developer. If the problem seems to bottom on causing all of the lights to turn red at the same time, causing, you know, certain streets to to be automatically closed incorrectly, then why did the system do that? And what decision making in the programming of this or the training models or the data that was fed into the system, what effect did any of that have on the ultimate actions by the system in reacting to the unforeseen circumstances on the ground? Sam.