← Public Defenseless
Public Defenseless
505 | Will Automation Make it Harder to Keep Junk Science out of Court? w/Lisa Waters
00:56:09
Loading summary
Transcript30 lines
- [00:00]
A
Foreign.
- [00:05]
B
Welcome back to Public Defenseless with Hunter Parnell as we explore the rot in the criminal justice system and what we can do about it. Hey there, everybody. Welcome back to another fantastic episode. Today I was joined by Professor Lisa Waters. Lisa is a professor of law at CUNY School of Law. What are we talking about today? Well, folks, today's episode is about a law review article that Lisa published titled Automating Junk Science. And why did I want to bring this episode to you? What do I hope you take away from it? Well, this law review article really presents some alarming developments in the practice of criminal law, of criminal defense, and really brings to forefront some intractable problems that our legal system is gonna face as more and more automated tools for police and prosecutors become prevalent in criminal prosecution. What do I mean by that? Well, today we are talking about the junk science field of pattern matching, specifically related to forensic gun examinations. The type of forensic science that seeks to say, this bullet must have been fired from that gun because only that gun makes that kind of mark on this bullet. And we're going to talk about why this is a junk science, how it's not reliable, why it shouldn't be in courtrooms at all. But what I really want to focus on is the automation part, because pretty rapidly these automated technologies are being released out into the public without anybody actually verifying. If they simply replicate the problems that existed prior to automation, if they actually do what they are alleged to have done, and as you will hear today, oftentimes they're being sent out into the world without us having any knowledge of the underlying algorithms that allow them to run all of these problems are going to be incredibly difficult for public defenders to penetrate. As you're going to hear today, this might require us to learn a little bit about IP law, which is something that no criminal defense lawyer is really looking forward to. But I think this speaks to a bigger, broader issue. In order to dedicate the time and energy and resources to find when these automated tools are being used to find the new technology that police and prosecutors are implementing to make it easier to surveil, arrest, and prosecute people, public defenders need time. They need access to expert resources. They need access to the types of knowledge that can help them sort through these very difficult fields to understand. And for so many public defenders, the time and the access to experts, well, that's just not available in a lot of cases. So if you're a public defender leader out there and you're seeing the explosion of new technology in the ways that our clients are being prosecuted. I think we need to have considerations for how this impact impacts workloads and how this impacts our work and what we can do at a systemic level to ensure that that our line defenders have the resources and the time they need to get into the weeds on this stuff. Because if you don't get down into the weeds on this stuff, police and prosecutors are going to be able to use this technology to the detriment of our clients without any actual ability to determine if this technology is even reliable or if it's merely a digital version of the same junk science that we know should be nowhere near criminal courtroom. As always folks, if you're liking the show, like subscribe five star rating, five star review. Feel free to reach out to me on Twitter, Instagram, via email, or on Blue Sky. If you are liking what you're hearing and you would like to contribute to the ongoing costs of production for the Public Defenseless Podcast, you can go down below in the show notes to PayPal, Patreon, or Stripe, where you can contribute for as little as $5 a month or as much as you feel you can afford to give. As a reminder folks, I take no sponsorship money. I do no advertising reads, this show is funded predominantly from generous contributions of listeners like yourself and out of my own pockets. So if you're liking what you're hearing, you want to contribute to the cost of the show, I encourage you to go below and subscribe on Patreon today. If you do so, you will gain access to the video versions of the show. You will get the episodes a day early. And all that means. Well, folks, ordinarily I would be picking up beans. But if you have the video version of the show, you will see that young beans here, he is not too interested in being picked up. He has moved just off frame to ensure that I cannot grab him because so you got a brief glimpse of him down on the floor. But ordinarily I would be picking up my adorable co host, my miniature dots and puppy beans and showing him to the camera. But not today, folks. Not today. And now without anything more from me, let's get to the episode. Lisa, welcome to the Public Defenseless podcast. How are you doing today?
- [05:01]
A
I'm great. Thanks so much for having me on the show.
- [05:03]
B
Of course. Excited to talk with you. Always enjoy diving into the world of junk science because, you know, it just never gets any better. And the story just seems to be so familiar that you'd hope it would get better. But sadly we know it doesn't. So before we dive into the law review article. Let's go ahead and just do what we always do with our guest. If you could just introduce yourself a little bit, tell us a little bit about your career as a public defender, why you wanted to do public defense, and then we'll talk a little bit about this article because it's related to your career.
- [05:32]
A
Yeah, absolutely. So I kind of always wanted to be a public defender. And my motivation for doing the work evolved, got a lot more informed and nuanced as I learned more about the realities and the history of our court system and policing systems. But the truest reason I became a public defender, or what put me on the path to being a public defender, was that my dad was a public defender. And so I grew up just like, hearing the stories he would bring home of his clients and how they were treated and the cases. And I was just immediately drawn to the work and the, you know, the humanity and inhumanity in it and just felt this like, very deep, like, very, like, deep feeling like jail is bad, like prison is cruel, and this is my fight. So I went to law school to become a public defender. I was a public defender in New Jersey for many years, and now I'm teaching at CUNY School of Law. So I'm not a public defender. But I would say maybe without the stats to back it up, but that CUNY probably graduates and employs more public defenders than pretty much any law school. So it's a great place to be.
- [06:34]
B
Fantastic. And it's a part of your career where we want to start this story. So you. I think public defenders who have done felony level stuff, especially things that more involve firearms, are used to seeing pattern matching, bullet matching. They're familiar with it to some extent. But tell us about how you became really interested in this automation of this topic and then we'll go from there.
- [06:57]
A
Sure. So my article, Automating Junk Science, looks at something that I saw as a public defender and that we see throughout the criminal legal system as a trend in forensics. And that is this like increasing automation, this increasing introduction of automated tools into forensic methods and into forensic methods that are, you know, with or without automation, foundationally unscientific and unsound. And there's a lot of reasons to be concerned about that. But I got interested in this topic and exposed to this particular topic from one particular case that I worked on as a public defender. I was at the New Jersey office of the Public Defender for many years, and during that time I worked on a case on behalf of my client, David Giglioti. And the case did result in a published opinion. It's up on Westlaw NexisNexis. I'm not going to talk about any facts that are not in that opinion. But in this particular case, Mr. Giglioti was charged with murder based entirely on the conclusion of a firearm and toolmark examiner who used this automated tool called bullet tract. And so to kind of explain that, I'm going to give, I want to give like the shortest that I am capable version of this case, which is like, you know, bear with me because it's a fun case. So what happened in this case? You know, it started way back in 2004. There was a shooting fatality in New Jersey. A man was shot in the head and left on the side of the road in New Jersey. Did not survive. And with the investigation, there were no real like actionable leads that came out of it. No evidence pointed to a particular suspect. And so no one was arrested and charged and nothing happened. And then about A year later, Mr. Giglioti was arrested in for possession of a gun without the proper permit. Pled guilty to that. But unknown to him, there was this firearm examination and kind of like looking into that gun that he pled guilty to possessing in 2005 alongside bullets from the 2004 shooting. And the reason they did that is because this gun was like somehow traced back to the decedent's brother, which would sound suspicious except that Mr. Gigliotti was friends with them and it was just not that terribly unusual. But the result of that examination was actually like an exclusion, saying like, hey, this 2005 gun did not fire those 2004 homicide bullets. And so like no one ever really heard about that, that that examination took place at all for a decade. And so fast forwarding a decade in 2015, the Union County Prosecutor's office was doing I guess like a cold case review, pulled out this old file and the detective got like kind of like fixated on this firearms examination and asked their new by this point they had like an in house firearms and tool marks examination lab, an in house firearms and tool mark examiner asked him to like take a second look, take another look at this evidence. And so the examiner, Lieutenant Michael Sanford did that and he pulled out his microscope, looked at the evidence and couldn't really come to any conclusions one way or another. Kind of like sits on it. And then game changer is that this examiner goes to the association of Firearm and Toolmark Examiners AFB conference and that's where he learns about bullet track and so he learns about bullet tracks and these companion software called matchpoint, which are these technologies that use algorithms to carry out and kind of like assist with various stages of the firearms and toolmark examination process. And specifically by digital images, like computer generated images, not photographs, computer generated images of the surface of the bullets. And then there's these functions that allow the examiners to kind of like modify those images to, you know, in theory, like enable examination tricky cases based on various conditions. But like, what they do is they change, you know, they change what the images look like. So they use these tools. He set up like a meeting at their lab, brought the evidence, used bullet tracks, used Matchpoint to create and compare images from the evidence in this case, marked it up, and then did this very important final step, right? So Stanford uses bullet tracks, uses Matchpoint, creates these Images, is comparing 2004 homicide bullets with gun seized from Giglioti in 2005. But what he then does is returns to New Jersey, pulls out his old trusty microscope, looks at the actual bullets again, and it's only then that at least he writes down that he comes to the conclusion for the first time in the entire long history of this case that it's a match and says, this gun that we seized from Mr. Giglioti in 2005 fired those fatal bullets. And the reason they use that microscope step was to kind of try to like downplay or obscure the very central critical role that the bullet tracks and related technology played in this case. But in this particular case, like, it was just, like it was just too obvious that this was the determining factor. It was obvious to us. It was also obvious to the trial court, the appellate court, and even the examiner himself, like, eventually admitted that those images guided by like a gp. Yes, those were his words. And then, so it's based on that examination, though, that conclusion that Mr. Giglioti is charged with murder and nothing else had changed in the case, like it was just this and he was charged with murder, facing up the life in prison, held in jail on multimillion dollar bail. I joined the case kind of late and we got to work to try to litigate this firearms conclusion, of course, until really challenged the use of this technology and the way the method was carried out. And then what I found, along with the other attorneys on the case, was that we kept encountering all of these challenges and barriers in trying to find out information about the technology and trying to find a defense expert who knew about the technology and trying to find any other defense attorney in the country who had ever Encountered it and also trying to find any kind of, like, guidance in the law for, like, how we're supposed to deal with this kind of stuff. So, you know, we kind of, like, did the best. Did the best we could litigating it, but never actually got to see the litigation through because as often is the case, you know, as soon as things were, like, heading in a pretty exciting direction in court, a really favorable plea offer came through that allowed Mr. Giglioti to go home. And so that was the end of the case, but it was not the end of automated forensic technology. And so. And not the end of the kind of, like, barriers that are presented.
- [13:51]
B
Yeah, and very. It's so rare that because, you know, public defenders become law professors, they have a case that, like, influences them. But it's so rare that every single issue with the like field is present in an individual case. But every single step of the way, we have all of the potential problems, not just with the automation of this, but with the pattern matching family of junk science as a whole. So I think in order to understand why the automation is so problematic, why it creates a veneer of legitimacy, we first need to understand why pattern matching, first is just a broad category, is illegitimate. Then we'll talk about bullet matching specifically, and then we'll get to the automation. So let's just start broad stroke pattern matching generally. What is it? What are some examples of it, and why is it so problematic?
- [14:43]
A
Yeah, so pattern matching is, I mean, I think firearms and tool mark examination is like, the most common pattern matching technique we see in courts. But fingerprint identifications, bite mark, you know, not the most common, but perhaps the most infamous. Right. Like bite mark identification and what these fields do. And again, like, broad strokes, generally speaking, is that you're looking at, I mean, pattern map. Right. You're comparing patterns. Right. So, like, when you're talking about bullets or shell casings, you're looking at markings on one bullet. And to compare them with markings on another bullet, or if it's fingerprints, you're looking at, you know, patterns for one fingerprint compared with patterns for another fingerprint. And the examiners generally based on kind of, like, a lack of any objective metrics in their methodology and, like, very subjective and without any kind of, like, falsifiability or validity. These experts are just saying, like, do these two things look enough alike so that I can say they came from the same source? I like to, you know, I think of it kind of like a children's pattern match. You know, it's like, find the Match. Except in some ways it's like worse because in those games, like, you know, the right answer.
- [15:56]
B
The match exists. Yeah, the match exists.
- [15:58]
A
There's a match and you either get it right or wrong. But like with pattern matching, like we have no idea. Like there's no ground truth. So it's. Yeah, that's like the broad strokes. And then, you know, with firearms and tool marks examination is, you know, I do think it's kind of like the most commonly seen pattern matching technique in criminal courts. And the way that this plays out generally is that these firearms examiners who are usually like specially trained police or law enforcement, take bullets or cartridge casings that are left behind at like an evidence scene in some way. And then they have usually like a gun that they believe or want to, you know, come up with some, you know, or investigate whether perhaps it fired those projectiles. They shoot this gun in a laboratory setting to create test fired shots. And then they compare the test fired shots alongside the evidence projectiles under a microscope so that they can see these really tiny, what are called tool marks, which are. There's all these, you know, very fancy terminology for the different types of tool marks. But like, I kind of like to call it what it, like they're scratches and they're dense. Right. And compares, you know, based again like no objective metrics, subject, totally subjective, no testable, repeatable methodology. And then these examiners come to like one of the conclusions. Either these patterns, these markings are so similar that they must have come from the same source or that's like a positive identification or they're so different that they could not have, that would be negative or exclusion or I just like can't tell either way and that's an inconclusive. And so, you know, there's a lot of problems with this field, but basically like there is just not the data to support that these methods are reliable, that they're valid, that they're repeatable, that they're reproducible and we don't have the empirical data to support it. And just like there's also just specific to firearm and toolmark examinations, there's also this just like very foundational issue with it, which is that this entire discipline, like this entire idea of firearm and toolmark examinations, like even taking out the problems with the methods or lack thereof, but it rests on the assumption, this idea that guns create unique tool marks when they fire projectiles. So like every gun must create unique and distinct tool marks for us to be able to match the projectiles to that gun. But we Actually have no empirical support for that assumption. So like going to like, just the core basic assumptions that this entire discipline rests on. Like they are not founded in data at all, folks.
- [18:43]
B
To go through some of the other pattern matching problems. You know, we've talked a lot about bite mark evidence. If you listen to this show, you're probably familiar with it. But bite marks, it's the same exact underlying problem. The idea that every time an individual human bite bites anything, it just makes the same bite every single time. But we know that's not true because not every bite has the same force. You don't use the same teeth, you don't have the same points of contact. If you bite skin, skin deteriorates and skin starts to change how that bite mart might look because bodies heal or bodies decay, if that's a dead body. On shoe prints, we don't actually have a database of every single shoe print in the world. So we don't know how many types. What's often missing here is the denominator. So we might have, ah, yes, a fingerprint. Put that, that looks, that looks good. But we don't actually know how many fingerprints make that specific mark. We're guessing, we're estimating. And because we don't have a denominator, it's always just a guess or an estimate. And with fingerprints, you very rarely get a perfect fingerprint. Usually it's a partial fingerprint. So we don't know if the top part of the thumb is all you have. How many top parts of thumbs look exactly like that. Like these are all of the questions we should be asking. And yet we do not, because it seems very intuitive. Right. There is an intuition to go like, oh, a gun, well, it's all made of metal and it's fired out of the same chamber and it's got to make the same stuff. Right. But if you know things about firearms, the big, the big reason why I question this is every single time you fire a bullet through a chamber, it leaves debris, it leaves residue, it leaves gunpowder. That's why you have to clean your gun after every use. So every single time you put a bullet through the chamber, it leaves stuff in it. If you fire enough, if you're flying at a high rate or your, your barrel gets hot, that can change the shape of the barrel. There's so many variables that go into this. Not to mention, when that bullet hits something, is that going to maintain it? Exactly. There's so many things that you just can't get through it. And yet I feel like especially with guns, I think people are getting aware of bite marks. I think people are becoming more aware of things like shoe prints, but fingerprints and guns, that's tough to break people out of because I think it just has more of a intuition to it. Something we've been told, whether it's CSI or whatever, the media that has convinced people that these are a lot more legitimate than they are, it's difficult to break through. So let's talk about why the automation doesn't solve these problems. Because, Lisa, one of the issues you mentioned was subjectivity, right? And people, I think when they hear the word automation, they go, okay, so the humans aren't involved in this anymore. So if the humans aren't involved in this anymore, aren't we getting rid of this objectivity? Aren't the computers able to solve this problem? Lisa, tell them why they're wrong.
- [21:42]
A
Yeah, I mean, I think one of the things that's important to consider with automation is like, what is it changing about the methods and what is it not changing about methods? Right. And so when it comes to subjectivity and the role of, you know, the human role, humans play continue to play like a huge role in the firearms examination method. Right? Because, you know, these automated tools are controlled by computer programs that enable the algorithms to carry out their functions. And these computer programs are, or at least have been created by humans. I don't know, they're probably going to start being coded by AI, which will be a whole other thing that I'm. Maybe we'll get into that when we get there. But you know, these computer codes are written by human beings which are prone to error, but they're also coding based on like whatever data they have and based on the assumptions that are already built into this field. And what we know from like other areas of automated technology, such as facial recognition technology, is that like algorithms are. Can be loaded with bias based on the data that goes into them and that they're prone to error. And so. And that those errors and those biases are really hard to uncover and to identify because these codes are not revealed and they're hidden from disclosure. And so like, there's the human role at the coding stage. Then there's also like human. Humans are operating these systems, right? There's these kind of. There's kind of like a sliding scale of different types of technologies and different functions that these automated tools are playing and specific to firearms examinations. It's like there's 3D scanners that create the computer generated images. Then there's these filters or these functions that the human operator is going to apply. Right. So like in the Giglioti case, the human operators applied certain filters to change the lighting settings or in certain ways. And so these humans are making these subjective decisions about how to manipulate these images. So that's impacting the outcomes. And then the final way these are used with are with this like automated, actually fully automated comparisons. And we're not seeing that a lot these days. And then what we're seeing instead is that humans so far is that like human examiners are taking these kind of like automated computer generated images, using their like subjective decisions to like manipulate them and then they're using their like subjective opinions to like interpret them and give them value. And this is not even going into the kind of like background of kind of like these decision making about like who's being investigated in the first place. Right. And so we have all of these ways that subjectivity is just baked into these processes like with or without automation. And I don't know if the goal is that like we remove humans and like, I don't, I don't think that's the solution either. But I think we at least need to be honest about the fact that these methods are not taking the human factor out of it. They are not taking subjectivity out of it. They are kind of like giving this like false sense of objectivity and this like false sense of legitimacy when really like humans are so intricately involved in every stage of the process.
- [24:55]
B
Yeah, people hear automation and they don't understand. They have like this very futuristic sci fi view of automation of like this humanoid robot doing everything without a human and then just putting out an output and the human goes, is good moving forward. But that's not how this works in practice. And as you alluded to it still doesn't overcome the original sin, which is is this even. Is the key assumption true, the, the computer might be able to do this quicker than a human being, but it's still the computer and the human are both assuming that this is a valid method, that this is an actual thing that Guns and tools do. And it just doesn't appear to be factual based on all the evidence that we have. I want to talk a little bit about what you alluded to with the difficulty though of actually understanding how the computer is going to do its work. Because with the cop, right, for the people who's never done a trial, never been inside of a trial, normally we're going to do some sort of expert testimony. A cop gets called in, the person, the technician, whatever the person is, who is qualified on this. We then get to interrogate them about how they went about doing their pattern matching and we can point out some of the inconsistencies when they're up there. But Lisa, when it's automated and the person isn't the one making the call, how do we determine what's going on? Talk to us about the difficulty of piercing through the actual algorithm that is driving this.
- [26:26]
A
Yeah, sure. So, I mean, I feel like this is like the big battle, right, in litigation right now in terms of like getting the discovery is kind of like the big thing. And by discovery I mean like access to the computer codes that are carrying out the systems. Because when you have an automated forensic method or like one step of the method is automated, right? Even if the entire method isn't, what, in order to actually like, you know, confront, right, like or question or push back or you know, test the efficacy or reliability of that method, you have to know how it's being carried out. And so like, you know, like you said, like when it's a human being doing it, no matter how, you know, limited or whatever the method is, at least they can tell you what they did and you can, you know, see the process there. But when it's a computer algorithm, you don't see it. And the big issue here is, you know, we make these discovery demands and this is not unique to firearms like the DNA genotyping world kind of like was pioneers of this in pushing for access in discovery to the computer codes that are controlling the algorithmic processes. Because seeing those codes and not just seeing them, but then having an expert who knows what the heck they mean is the only way that we can actually see what is happening. And if we don't see what is happening and how conclusions are made, then how can we possibly test whether they were done in a correct or legitimate or valid or reliable way? But we're not getting those codes. Like, we're not getting those codes. And the reason we're not getting those codes is because these forensic companies, which are private forensic companies who market their technology exclusively to law enforcement agency, right? There's no other market for it, are not disclosing them. These are notoriously non transparent industries and they're really relying on intellectual property rights, actually like intellectual property law and trade secret rights to say like, hey, we don't have to disclose any of this stuff. We are legally protected because we have a financial interest in keeping this code, keeping these, this information private. And so what we're seeing in court, in These battles is defenders saying, like, we have to have access to these computer codes because these computer codes are generating incriminating evidence. They are witnesses against our client. So in order to challenge them, in order to confront them, we must have the computer code. You know, if we can't cross examine it in the traditional way, we can examine it, you know, through expert review. And that's a constitutional right. And then we have on the other side, you know, prosecutors often, it's sort of like mouthpieces for these forensic tech companies saying, you know, IP rights, IP rights, IP rights. But what's like. So these battles have been playing out in different, you know, fields of forensics with actually, like mixed results. But I think that there's like a lot more that can be done with that in terms of pushing back on these IP IP laws there. Like, I don't think we. I don't think this is a losing battle.
- [29:27]
B
I love this article because I think it's the first time that I've had somebody on the show recommend a criminal defense. Lawyers, they need to learn a little bit about IP law. Not a bunch, but a little bit. Not popular. But it is important to this. But, you know, I do, I do think that sidesteps a broader issue with the legal system is that public defenders shouldn't need to learn IP law. It might be beneficial, useful, helpful, all that stuff. But the Constitution is very clear of what the supreme law of the land is. And the right to confrontation is a part of the Constitution. So if you as a company decide that your technology is going to be sold and used in criminal prosecutions, I think you've just forfeited the right in a criminal prosecution to have that be protected. Now I think there's ways that we can still give some amount of protection. Obviously, the lawyers involved in the case are bound by confidentiality, so they can't disclose what they discover throughout this and that part of the hearing. Just make it closed. Like there's ways to do this, to offer some level of protection, to not completely disregard it. But at the end of the day, that should not be a defense for a corporation that has made a decision. Nobody put a gun to their head and said you have to sell your technology to cops in order for them to be able to prosecute people. Nobody made them do that. That's a business decision they made. And welcome to the cost of business. In the United States Constitution, the person has the right to confrontation. Unfortunately, we know that in a lot of jurisdiction, I'm sure that there are judges who are going to be far more sympathetic to people's IP rights than a person's right to confrontation. So we have to live in that world. There is another aspect of this. I think the most troubling part of this story is because everything that you described in that case involved the theory that we were going to have an opportunity to confront it. But one of the ways that cops do this with all kinds of technology is this parallel construction that you talked about. You didn't use the exact words, but tell us a little bit about how this can make it so difficult to actually identify when this is happening and potential things we could do about parallel construction.
- [31:48]
A
Yeah, so this like parallel reconstruction is this really, unfortunately, really effective investigative technique that has obscured a lot of what's actually happening in law enforcement and prosecutions. And it shows up like in different ways throughout police investigations, not just with forensics. This could be like in the fourth minute context, like there's an illegal search that turns up some like really juicy evidence. And so they're like, oh, we need to just like we know that we screwed this up. Like let's, let's do it the right way. And then they'll go back and do like a legal search, but that they would never would have done but for that illegal search. And so then they just talk about the legal search. And then what it really does is this kind of like very disingenuous erasure of the actual investigation. So in. But in the forensic context, with these novel technologies that have not really been meaningfully subjected to either scientific or legal review overwhelmingly the way this happens. And actually like what. Actually the Giglioti is like the perfect example. Right. So they used bullet tracks to do this examination. Like it's very clear, like if you have, you know, eyes and ears and a brain, like they did the examination with bullet tracks, which had never been litigated in court. And they also knew they weren't going to get the, there was going to be discovery issues. Right. And so then what they did is they did this, you know, parallel reconstruction by doing the microscopic method, which is, you know, itself super flawed, but they, you know, had been, has been accepted in courts for a very long time. And they did this subsequent microscopic method of firearms examination to kind of reconstruct the examination to come to this result that they wanted to get into court and by doing so tried to really like downplay or obscure the role of bullet track. And in most cases or many cases, they sort of are able to do this in a way where they don't even Talk about the technology at all. And defenders aren't even aware that it's being used. I think in this particular case, like, it was impossible. And also I think they were kind of, like, proud of, like, using fancier technology. And so, like, it wasn't totally hidden, but I think in many cases it is. And so this really. It creates this, like, really misleading kind of erasure of the actual investigation. And it also, you know, inhibits the discovery for defenders of, like, the information that's actually generating criminal evidence against their clients. And it prevents the meaningful litigation and adjudication around issues of discovery and admissibility when we have these emerging technologies. Because what actually we know is that these. Like, this case was not, like, the first time automation was used in firearms examination. You know, automated firearms examination tools have been used for decades by the federal government with their nibin, like, ballistic database. But they kind of, like, hide it in the earlier investigative stages and then add these layers of, like, human verification. And then what you get in discovery is, like, the human verification. But again, like, that would never have happened but for the use of technology. And, yeah, so it just is this, like, really troubling and again, like, really effective, unfortunately, technique that law enforcement uses to kind of, like, prevent revealing what is being used, how it's being used, the role it's playing, and to prevent, like, the protection of, you know, our clients who are, you know, the victims of these things.
- [35:17]
B
And it really plays into the prosecutorial advantage in the theater of a trial. So, like, I compare it to doing identification in court. You know, criminal defense lawyers, public defenders, we know that a lot of times the person on the stand has never, in fact, seen our client outside of the courtroom. That never happened. They don't know what they look like, but they know that the person sitting at the defendant's table is the person they're supposed to point at when they say id. And no matter how much we litigate, no matter how much we fight, no matter how much we try and convince the jury, hey, you've never identified this person until today. A lot of people assume, oh, yeah, well, they must have the right person. Right? They couldn't have gotten that part wrong.
- [36:11]
A
That's.
- [36:12]
B
That's got to be right. So they just take that identification at face value. We have the same exact problem here. Cop gets on the stand and says, yeah, I went through this pattern matching identification, and wouldn't you know it, I came out to this all on my own. People assume that the pattern matching is legitimate science. So once they hear that cop Explain that. Oh, yeah, I did this. Jurors aren't going to care if we point out that actually the only reason you took the time to do this is because there's a technology that also is illegitimate. Because they believe in the assumption that both the cop and the automation are operating under. Just like they believe the assumption that a cop who makes an ID in courtroom is under that of course they have the right person. So this really leans into the prosecutor understanding the assumptions of jurors tend to cut in their favor and that they. And without us having the smoking gun of, no, this is bullshit, we can't do much in that regard. So when you, or when you were thinking about confronting that officer, now that you, you know, you had the advantage, you knew, how were you thinking about it? Obviously you didn't get a chance to do it, but how are you thinking about attacking this to try and undercut that assumption?
- [37:27]
A
Yeah, I mean, we did get a chance to kind of like litigate some pretrial hearings and cross examine, but what we're really looking for here is keeping this out of court. And so, and I think that what. This isn't exactly an answer, but I think, like, what you're. These realities that you're speaking to really show the importance of the gatekeeping role of courts. Because we know, like, no matter how much science or logic we speak to a jury or anybody or ourselves, right? Like, yeah, we have this, like, gut impulse of like, or some, you know, desire to sort of, like, take things at their face value or like, you know, even me who is like, knows like, forensics are basically, it's basically all garbage. But I would get a case with forensics. And my first thought is kind of like, you know, yikes, this is hard. And so we have this kind of like, debt, like, somehow, like ingrained, like, deference to things that sound scientific and things that sound like technology and assume their legitimacy. And so, you know, I think that what this speaks to is like, the really huge importance of the, of litigating these, like, admissibility stages and pushing back on that. Because when we get in front of a jury, we haven't done it yet, right? Like, no one has done it like that. I'm aware of. No one has done it yet. And this is like one of the other barriers, the sort of like, that we didn't get to yet, which is sort of like, is this. If this does come in, like, it's so obvious to us that this is garbage and that this is, should not be relied on. But what does that sound like to a juror. Right, like to a juror who like maybe works on their computer all day and is like so is very familiar and trusting of technology and of science and of law enforcement. And we don't have guidance in our legal rules for jurors on that either. We don't have any jury instructions we can turn to to say like, hey jurors, look, the judge says, like you have to consider, you know, automation bias. Like you have, that's a, that's a phenomenon and you have to consider potential for error. You have to consider algorithmic bias. You have to consider, you know, machine error and you know, calibration. We don't have any of those tools that we can point to. So I think what we're left with is sort of like, yeah, not a lot of guidance and no, like, you know, playbook and a system that has really like erected all of these barriers along the way to meaningful litigation. Of that, you know, like we just. The system is not set up to enable meaningful pre trial litigation. It's not set up to enable meaningful trial litigation. It's not set up to actually grapple with issues of science or technology.
- [40:11]
B
Yeah. And the core part you talked about, the gatekeeping function. I find it odd that with everything we now know about forensic evidence being as bad as it is, which we have pretty conclusive evidence, that really outside of DNA, this is not that great or replicable valid that the way we do clerkships for judges don't include people who aren't lawyers. Now that Bruin is a part of our world, there should probably be some historians who clerk with judges because we're having to do more historical analysis. That is not something judges should be doing. It's not. There's something they're equipped to do. And I can say in lower level trial courts, like say a Munich, some judges just don't give a shit about that stuff. It's the same with science. Right. As much as lawyers like to tell ourselves that we are the masters of the universe and there is no problem that we can't educate ourselves on. Folks, we went to law school for a reason. That reason is that we are not scientists. So it just makes sense that on staffs, in courthouses with judges is somebody who can help them understand these issues because we end up in a situation where our filtering mechanisms to qualify an expert or to get a. Is this technology valid? It's up to a judge who, you know, this judge might have been a criminal lawyer, but if they're an older judge They've been on the bench for 20, 25 years. They didn't deal with this stuff when they were a criminal lawyer. That's useless experience. Even if they are a judge. All the ways that you talked about how things are hidden from us, well, they're hidden from the judge too. Right. If they never get an opportunity to litigate this, if it's never brought before them, they haven't learned anything about it. And not to knock judges, but a lot of them are keeping up to date with evolutions in the scientific field. So one of the structural ways, and I surprisingly, I think this actually might be the hardest one, is just getting other experts involved in the court process to help the judge. Because we should not continue to make the false presumption that a judge can just figure all this out because they're around it enough. They're not. They don't see what's going on outside of courtrooms. They don't have access to the algorithms. Even if they did, what is a judge going to know what it's looking at? No, it's not. So having better access to those types of experts, I think could only help. What can we do about the discovery issues? How can we attack this issue to make it so? Because really, what we got to do is balance the rules in court a little bit better because they favor the prosecution so much with their ability to hold on to the stuff and to hide it and to go guard it, that we never get it. What could possibly change to make this a little bit more effective on our end?
- [43:05]
A
Sure. I mean, I actually think that the, you know, the, the cleanest way to deal with the, just the discovery rule, it would be at like the legislative level. And there actually was a, or that, I guess it is still like languishing somewhere. A bill put forward, the justice and Forensic Algorithms act put forward in 2024 with, you know, hasn't, hasn't been, you know, no action on it. But one of the things that this act would do, proposed to do, would be actually prohibiting the application of trade secrets rights from disallowing defense access to source codes when the codes are used in criminal cases against their clients. And just to say, like, you just can't rely kind of like what you were saying, or like, if you're going to use algorithms to generate evidence that's going to be used in a criminal case, you cannot rely on trade secrets rights to shield that from disclosure. So I actually think that, like, that is the cleanest way to do it because what we do, you know, what we end up with instead is, yeah, these individual judges who, like, of course, are not scientifically trained, they're legally trained to deal with legal issues which are very distinct both in nature and in kind of like the way you go about them from scientific questions, right. Like, in science, you know, I'm told. But like, in science, you know, when you're. When you're trying to find answers to something you like, you're not going in an adversarial system with, like, an agenda, right? You're going into these to kind of like test a hypothesis in theory, right. Like, if you're doing science right, you, like, have a hypothesis that is falsifiable, and then you test that hypothesis in a number of different ways to determine, like, whether it is valid or not. And you're doing it for the sake of, like, discovery and truth and knowledge and information. And I may be like, romanticizing science there. I'm sure there's lots of issues in it, too. But in law, right, like in these issues that are before a judge in court, you have two adversaries who may be bringing in experts, right? Like, maybe you're bringing in. Maybe there's bringing in, you know, a prosecution expert and there may be bringing a defense expert. And those experts may be very knowledgeable on the scientific issues, but they're also going to be framing that information from the perspective that the party presenting them wants them to be. You know, just some. Not like they're telling them what to say, but they're framing the arguments to an end. That is not like the pursuit of scientific discovery. Right?
- [45:28]
B
Yeah. We didn't call that expert because they're going to get on the stand and disagree with our position. Like, we called that expert because they agree with the proposition that we're putting forward.
- [45:37]
A
Yeah, totally. And so, like, you know, maybe there's this idea, like, maybe judges should have, like, should judges have, like, you know, independent experts or something like that, or should there be, you know, scientific advisors to the bench to make these decisions? And I think that, like, this is something like I don't have a clean answer to because I feel like any idea that's bounced around, I'm like, immediately like, oh, this sounds dangerous. Like, who are these scientists? Where are they coming from? Are they going to be, you know, like, you know, government appointees that are, you know, shaped by the administration that appoints them. Are they going to be beholden to the court in some way? But I think that it's, like, worth, like, when we're in a system where like, things are not working well. Like, this is when we should be dreaming and experimenting and kind of exploring what are the other ways to do it. And, like, maybe we. It's not like, things are going so great now that, like, trying something different is hugely risky. So, you know, I think that, like, the idea of incorporating scientists into, you know, a courtroom could provide, you know, more, you know, information. It could also provide, you know, the risk of further kind of, like, legitimizing something that's not legitimate. So I do think it's worth kind of, like, elevating those concerns and having, like, a conversation. But I don't think that, like, those risks should hold back the legal community from, like, exploring new ways to do something when the ways we're doing them aren't working. And I also think that perhaps, like, we shouldn't even be getting to court. Like, we shouldn't even be in court in most of these forensic methods, because, like, there is not scientific. There's just, like, not scientific support for their validity and reliability at all. And, like, we should have, like, earlier, some kind of, like, earlier screening processes, whether that be through, you know, mandatory standards or, you know, regulation. Right. Like regulation of these industries and the way that we regulate algorithms in, for example, like, banking and the financial industry, like, those are federally regulated. But when it comes to people's liberty, like, algorithms, anything goes. So, like, from regulation to, you know, other areas of legislation to kind of, you know, I also think, like, you know, maybe we don't have enough confidence then, but we should be turning to prosecutors to say, like, hey, you actually, you have a huge obligation here to not use tools that have not been established to be scientifically valid.
- [47:54]
B
If prosecutors were unable to use things like snitches or paid informants or paid witnesses or all this junk science, they would turn around and tell you, well, Lisa, how could I ever prosecute anybody? And the answer is. That's the point, buddy. That's the point, is that our legal system, this gets at the heart of this issue, is that there is no coincidence between the rise of mass incarceration and the rise of junk science in the legal system system, they go hand in hand in glove. That one fueled the other. That as time went along and as, I think, as defense lawyers became more professionalized, right, you get. Gideon, you get a little bit more institutional offices around the country, you get a little more formal training. As we start to be able to better combat the old methods of evidence, we get new methods that are terrible and very difficult for us to attack. And they create a situation where your client was in where, you know, prosecutors know this. Once I get the charge, I'm probably going to get a plea. So why do I need to worry about the rules of evidence? Why do I need to worry about what's coming into court? All I need to do is get that probable cause. Once I get that probable cause, you're probably going to plea. I'll make you a sweet deal. Right, like the one that you offered. I have no idea what the, what the specifics of it are, but you said he gets to go home. That especially in high level cases, we think about wrongful convictions, they bring the case back and then what does the prosecutor do? Oh, they don't promise they're not going to prosecute you any further, but they say if you agree to plead guilty, we'll let you go. A credit for time, sir. So they win. At the end of the day, this is a very deeply structural problem in the legal system, is that they became so focused on giving prosecutors the ability to prosecute people that they didn't care how they did it. So we ended up with all of this terrible science. We ended up with all of these coercive powers in the plea bargaining arena and they all go together. And it requires a big cultural shift in America. One to have a system set up that can deal with cases where something happened, but it could never be proven in a criminal court. We think of that as the civil legal system. But they have also made barriers, like incredible barriers to that. It's probably gotta be something that doesn't involve lawyers because once lawyers get involved, because we have adversarial system all the way through, both parties are going to lobby to make it more beneficial for their side. And on, you know, in the civil side, we know that large corporations have lobbied to make it more difficult for plaintiffs to get into court. And on the criminal side, we know prosecutors have lobbied to make it easier for them to prosecute people. So you probably need a non legal system to deal with issues. But because we don't have one, the political will has shaped our criminal legal system into something like, well, of course we need this evidence. If we don't, how will we ever hold people accountable? Missing the fact that maybe, just maybe that person didn't actually do what they're accused of doing. So we end up with just mountains and mountains and mountains of bad evidence. And I do agree with you. I think it should just be legislative bars. And I feel like it should be pretty easy. You get a bunch of experts in A room. Say they write a report. Say they write two reports, both commissioned by the president, and those reports come out and they say that this is bad evidence. You can't. You shouldn't use it. Legislator passed the law. Hey, until a scientific body reaches a consensus that this is valid, repeatable, falsifiable, all that stuff can't be used as evidence in court. It could actually be that simple. But we know exactly what that would look like, right? The political campaign behind that. Oh, you want people to get away with crimes? You look. And do politicians have the courage to do anything about that? Who knows? In today's society, at least very clearly, no. Okay, well, I think that's about all that I've got. Do you have any other parting shots for the defenders out there who might be listening about creative methods to attack this or other similar technologies that might be automating these forensic junk sciences?
- [52:08]
A
Yeah, I mean, I think basically just, like, we as like, a defender community need to be in communication, right? Like, in close communication about, like, what we're seeing and kind of like, strategizing how to, you know, challenge and confront these new technologies as we see them. Because we know that, like, the other side, like these forensic companies and law enforcement, like, they're being strategic, right? They're working together, they're being strategic. And we have these, like, individual, like, phenomenal public defenders all over the country who are seeing these individual cases. These different technologies come up in different ways and finding ways to, like, come together and strategize and kind of like, I think is the only effective way. And then I guess just finally, I'd say, you know, I always think when I think about, like, these forensic methods, there's kind of like these two distinct kind of like, big picture questions, right? Like, one is like, are these methods reliable and valid? And, like, no, they're not. But also there's a separate question of sort of, like, even if we're moving in a direction of, like, bringing more science in, becoming more accurate and more, you know, reliable, like, should we be using these technologies in the way that we are at all? And, like, I do feel like we're in a moment where with this kind of, like, mass surveillance, that maybe people can become, like, a little bit more open to the conversation about, like, you know, the. The use of forensics, whether we should be using these technologies and forensics to police and surveil and prosecute people the way that we are. So I do think that there's a. It's like, not just for me, about, like, get the science better. It's like, let's think about like whether we should. But yeah. So I hope, you know, I hope like that some tiny bit of my article or I also have an article in the Champion that's kind of more specifically aimed for public defender audience on the firearms technology called Automating Firearms Examination. I hope that there's like some way that could be helpful to a defender somewhere. And I'm happy to speak with any listener who wants to reach out. But thank you so much.
- [54:03]
B
Yeah, thank you very much. I really appreciate it. And you know, public defenders have one political advantage here. We are living in a time of increasing mistrust in big technology companies and we are in arguably the lowest trust in government, period that America's ever had. And it's big tech companies partnered with government to use these in a criminal prosecution. I think the message is there, it's just, will public defenders be able to get that across to people? That's the million dollar question. Lisa, thank you so much for joining me today. Where can people contact you, learn more, read, write, where would you like direct people towards?
- [54:40]
A
Sure. Again, I teach at cuny. That's the City University of New York School of Law. My contact information is up on the FAF faculty website there. My automating junk science is up on my SSRN page and it'll be up on Westlaw and Lexus Nexus. And you can reach me at Lisa Watersaw, cuny Edu. And thanks again.
- [55:02]
B
Fantastic. Well, thank you so much, Lisa, for joining me and hope to hear from you soon. Thanks again everybody for tuning in today. If you like today's episode and you're liking the show, please remember to like, subscribe, leave a rating, leave a review, and share with anyone who you might think is interested in learning more about our rotten criminal legal system. If you would like to contact me with any questions, concerns, feedback or potential guest recommendations, you can go to publicdefenselessmail.com and shoot me an email. You can also find more about the show at publicdefenseless.com or on Instagram at Public Defenseless Podcast or or on Twitter Defenseless Pod. As always folks, thank you again for your support. If you have the means to do so, it would be great if you could go down to the Show Notes, go to Patreon and subscribe for as little as $5 a month where you can get access to the show a day early and the video versions of the show. Again, thank you all so much for the support and hope to see you soon.