Loading summary
Transcript91 lines
- [00:00]
Alex Jutka
Foreign.
- [00:04]
Santa Ruiz
Hi, I'm Santa Ruiz. You're listening to Statecraft. And today we've got a very special guest on Alex Jutka, deputy director of the Allegheny County Department of Human Services. Before you tune out, because that's a bunch of bureaucratic jargon. Don't do that. Because the Allegheny County Department of Human Services is, to my eye, one of the most interesting states, state level departments out there. Alex, thank you for coming on.
- [00:30]
Alex Jutka
Do you want to take that again? It's a county level department.
- [00:33]
Santa Ruiz
County level department. Yeah. Yeah. Here we go. Perfect.
- [00:38]
Alex Jutka
See, this is what state local. State is state. Local. You could say one of the most interesting. Interesting government entities. Well, no, I'm just. I'm just.
- [00:46]
Santa Ruiz
Period. Globally.
- [00:47]
Alex Jutka
Yeah, you were so. You were so close on till you had it. The thing is, now you get in your own head, so you just got to like. So.
- [00:54]
Santa Ruiz
That's right.
- [00:54]
Alex Jutka
You're good. No, you're a pro at this.
- [00:56]
Santa Ruiz
I'm. I'm learning by running a lot of reps, a lot of this, you know, a whole bunch of this.
- [01:01]
Alex Jutka
Exactly.
- [01:02]
Santa Ruiz
Maybe we'll leave this in, you know. Okay. Don't tune out. The Allegheny County Department of Human Services is one of the most interesting state and local departments out there in my estimation. Alex, thank you for coming on.
- [01:15]
Alex Jutka
Thanks for having me. I'm a big fan of IFP and Statecraft.
- [01:18]
Santa Ruiz
Oh, thank you. We wanted to have you on to talk about data at the state and local level. Will you say a little bit, before we get into this about just what you're proud of having worked on in the Allegheny County Department of Human Services from a data perspective? Yeah.
- [01:34]
Alex Jutka
So the Department of Human Services in Allegheny county has arguably the best integrated data system of any state or local agency in the country. And so we've had a two and a half decade sort of run of using integrated data that starts. Started with really taking a bunch of disparate government agencies that in many other jurisdictions are siloed and separate and putting them all under one umbrella, the Department of Human Services here in the county. And that spans operations from child welfare through homeless housing, through the behavioral health treatment system for Medicaid patients. And while many of those systems are separated and sort of siloed in other jurisdictions, they brought them under one umbrella agency. And that really created the initial conditions for integrated data. So by that I mean, suppose we see Santee Ruiz in the jail. How do we know that. That Santi Ruiz is the same Santi Ruiz that we're serving in child welfare or in the behavioral Health treatment system. And as you know, many of these systems really don't operate independent of one another. There are tremendous spillovers and cross coordination that needs to be done. And so the department really took data as a way not to do the type of research that, you know, people like me get really fired up about, but really for sort of meat and potatoes service coordination. And that was an awesome way to start to say, hey, look, you want these things not just for academic purposes, but really to make our clients lives better.
- [02:58]
Santa Ruiz
There's this saying in the tech industry, and maybe just in business generally, that you ship your org chart. You know, that the way your organization is structured is the way you actually produce the thing you're producing. Will you talk a little bit more about a, what's under the umbrella of the department of human Services, and B, what are other ways of structuring it? Why is this so interesting or unique? Lots of counties do it other ways, right?
- [03:25]
Alex Jutka
That's right. So we have five main areas within the department of human Services in Allegheny County. We have an aging office, an office of intellectual disability, an office that's focused on behavioral health treatment for Medicaid patients and the uninsured. So by behavioral health, by the way, for people who don't know, I mean substance abuse and mental health services, we have an office of community services that's focused on early childhood supports, family strengthening and homeless and housing services. And then finally we have a large child welfare operation that investigates claims of abuse and neglect against children.
- [04:00]
Santa Ruiz
And all this lives in one department. I think a lot of folks will listen to that and think, okay, cool, doesn't everybody do that? It's like, this is all bundled stuff.
- [04:08]
Alex Jutka
That's right. Solving the org design challenge here is just massive because what exists in many other jurisdictions might be a strict subset of that compilation of services where you sort of have individuals running these different departments. Maybe there are sort of fiefdoms where they don't sort of cooperate very well. Maybe there are difficulties for things that we care a lot about in terms of data sharing. So, for example, getting lawyers from different county departments or different local agencies on the same page could be quite difficult and intractable in other places that I've seen. And by just resolving that one sort of org design issue, you sort of set in motion much, much greater fostering of collaboration in ways that just really is challenged, I think, in many other jurisdictions.
- [04:55]
Santa Ruiz
Will you talk a little bit about the kinds of problems that you get in Allegheny County? And for listeners who don't know, Allegheny county is Pittsburgh. Pittsburgh, Pennsylvania and the surrounding environs. What do you guys face that's the same as plenty of other mid sized cities in the country? And what's unique or what stands out?
- [05:13]
Alex Jutka
I think that's one of the coolest parts about working in local government is while it feels like you're only serving a small, very delimited number of people, because the problems that you're facing are quite similar to problems that you're seeing across the country, there's an opportunity, I think, to really build evidence and solutions here that scale nationally. So you could have differences certainly in terms of the exact composition of refugee populations, the exact composition of the drug supply. Those things are going to be different across the country and those all present different challenges to different departments. But at the end of the day, you have many common features and many common challenges that I think make this job incredibly exciting. So, for example, figuring out how best to address child maltreatment, how to really effectively address serious mental illness, downtown homelessness, street encampments, those sorts of things are incredibly important and I think quite relevant as I look across the national landscape.
- [06:10]
Santa Ruiz
Will you just say a little bit more about the specifics though? I mean, to take the drug point, what, what's the kind of mix of substance abuses that you guys get in particular in Pittsburgh?
- [06:21]
Alex Jutka
Yes, Allegheny county, we have an overdose death rate that looks a whole lot like our peer counties as far as what's happening in Philadelphia, Baltimore, places that you would sort of suspect based off of our Rust Belt heritage and the fact that some of these problems in terms of the opioid epidemic specifically, really wash ashore in some of these communities quite early. So our rate doesn't look a whole lot different than many of those other jurisdictions. And so I think that the through line though, for many of the services that we run are those types of problems where you sort of think about them as confined to the behavioral health system, for example, or problems that are confined private individuals. But really they drive a tremendous amount of criminal justice involvement. So they represent a disproportionate share of people booked into the jail. They represent a disproportionate share of the people involved in child welfare, in the homeless housing situation that we've got. And so in many different settings, these things, I think, have common underlying features that make it important for us to really think about common drivers and how best to. To solve them.
- [07:30]
Santa Ruiz
Will you put some numbers on those? I've got a talk that you gave at AEI here in front of me and I'd Just love. More context from you. You say in this talk in Allegheny county, about a fifth of parents who are child welfare involved have a diagnosis of an opioid use disorder, and individuals with serial mental illnesses are hospitalized at rates similar to incarceration at state and federal prison. Will you give me a little bit more context on the kind of revolving door of services here?
- [07:58]
Alex Jutka
That's right. So in Allegheny county, which is a county of 1.2 million people, we get somewhere on the order of 15,000 referrals of child maltreatment to our child welfare hotline every year. Every year, about half of those will be screened in for investigation. And so if we look specifically among that group, you start to see very high rates of indications of addiction.
- [08:24]
Santa Ruiz
Will you just explain screened in versus screened out?
- [08:27]
Alex Jutka
Yeah. So in the child welfare system, there's a sort of a front door where referrals for children who are suspected of being maltreated are first processed. And there the initial decision that's facing the child welfare agencies is whether I should screen a child in for investigation or screen them out. If I screen them out, that means that I've sort of declined to investigate the family or the claim, and things just sort of go along, as you would suspect. But if I then, as the agency, choose to investigate that referral, I will eventually send out child welfare caseworkers who will go and conduct home visits. They'll call the person who made the referral to understand a little bit more details. We'll dig into some of our data to understand a little bit of the history of the family. And this is also where in Allegheny county we've been really pioneers for developing what's known as the Allegheny Family Screening Tool, which is a machine learning algorithm that sits on the front door of child welfare. Because you might be asking yourself at this point, well, with 15,000 referrals, how do people make informed decisions about which referrals to go out and investigate and which ones not? And it's an incredibly high stakes decision. And it can be difficult to the call screening staff because you might only see a little bit of an anecdote, a narrative. The person who made the referral might have just seen something that really sort of worried them, but might not have the full story on the family. And it's also really difficult for humans to think in terms of risk. It's just not a sort of thing that we're. We're quite native at. So there's a great book by Philip Tetlock called Expert Political Judgment that's on the sort of limited ability for intelligence analysts, for example, to beat out people who are generalist readers of the New York Times or simple algorithms. In terms of forecasting geopolitical risk, I think you face a very sort of similar set of circumstances here in child welfare. And so what the algorithm is trying to do is to predict over the next two years will concerns of maltreatment rise to the level such that a judge would order the child to be removed from the home. And so really sort of using that risk score as a way to think about where we should be allocating scarce caseworker resources and really where we have the greatest concerns is super important. And we take that machine learning information, those risk scores, and we pair it with the insights from the call screening staff. So that is, the model doesn't totally substitute for human decision making. They work in close tandem with one another to actually make more informed decisions overall.
- [11:09]
Santa Ruiz
So just to map this out, 15,000 referrals in a year is something like 40 a day. And you've got a few people at CPS who take these calls. And the calls are coming from some cross section of neighbors or the preschool teacher or someone on the street. Will you just flesh out for me? I'm on the hotline at CPS and I'm sitting in front of a terminal and I get a call from preschool teacher worried about a kid in their class. What happens next?
- [11:38]
Alex Jutka
Yeah, so typically what we will do is start to fill out information about the victim child and the alleged perpetrator to really sort of understand what are the dynamics here. So we'll often make calls back to the referral source to get more information, understand the story a little bit better. At that point we'll sort of understand whether we can put together sort of a complete picture of the, of the referral. And that really starts to then generate the, the downstream processes in terms of getting enough information to then make an informed decision on screening a child in or out.
- [12:18]
Santa Ruiz
And what's that process difference between how somebody working at CPS in Allegheny County.
- [12:25]
Alex Jutka
Yeah.
- [12:25]
Santa Ruiz
Would go through this and somebody in, I don't know, Idaho. Are folks at most counties not able to cross check this data to see if the parents been in jail recently, if they've been in the hospital recently?
- [12:36]
Alex Jutka
So my sense is that what's unique is maybe the sheer breadth of information that we have available to us. I don't know every jurisdiction in terms of their child welfare processes, exactly what information they'll all be looking at. I suspect many of them are going to do exactly what you were sort of talking about. But I think what's unique for us is just the sheer amount of information and the historical capture of that data. So for example, you could see educational information about the child, you could see across system involvement. You could see, for example, if there's an alleged perpetrator who themselves was in child welfare historically, you start to really sort of build out, I think, a very rich understanding about who the people that you're talking about on a given referral really are.
- [13:26]
Santa Ruiz
And just to clarify for me, the CPS folks on the end of the call, they're making their own call about like what's the risk to this kid? And at the same time the Allegheny family screening tool is generating its own score and that may or may not line up with the judgment of the person taking the call.
- [13:44]
Alex Jutka
That's exactly right. And so ultimately the discretionary exist with the staff, even when there's a sort of defaulted option that the child should be screened in or should be screened out that they can always override. So that always exists at their discretion. And I think it's an important sort of check set of check imbalances against, against algorithms where these things can really sort of complement each other. Because there's of course, things that the algorithm won't pick up that might be gleaned from contacts with referral sources, for example.
- [14:17]
Santa Ruiz
Can I get a couple examples from you? I'd love to get an example of when might a human screen in someone that the algorithm says low risk and when might a human screen out somebody who the algorithm is saying that seems like a high risk case.
- [14:31]
Alex Jutka
There can be a set of children who might be at high risk of adverse outcomes whose primary concern is in child maltreatments. So think about a youth, for example, who might be at risk of juvenile probation involvement, for example. That's the sort of thing where somebody might call child welfare about it. But unless we think that there's a child maltreatment risk at issue, we wouldn't want to screen that in.
- [14:59]
Santa Ruiz
They may be at risk of all kinds of stuff, but it's not danger in the home.
- [15:03]
Alex Jutka
Exactly. A low risk situation where you might want to screen in is that maybe the historical data around a family isn't pointing to strong risk factors, but the specific nature of the allegations could be quite substantial. And so you could think about ad hoc one off events where something terrible happens where there was, it sort of came out of the blue, for example, and those are why you have call screening staff especially to really do that. Due diligence and make sure that, that we're thinking about these things clearly.
- [15:36]
Santa Ruiz
And talk to me about the effect of using something like the Allegheny Family Screening tool. I mean, what's the pre post effect of using these algorithms?
- [15:45]
Alex Jutka
So we've had a couple of evaluations of this work done. I should say that just at the beginning, people might be skeptical about why. Is to removal from the home the right set of outcomes? Isn't that something that you'd be worried about, given that it's connected with all sorts of system involvement and it's the decision of a judge and that could be biased? And I think that those are the types of problems we've tried to think really deeply about. And so one of the initial studies that was done on the tool was really this kind of validation. So if you wanted to, for example, see how there are predictions of home removal measured against hospitalization for maltreatment, for example, we've sort of done that comparison. And those two predictions are very highly correlated. And so that gives us some comfort that this isn't just capturing this one set of harms that we think aren't really mapping to other ways to think about maltreatment. But these things seem highly correlated, which is great.
- [16:44]
Santa Ruiz
Will you just explain maybe at, you know, a fifth grade level. What are you measuring with that correlation? The fact that home removal is highly correlated with hospitalization for maltreatment. Give me that again in layman's terms.
- [16:56]
Alex Jutka
Yeah. So I think what you might be concerned about is that the concept of child maltreatment is sort of amorphous. So how do you actually measure that? And home removal is one very solid way because you have to go through a judicial process. The judge ultimately has to make that determination. They're making this determination based off of child welfare. But there are other ways that you could plausibly measure this. One other way would be, for example, if you show up to a hospital and you need to receive services for injuries that are consistent with maltreatment. And so that's what this comparison really got to is how does the prediction and the risk for individuals who are at high risk of home removal compared to their risk for showing up in the hospital due to maltreatment related injuries. And those things go together quite well. And that gives us some comfort that we're not just capturing some very idiosyncratic, very specific effect of the removal decision and the definition of the removal is the outcome, but rather sort of broadly capturing what we would think about as.
- [18:01]
Santa Ruiz
Child maltreatment because hospitalization for physical injuries is hard to introduce bias there.
- [18:07]
Alex Jutka
It's hard to disagree that it isn't some measure of something. It has its imperfections as well. But again, I think that we're trying to sort of argue that we shouldn't let the imperfect be or the, the perfect be the enemy, the good here. And rather let's just be our own best critic, let's take sensible approaches. And then as your question sort of asked earlier, you can get into the impact of these things, which I think is super exciting. So there's two different concerns with algorithms. One is how is the algorithm actually performing in terms of out of sample prediction? So that's always the proof point for algorithms is you want to know when the algorithm sees data that it hasn't seen before, that it's actually pretty accurate in terms of forecasting what's going to happen to those individuals or those referrals. So that's one test. I think we've got a lot of data that sort of gives us comfort there. But then the other thing is, relative to this counterfactual world in which the algorithm doesn't exist, did we actually improve children and family outcomes? And here we've got some very interesting new data by Katherine Rittenhouse and co authors that looks at the introduction of the AFST and finds that conditional and being screened in it eliminates the black white gap in removal rates. So this is an issue of racial bias that has been incredibly politically sensitive but incredibly salient to people working in child welfare, that these algorithmic tools could exacerbate those sorts of problems. And in fact, the researchers here found some quite comforting and quite encouraging evidence that these things are eliminating the black white gap.
- [19:45]
Santa Ruiz
This is really interesting and I want to just push a little harder because like you say, there was when these tools were first being rolled out in the Allegheny Family Screening tool was in the middle of the last decade, I think in 2016, there was all this debate over algorithmic bias and the idea was exactly what you're describing, that oh no, these tools are going to sample on. They're going to incorporate data that we already have as full of racial disparities and bias and it's going to aggravate whatever problems we already have in the sample. And so I just want to kind of double down on this because it's really striking exactly the opposite happened with this tool. You had less racial bias by this measurement in outcomes.
- [20:28]
Alex Jutka
And our department director now, Aaron Dalton, along with our then department director, Mark Turner, really spearheaded this initiative. And I think it's great that they were proven out by the data and that the team has been proven out by the data. But one of the things that really struck me coming into my role and stepping into Aaron's big shoes to fill in this position, was that they took incredible pains to, I think, really engage community and hear their concerns and understand how best to incorporate that into the tool. And they, I think, painstakingly went through the process of writing an rfp, getting out external researchers with impeccable credentials to come in and advise. They went through an ethical review. We've tried to make every analysis transparent that we've done on the tool. And so I think it's great to be sort of validated after the fact. But I think that Aaron and Mark and the team really went through the right process steps here, which were important to even earning the trust for community and from staff to implement this to begin with.
- [21:37]
Santa Ruiz
I feel like this is a useful case study for a broader dynamic that you kind of want to see more of in public policy, which is there was a very plausible story about how this could go wrong. Algorithmic bias was just very easy to kind of describe. It was the conventional wisdom in some ways about why these tools wouldn't work. It just wasn't true when you guys went with proper care and attention to detail.
- [22:02]
Alex Jutka
That's right. And so many of these things, I think, come down to process and are somewhat empirical questions, like there are certain circumstances under which building algorithms would be the wrong thing to do. But there, I think there is a playbook here that one could take and apply in different settings and give themselves that comfort. And I do think that that measurement after the fact is incredibly important because if you see, for example, that this is having adverse effects, then you then have an opportunity to actually iterate. And so much of what I think public, the public sectors really challenged by is that we don't have an objective function. Like if I'm an ice cream shop or whatever and I make terrible ice cream, I'm going to go out of business real quick. I'm going to figure that out very, very, very soon. In child welfare, that's not true. And so being able to really be transparent with ourselves about really crisp feedback loops in terms of what impact the decisions we're making are having in the community and on, on the children that we're responsible for from a maltreatment perspective, protection perspective is incredibly important. Yeah.
- [23:05]
Santa Ruiz
For those of us who did not study linear programming, what is an objective function?
- [23:11]
Alex Jutka
Right I would say, you know, it's the thing that you're trying to maximize or minimize within. Within any setting. So in business, what you're trying to do is to typically maximize something like free cash flow or profits. And that sort of crisp statement that provides the kind of the North Star vision for what you're trying to accomplish, or at least one plausible measurement of it then just really doesn't exist in the. In much of human services.
- [23:39]
Santa Ruiz
I'm looking at this report about Mark Cherna, who was the director of the Department of Human Services, and he had an independent ethics review conducted of this predictive program before it began. And it had this takeaway that I thought was well put, that it's not just that implementing the program is probably ethical, but also that not using it might be unethical. That is, if you have the most accurate predictive instrument available to you to assess risk for children, it's a strong argument that it's unethical not to use it in your work. In early 2023, there were a couple reports that then the Biden era Justice Department was looking at or making inquiries into the way this tool worked and the concern from some outside groups that was that it could result in discrimination against families with disabilities. How did you guys think about that risk?
- [24:34]
Alex Jutka
I won't comment on the specifics of the inquiry or the sort of, the nature of what is kind of a fluid back and forth with us in the Department of Justice. But in general, I think what people are concerned about is discrimination against individuals. So, for example, there have been cases in the past where people have been concerned about discrimination against individuals in recovery from being foster parents, for example. That's the sort of thing that I think the Department of Justice is probably concerned about here. And you know, on our side, the way that that manifests is things like what are the features that are feeding the model? So, for example, do you have things like diagnostic history and, you know, those sorts of things that I think could be sensitive to individuals in our setting? What I think we're interested in is thinking about all of that information together and what's actually predictive of risk. And there are certain sort of patterns of behavioral health utilization that, you know, are consistent with higher risk. It doesn't mean, for example, that these things are presumptive on any level, but there are patterns there that, you know, that the algorithm learns over time.
- [25:46]
Santa Ruiz
Sure. And I'm assuming even with the other example you gave of people in recovery being foster parents, there probably is a population level, a higher risk to the child. If your foster parents have, you know, had a lot of encounters with substance abuse in the recent past.
- [26:04]
Alex Jutka
Yeah, right. You know, I think that what they're concerned about is sort of broad, blanket sort of restrictions and rules rather than something in that case that would take advantage of like more nuanced information.
- [26:15]
Santa Ruiz
Sure. You guys built this tool, the Allegheny County Screening Tool, in house. It's software developed by the county rather than taking it off the shelf from elsewhere. I think if people are familiar with this sort of thing, they'll be familiar with outside contractors that the government is contrary to try to bring in. Talk to me about the decision to build it and to kind of sustain these systems internally.
- [26:36]
Alex Jutka
So we had competitively bid for and solicited for external help on developing the tool. But you're totally right that the tool is internal in the sense of using our data and being owned by the Department of Human Services. But we wanted, and where I think that there's often quite good value in terms of tapping in some external help. We wanted sort of the best available expert guidance that we could possibly have on developing this algorithm in child welfare in the right way. And so, you know, over time, I think what the department has really tried to put an emphasis on is internal state capacity building. So we have a 40 person analytics team, we have a 10 person data science team, we have a 10 person engineering team that's developing our own software now. And what I see often is that because there's a lack of skill complementarities between internal government folks and external advisors or consultants or vendors, you often get into situations in which timelines get really lengthened. The ambition can sometimes be incredibly lopsided in terms of coming from the outside versus the inside. That can lead to all sorts of problems. It can really result in, I think, not only much slower, but much lower quality software and data science and analytics being delivered because, for example, the agency hasn't taken control and adequately specified what its vision is, how it wants to use this, what the context is, what are the goals. And so I think it really puts government agencies often in a very lopsided position where it feels like an efficient use of government resources to say we're not going to hire internal staff, but in fact, it becomes costly in a million different ways.
- [28:14]
Santa Ruiz
Right, Let me change gears. I want to talk about something else that the Department of Human Services focuses on, and that's folks with serious mental health issues. These things interact, but you also have a totally different stream. I was really struck by the number of involuntary commitments every year in the US people involuntarily committed into psychiatric or medical treatment, 1 million commitments. I assume a lot of those are repeat customers. Who is being involuntarily committed. I'm assuming it's not 1 million people a year. I'm assuming the 8020 rule applies here a little bit too. But yeah, talk to me a little bit about the population that you guys work with and how that compares nationally.
- [28:50]
Alex Jutka
Yeah. So every state in the US Has a law in the books that is designed to help individuals who are at risk of harm to themselves or others of receiving treatment. And one of the things that we've really dug into over the last couple of years as a team is how to really think about what's going on with this group and how to really improve their health outcomes downstream. And so this has been one of my favorite projects because you very rarely get to see the end to end picture of how do we sort of describe what people are going through, how do we sort of think about underlying causal mechanisms and then how do we propose some solutions to the problem? So a couple of years ago, we sort of sat down as a team and we said, okay, great, we're going to take a deep dive into this data. And a side note here is that Allegheny county might be the only place in the entire country where you could pull this type of analysis off. But we, for example, really looked at what's going on in terms of this group's labor market outcomes. How is their medication adherence, what does their mortality rate look like? And one of the awesome things that this analysis facilitated was this, I think, quite disturbing finding that there's an 8% mortality rate for these individuals in Allegheny County a year after being examined for involuntary hospitalization. So just for comparison, for example, if we look at that against the mortality rate on an age and gender adjusted basis for people who are coming out of our jail or who are going into homeless shelters or the general population of low income individuals in the county, for example, this is a rate that just far exceeds that. And so while I think everybody really understands that individual with serious mental illness might have higher mortality rates, I think being able to put a specific number on it and to go to our partners, go to ourselves and say it's not some general conceptual problem that's going on, it's an 8% mortality rate, it looks like some pretty serious cancers in one year. What are we going to do about this? So that's really a credit to my teammate Pim Welly, who's our chief data scientist here. Really sort of fleshed all of that out. And then what you actually see in the data is this incredibly cool kind of natural experiment that's going on, which is that different clinicians view risk quite differently. So if you're a clinician, Santi, and I'm a clinician, we might have different propensities to actually involuntarily commit people. So you might be quite lenient and say, no, that's not. That doesn't read my bar. That's not that concerning. And I might be quite conservative and so really commit many people. Those differences across physicians really presents this really interesting opportunity because then we can not only study the fact that downstream people have poor outcomes, but we can then look at to what extent are these being caused by being upheld in an involuntary commitment. And this is a place where data, I think, matters a ton because of people who have diametrically opposed views on this, who are quite informed. So you have people who think involuntary hospitalization is terrible, it's incredibly disruptive, it's traumatic to the person the way that we do in the United States. Maybe it's not that effective. You have people who look at the other side of it and say, I really don't know how to get help for my loved one, or the police saying, I don't know how to get help for this individual other than psychiatry for psychiatric care. And so being able to provide evidence on that is something that we're stepping into now with not just pim, but a couple of our research partners, Natalia Emanuel and Val Bilotny. And what we're finding is that actual involuntary commitment for people that are sort of uncertain, where different clinicians could possibly make different decisions, judgments, seems to actually be exacerbating the negative outcomes that we're seeing. So it actually seems to be making things like the risk of harm to self or others increase, not decrease.
- [32:42]
Santa Ruiz
And just to be clear, this is for people who are in that kind of gray area category. One clinician might commit, one clinician might not.
- [32:48]
Alex Jutka
That's right. So you can think about the set of clients who are coming to the attention of an involuntary commitment petition as sort of falling into three buckets. One is people for sure, that you would never commit because they're. They're fine. You have people who are incredibly severe. And so no clinician is ever going to think that it would be a good idea not to hospitalize them. And then you have a group in the middle where there's genuine uncertainty and different clinicians might feel differently. And that's really the group that the team has been studying. And for that group, the way that involuntary hospitalization has been performed thus far does not seem to be benefiting clients in terms of health outcomes and criminal justice involvement. And that's incredibly, I think, important because then it poses a bunch of questions to us about how we can do this work better.
- [33:38]
Santa Ruiz
Yeah. Can you give me a sense of the scale? If I see a clinician about whether I should be involuntarily committed before you know anything about me, what are my odds of being committed?
- [33:49]
Alex Jutka
So, on average, the upheld rate for a petition is about 80%, but it varies significantly across the types of clinicians and the type of people who could be making that determination. So, for example, psychiatrists tend to uphold at a much lower rate. So that might make sense. They might be much more comfortable in terms of the types of risks that get presented to them of distinguishing between what's a substance use problem and what's serious mental illness. Whereas others who might not have that training might feel less comfortable, and so they might just uphold more of the petitions that they're seeing in a given year.
- [34:26]
Santa Ruiz
What other kinds of clinicians might see Me, a psychiatrist, or like an emergency.
- [34:31]
Alex Jutka
Room doctor, for example, who might not have specific psychiatric training, but who could be a site that actually sees the petition?
- [34:39]
Santa Ruiz
Got it. This 8020 rule seems to apply in so many places. I mean, your colleague Pim has a thread talking about how these people who are getting involuntarily committed make up a tiny fraction of Medicaid enrollees, but they account for 25% of behavioral health spending.
- [34:55]
Alex Jutka
Well, and it's fascinating because I think every jurisdiction in the country wants to know this frequent utilizer problem. And the way that they try to do that, to me, is very top down. They try to sort of say, what are the sort of the patterns of use where, you know, you're. You're frequently coming into contact with the emergency room or whatever. This was a very sort of organic way of coming across, across that group where we said, hey, for the 5,000 people in Allegheny county who are being involuntarily committed, they represent about 2% of our Medicaid population. They represent a quarter of our behavioral health spending on Medicaid. And so if you really want to bend the health care cost curve in behavioral health, you really have to figure out better solutions to that group, in addition to all of the human welfare concerns that you might have, just for the fact that their mortality and their criminal justice involved, for example.
- [35:45]
Santa Ruiz
I'm not going to ask you to solve the mental health problems in the country or even in the county entirely. Right, but what kinds of solutions do you look at? This is a population, like you said, it's like about 5,000 people who are soaking up a ton of the care that you guys provide. How do you think about that population?
- [35:59]
Alex Jutka
Well, and this is where I think that the example of going from problem identification, description, causal mechanisms to solution is super powerful. Because now you're starting to put all the pieces together. You're starting to say, this group is incredibly high acuity. The current way that we're doing involuntary hospitalization doesn't seem to be that beneficial. What can we do about it? And so, to the team's credit, we just started digging in and asking some questions about, like, what could be go. What could be going on? One thing that you see in the data that seems to be important here is that adherence to medication for individuals with bipolar disorder or schizophrenia, for example, which is a keystone to good care here, is just incredibly low.
- [36:41]
Santa Ruiz
It's like 20%, right?
- [36:42]
Alex Jutka
It's about 20% both before and after hospitalization. And so that's a really good indicator that these people aren't getting connected to care effectively either before or after. And so it presents a challenge to us in terms of what are the different mechanisms that could lead to higher adoption of behavioral health services. One thing that was really interesting at the time is we were reading through the sort of literature and thinking about what could be effective here. And the literature around medication adherence is incredibly bleak. It's all like, hey, this is very context specific. It works in this developing country. It doesn't work so well here in this richer country. But one thing that seems to be robust across jurisdictions is this idea of using financial incentives to encourage the adoption of different behaviors. And so here what we are thinking about is the fact that you can imagine if you're already in a situation where your health has deteriorated, maybe taking a daily medication is incredibly difficult. But what if I could give you a once monthly injectable form of that medication that makes you not have to do something 365 times, but just makes you do something 12 times? And what if I can tie financial incentives to that so that I can encourage you to show up? And what's really interesting, when you talk to individuals with schizoaffective disorder, for example, I had done this with a few of our peer support specialists that work at our psychiatric hospital down the street. And it's amazing to me how many of them said to me something like, hey, it took me eight years and three involuntary hospitalizations to realize that it was really important for me to keep on my medication. And even if I was going to gain 50 pounds of weight, for example, which is a side effect that sometimes comes with the medications, I had to make the adult decision to do that. And I think for us the call to arms was really how do we speed up that cycle? Because there is some sort of utilitarian cost benefit calculus that's going on there. So anything that we can do to shorten that gap and to make it to say, hey, look, this is a non pecuniary cost to you. It's costly to show up to, to an appointment, it's costly to take on the risks of side effects. It's painful to get a shot sometimes. It's not, it's not the thing that I think anybody looks forward to doing. But it's super important, we think to, we hypothesize to the stability of their care. And so what we're doing right now were funded generously through Stanford Impact Labs, where we're now getting to run that in a randomized controlled trial. And it'll be the first test of that type in the US and so coming out of it, we'll really understand how do people respond to financial incentives in terms of medication adherence? And then how does medication adherence then translate into better outcomes in terms of mortality, lower utilization of behavioral health system, better labor market outcomes and so on?
- [39:34]
Santa Ruiz
What would constitute success for you? Like what's the percentage of people taking their medication? That would be like a. Wow, this is amazing because I guessing that we're not getting 100% adherence to medication no matter what financial incentives you tie to this program.
- [39:48]
Alex Jutka
No, I think it's, that's an amazing question. So what most of the work that had been gone on in this space in Europe had sort of found, you know, treatment effects on the order of 10 to 15 percentage points. As far as adherence, we think that we might be starting with a more acute population. And so we're hopeful that these things could be even bigger. So I don't know, something like doubling the amount of people would be great. But I think this is the sort of problem where you have to pick the low hanging fruit. And to me, paying people is a low hanging fruit. And then there's people who yet still are resistant or for whom this treatment doesn't work super well. I don't think you therefore give up on them. But I also don't think that you're totally discouraged by the treatment. If that's what, what if that's what the incentives sort of show, I think you just sort of have to continue to iterate. And this is where I think rapid iteration cycles in government are sort of super critical. You can't just be on this timeline of like, I'm going to try one thing and then pack up and go home. It has to be sort of a way that you stack wins on top of each other.
- [40:56]
Santa Ruiz
Yesterday I spoke to a expert on policing. I'm not going to say his name because I think we'll publish this conversation before we publish his, but we were talking about a version of this 80, 20 problem where Jordan Neely was killed on the subway last year. Last year, I believe in New York had been on the city's, you know, 50 people most in need of help for a long time. And it just kind of highlighted the. There's a specific population of people and often you can really ID like it's these 5,000 people in the case you're talking about, or it's these 50 people. And then the challenge is just like, how do you deliver services or help those folks or make sure they don't hurt other people. But it's like a remarkably small population that you really need to focus on. And as you're saying, if you can get a 10 to 15% improvement or a doubling, close to half of those folks are now not off their meds all the time. That's a big win.
- [41:50]
Alex Jutka
That's right. And I think some of it is going to be that you have to pick the low hanging fruit and some of it is going to be that you have to think about other mechanisms to encourage adoption of care. So for example, one of the things that we're doing in Allegheny county is opting into the state's assisted outpatient treatment law, which is a way to get individuals who are deteriorating in their condition in community into care without needing to be involuntarily hospitalized first. So much of the way the behavioral treatment system often sort of takes action is once people have absolutely hit bottom. So think about somebody with substance use disorder who is seeking treatment for withdrawal management, for example, or who has lost their family, lost their job, and then they're willing to go into treatment. A similar thing happens with serious mental illness where people deteriorate to the point where they need to be involuntarily hospitalized. What if we could go upstream and offer support and care to individuals before that becomes an issue and leverage some potential legal options to assist individuals to really get treatment and stick on this. And New York City has been very innovative in this space. And I think it's something we have to continue to invest in and monitor.
- [43:02]
Santa Ruiz
The challenge for you, One of the challenges, at least, is what sociologist Peter Rossi called the iron law of evaluation. It was this famous 1987 paper and it's talking about the fact that social programs that are rigorously studied generally have disappointing results. The iron law is something like the expected value of any net impact assessment of any large scale social program is zero. How do you think about that reality that it's really hard to build really good, effective, measurable social programs?
- [43:32]
Alex Jutka
Yeah, I mean, one of my takeaways from working in human services for a few years has just been that if we took a little index card and we wrote down everything that we know works really well, we'd have a whole heck of a lot, lot of white space. So I think some people look at it at that and they are very discouraged by it. And I look at it especially in Allegheny county, with the leadership we have, with the data we have, with the partners in the community that we have. And I say, why can't we be the country's leading research and development lab for local government that allows us to really quickly build evidence on things. I think part of what that's getting to is that it's really actually quite difficult to even stand up things to test. It takes quite a long time. It's hard to do it at significant scale and efficacy. It's hard to find the right implementation partners. But I think that we really have a world class opportunity here in Pittsburgh to do that. And so the work that we're talking about with serious mental illness is one of them. We've got work in the pipeline around addiction where I think that there's opportunities for us to innovate both in terms of the medications that are being used for individuals with addictive diseases, as well as the way that we deliver those services. So I'm certainly not discouraged by that. I look at it and I say that's a great challenge. And it doesn't surprise me too. I think you have to really start digging. But I think all those failures also really help us to get closer to the solution by ruling out all the things that we know can't possibly be that important.
- [44:58]
Santa Ruiz
The Department of Human Services tackles a lot of these kind of famously difficult to deal with social problems. I'm curious just from your personal experience or your, your read of the things that you guys tackle getting people into housing, mental health, child Welfare, opiate addiction, other substance use disorders. Give me kind of a spectrum of which ones you think are the most tractable and which ones you're like, this is even harder than the other really hard stuff.
- [45:24]
Alex Jutka
I love working in human services because we really do work on some of the hardest problems on earth. I mean, they are incredibly difficult. Community violence is difficult. Child maltreatment is difficult. Addiction is difficult. So it's hard to give a rank ordering among the impossible. I'll say that the thing I have been, I think, particularly driven by increasing the rate of innovation in behavioral health, in part because I think that there's a large number of individuals who are suffering and there is such an opportunity here to do better. And if I look at, for example, the rate of new drug, novel drugs, for the rate of approval for novel drugs to treat addiction, for example, it's been 20 years since we've had a new drug approved for the treatment of alcohol use disorder. It's been 20 years since a drug is a new drug has been approved for opioid use disorder. There still is no FDA approved medication for individuals who are addicted to stimulants. And so all of the work that has been done more recently on this is, to me, felt maybe important, but incremental. So things like injectable formulations of existing medications. But in order to get out of the situation that we're in, I think we're going to have to innovate on treatment efficacy and treatment delivery, or we're at risk of just watching the addiction problem burn out here. And I don't think that that's something that I or anybody else who's working here at any other government agency or any other concerned citizen really wants to. Wants to see. I think that there's an opportunity really, to do much better and just to.
- [47:10]
Santa Ruiz
Define terms for folks, the addiction problem, to burn out. For that to happen, that means enough addicts die of. Of their addiction that it's no longer a problem that shows up on our numbers.
- [47:21]
Alex Jutka
Right. And I sort of think about this as what's happened in this country with alcohol use disorder, where it's sort of a background hum of suffering and despair that has tremendous cost to people and to society and to kids and on all sorts of levels. And it just. We haven't raised the federal alcohol tax in three decades. Not a whole lot of policy innovation. There's no drug innovation here. And I worry deeply about capitulation for opioids, even though it's been such a massive challenge. And you're Seeing I think some worrying signs that maybe this cycle is repeating with maybe less fatal ways, but with sports gambling. And so I think that those are, those are all really concerning areas. One other thing I'll say about behavioral health innovation that I think is super important here is, you know, the National Institutes of health funded a $350 million, 4 state, 67 county randomized control trial called the Communities that Heal project. And this was using the best evidence based strategies that the behavioral health experts think about when they think about combating opioid use disorder. And they found null results. So that is, they didn't find strong enough results that were distinguishable from zero. And even if you take their point estimate at face value, so even if you ignore the statistical uncertainty, you just take that number of face value and suggest something on the order of a 10 to 15% drop in opioid overdose deaths. So a win, we would take that. But certainly when you're thinking about really making a big impact on opioid overdose deaths, it's just clear that the strategies that we have been pursuing aren't good enough.
- [49:10]
Santa Ruiz
Right. You've been advising an organization, Casper. We co published something with them at our home institution IFP on an innovation agenda for addiction. And I just want to flag a couple projects. There's a phase 3 clinical trial testing semaglutide on alcohol use disorder with the va and there's an experiment in Pittsburgh. Maybe you want to tell us a little bit about that one.
- [49:31]
Alex Jutka
Yeah. So I've been really fortunate to partner with Nick Reville and his organization casper, which is the center for Addiction Science Policy and research. And he's, and he, he and his co founder Lindsay Holden have been total forces of nature here. So he wrote a really awesome substack post called Addiction Needs. And I think what was the title of it, it's something like we need an operation warp stage speed, but for addiction. And it was one of those things where like the magic of the Internet comes together when you read it and you say man, this is so deeply resonant with me. And I think he was sort of really responding to some of the nascent evidence around the so called GLP1 drugs for addiction. And so for people at home, the, the GLP1 drugs are things that become blockbuster hits such as Ozempic and Wegovy and Zepp Bound. So things that have been found to really curb food noise for people have also started to show signs that they could be curbing other sides of craving and addiction. And that's incredibly important. And so what I've been working with them on is exactly as you said, sort of the traditional phase three clinical trial that would happen in the VA health system and then more of a public health focused trial that would focus on introducing this drug for people in high acuity states. So think about individuals after they leave the jail with alcohol use disorder. This group has a 2 in 5 risk of going back to jail in a year. They're really having poor health outcomes. Addiction's a major driver of criminal justice involvement. If these things could be effective for that type of a population, that could be, you know, a total game changer for society, I think. And that's exactly what we want to get go after. So I think reasonable people can disagree in terms of their, their view on whether that would happen or not. But in my view, what's important is that we just go really fast and try to seize the opportunity as quickly as we can. And I think Pittsburgh, with our unique healthcare resources, plus the ability to really weave in administrative outcomes, something that's not really ever done in these sorts of clinical trials, could be incredibly important. So we're in the middle of talking to some great philanthropic partners who are interested in supporting us on that and we're hopeful about getting that off the ground this year.
- [51:49]
Santa Ruiz
There's obviously been a big federal level push for government efficiency in the news these past few weeks. And I wanted to ask you about how you guys have thought about the county level version of this problem. Allegheny, the Department of Human Services, runs the mental health housing system. And one of your colleagues wrote about how, you know, a few years ago was providing housing for around 500 people at a total annual cost of over 40 million. So that's over $80,000 a person per year for housing alone. And this is for people with serious mental health issues who are also getting often county funded treatment or other help from the county. It's obviously a crazy number. How did you guys think about bringing that down or getting delivering services for like a tenable cost?
- [52:31]
Alex Jutka
Well, I think one thing that's true about the department is that we always want to be scrutinizing the way that we're using our funding to make sure that it's going to the highest value uses. And one thing that can sometimes be true is that how services are priced over time just needs to be constantly revisited. Is this in line with actual costs? Are there ways to actually gain efficiencies by using Medicaid fee for services, for example? And this is one of those areas where I think because of the budget constraints that we face for those funding sources, we just had a high degree of interest in making sure that we were using that funding through its highest value use cases. And so this is an initiative that's been spearheaded by our director, Aaron Dalton. Really sort of thinking about what makes sense here as far as, right, sizing these contracts. How do we make sure that these individuals are receiving the right mental health care, that they're getting supportive services, that the staffing levels are right. But those sorts of decisions sometimes, I think, come with short term pain to providers as changes are made. But I think thinking about pricing efficiencies more broadly here is something that we're incredibly interested in doing. You do see really interesting incentives in some of these systems. So, for example, if you have homogenous flat pricing for a given service, but people differ in terms of their cost to serve, then you start to give providers incredibly strong incentives to take the lowest risk, least costly to serve individuals. And those aren't things that people do because they're nefarious or because they're trying to game the system. I think that they sort of organically happen over time if they're not kind of managed well. And that's where things like algorithms and tight management really come into play, I think.
- [54:19]
Santa Ruiz
Alex, you came in to public service from the outside, from the tech industry, and you came into a department that is already one of the best in the country. I would say from a data management perspective and from, you know, thinking from first principles about how you provide these services, I'm curious, what's the secret sauce? Tyler Cowen would ask, what's the Allegheny County Department of Human Services production function? You know, how does, how does it generate its results?
- [54:44]
Alex Jutka
That's right. I think it's been a combination of great infrastructure and partners and really strong culture, especially at the leadership level. So you can have the best data in the world and really great talent. But if your leadership isn't willing to take some risks, if you're not willing to be courageous in certain settings where things might not be politically popular, you won't make as much progress. And I think the department, through Aaron Dalton now and Mark Turner then, has really not just sort of thought about data in smart ways, but really thought about it in bold ways. So that's been incredibly empowering. And when I came into the role, I said, well, you guys have done this so well, why not double down on it, triple down on it? Why rely on anybody externally? Just hire world class researchers, world class engineers to come work for you. And so I've been blessed to sort of get that buy in and start to build out the team. And you know, PIM on the data science side has done an awesome job of taking that baton and building out a machine learning team plus economist, and they're really sort of speeding up the research production function. And then his counterpart, Rachel Silver on the technology side has built out an engineering team. And this is allowing us to do things like save cost from vendors while also building high quality software. So I think we'll always have this important relationship with external academics and external vendors in certain cases. But what we're finding is that we're really loving building on top of DHS's incredibly rich legacy with internal state capacity. And I think, just as Jen Palka has highlighted, at the federal level, these things matter a great deal at the state and local level. There's this concept for Justice Brandeis that state and local government is a laboratory of democracy. And this is where we can try and iterate on a whole lot of really important social problems. And so having that state capacity, I think, to actually execute on really important problems matters a great deal.
- [56:41]
Santa Ruiz
Let me close with something you mentioned just before we started recording, which was just that. Based on when predictive algorithms were introduced in these contexts in the early to mid 2010s, a neutral observer might have expected that they'd be far more common in all kinds of sectors. Not because you've had fantastic results and outcomes for kids and all these obvious clear wins where you're in some cases saving the lives of children. Right. So what's your read on why we don't see more adoption at the federal or the state and local level? And are you optimistic over the next 10 years that we'll get more?
- [57:17]
Alex Jutka
People often reach out to Allegheny County Department of Human Services because they want to talk about integrated data data and they want to ask my boss, Aaron, about how she built the data warehouse and how they pulled off all of these things. And what's so interesting to me is that very quickly people get into the technical details of how that's gone without asking what I think is a more fundamental question, which is what is it about her and the prior department leadership that's really enabled this to flourish? And so I think some of it is about the uniqueness of the data warehouse and having administrative data. Frankly, I think that that's often a poor excuse, though I think many jurisdictions could build totally capable data science and algorithmic operations based off of the resources that they have. And sometimes I think that there could be technical skill gaps there. But I think oftentimes these things come down to culture and desire, and we just have not had that at a large scale. And I think we're always interested here, the department of thinking about ways that we could be part of a movement to do things that are not just relevant to our clients here, but to solving real sort of national problems, including capacity building.
- [58:27]
Santa Ruiz
Well, I think the capacity building you guys are doing is great. Alex, thank you for joining.
- [58:31]
Alex Jutka
Thanks for having me.