B (3:12)

Yeah. So a good rule of thumb in science we have basically four tiers of research that we, we tend to talk about. Tier one, that's, that's the good stuff, that's the stuff you want. We call those Q1, Q2 journals that you've been vetted, everything's good there. Tier 2 is when you get into the bottom level journals, the Q3, Q4 journals. It's not wrong, it's just that's where you start to get into more of that predatory publishing. There's less oversight. We're not certain how good the peer review is. So you, you give it the benefit of the doubt as much as you can. But there's always kind of reservations like all right, tier three is when you just publish in complete non published like they're journals that exist but they're not indexed anywhere that we'll ever know. So that's basically you're definitely living, that's pay to publish. You're, you're living in Predatorsville now. No one's gon to really vet that. It's. Did you give me the money? We'll publish your stuff. And then tier four is kind of that, that extra stuff. That's where we talk about PhD dissertations, that's where we talk about conference proceedings, things that aren't right or wrong. They're just not ready for mainstream there we, we consider that like cool, that's, that's a great idea. Now get to work, do the research now.

A (4:24)

Got it.

B (4:24)

Okay, so I get it. So someone's talking about Iready and I'm like yeah, sure, it's been around since 2011. Must be really good. In all of my research I have been able to find four academically published papers, zero of them are in Tier 1. Two of them are in Tier 2, that lower quartile 3 and 4, and the other two are in unpublished journals. So if you kind of allow yourself to get rid of those unpublished journals, we have two pieces of data on I Ready in 15 years. The rest is just white papers, blog posts, there's some dissertations and stuff out there, but it's not stuff we would meaningfully accept as valid evidence. That is absolutely and utterly shocking to me. Now, even if that research that was there was great and said this tool works an absolute treat, which we'll see. It doesn't say that two papers. I published two papers accidentally last year. How do you only have two pieces of work on something that is worth billions of dollars? And so that alone to me is enough to just say, why would anyone even touch this? You've had, you've had ample time to prove your arguments and you've never actually taken the time to do it. So where do we go from here?

A (5:43)

Why do you think there's such a lack of research around this tool?

B (5:49)

I published something last week about Alpha schools, which is going to spin into the same basic story. I got so much pushback from them. Oh my Lord. I took it down. I couldn't, I couldn't put up with the, the backlash I was getting from that. No. Oh man, it's just too much. I got, I got my family to think about. I ain't got time for that stuff.

A (6:09)

Oh, wow.

B (6:10)

But it's, it was just interest is interesting to me how vehemently these people get behind something with zero evidence base. What I learned from the Alpha from talking to them is basically when you believe something really hardcore, you believe it really hardcore. There is no data in the world that can prove or disprove it. It's just 100% faith and you've already made up your mind. That's what I feel like the I Ready stuff is doing is they're running their show off of faith and they're asking us to believe their claims off of faith and they're asking us to like them well enough to believe that that's. Faith is a good enough standard to make educational decisions for our kids. Faith is a great standard in the church. Faith is not a great in public education when we're trying to determine how best to raise our children. So I, I think they, they had similar kind of idea. They wanted to make acolytes that would just fight for them, but they don't want anyone actually digging in because then they're going to find the gaps, they're going to find the decrepitude. They're going to find that this doesn't do what they're claiming it does. And it's real hard to sell a product when hard research says, oh, by the way, it's not actually working.

A (7:23)

Okay, and we'll get into that in a second. But so are you assuming what happened here is schools were fooled based off of the confidence that they put through and the small amount of data that they had. What do you think happened here?

B (7:37)

Absolutely. The sales pitches, the blog posts, these things. The one thing Big Tech is good at is spinning a narrative. They weave a wickedly compelling story. And when we don't know better, that story is really, really powerful. And so we'd be silly not to when, when we see the power of tech in our own lives and we're told your kids will fall behind in their lives and their careers and they won't be able to survive in this world unless we start to, of course I'm going to do it. And by the way, your teachers are all broken and we can standardize it and personalize it for every kid. Of course we're sold on these ideas. They sound magical, but that's the exact point. They're meant to sound magical. I, I need to know, are they magical? And almost exclusively, when you dig down, the answer is going to be no. There's no magic to any of this. It's just more sales pitch. And tech is good at that, man. That's what AI is the best example of this. How many more times are we going to buy the next big thing in AI without realizing that this is the same playbook we've been seeing for 20 years. If the Internet didn't change education and how kids learn and how we progress, if, uh, block coding in the cloud didn't change how we learn and how we progress, why would anything else change that? And they're so. They're trying to basically make us, trick us into thinking there's more of a problem than there is and that they've got the solution for us.

A (9:00)

Right? And I love that you called it Big Tech, because Ed Tech is big, Big Tech. In this case, like this is all. These are all the same companies doing the same predatory thing, right?

B (9:10)

None of them, none of them care about Ed. Ed was just go back to the very beginning. The very first Chromebook entered school not because they cared about school, but because they needed to recru coup money on a product that wasn't selling. It's by no means does any of this, is any of this aligned with education. It's just Big Tech's arm saying where else can we get customers? Where else can we make money?

A (9:30)

Oh my. Okay, so we, so fill us in on the truth about I Ready. Does it help kids learn?

B (9:40)

So of those four studies, let's, let's really just take a look at the two that were there. One took a look at iReady as a diagnostic tool and that's what it was originally intended to do. Just if you've got struggling students, put them through I Ready and we can tell you why they're struggling so that your teachers can now do stuff with them. It was only in the last gosh, probably seven years that they then built their own programs to say we're going to diagnose you and we can cure you. We're we, we do it. We're a one stop shop. So of the two papers that are worth looking at, one looks only at diagnostics, one looks only at learning the diagnostics. One says the quality of diagnostics from iReady is less than standardized state exams. If you just give your kid the exams that we're going to give them anyway, the data you get from that sorts them out into problems and solutions better than I Ready. And when you add I ready on top of it, it doesn't significantly change your outcome. So the one published study on the diagnostic aspect says it ain't good.

A (10:43)

How does I Ready spin that? They just don't do the comparison to the regular standardized testing.

B (10:48)

They exactly. This is where they spin this by running their own in house research. Basically if I run in house research I can show anything I want to. And one way they will do that is exactly that is what is your comparator. Your comparator is typically nothing is. So go back to the, to the next study that says when kids use I already they learn compared to kids who don't use I Ready. That sounds good. But you gotta remember we got time on task is the issue. Any kid who spends any time learning anything will learn more than a kid who spends no time learning that thing. Oh God. That's not due to the tool. That's just due to learning. If I, if you. I spend 20 minutes doing a Rubik's Cube and you don't I better be better at a Rubik's Cube. That's just time on task. And that's how they kind of spin this is. What's your comparator? When we compare I Ready to nothing, we see Boosts. That's the Saul Kahn experiment. That's the most of these tech. When we show that kids who use this 30 minutes a day, five days a week learn more compared to kids who spend 30 minutes a day, five days a week doing nothing. That's not a valid comparator. What you need to do if you want to prove your tool, and no one has ever done this with I ready or anyone else really is you need to compare the exact same pedagogy in two contexts so that the only difference is the digital tool. So what you really want to say is kids who spend 30 minutes doing adaptive math problems on my screen learn more than kids who spend 30 minutes a day doing adaptive math problems with a teacher guiding the adaptation. There's how you determine is my tool better than what we're typically doing? And in those instances, the answer is almost always going to be it's not better.

A (12:32)

Okay, so your hypothesis of that would be the analog version will be better.

B (12:37)

Yeah. Well, that's where now you can look at other research that compares analog pedagogies to the same digital pedagogies. And digital almost always underperforms analog. Because we're humans, we're built to learn from humans. The type of feedback a teacher can give us and the adaptability they have is so much more vast than the adaptability and feedback a program can give you. So, yeah, there's an interesting thing. Sorry to go on a tangent here.

A (13:01)

No, I'm trying to.

B (13:02)

One of the most powerful tools we have in education is something called formative assessment. And this is basically the foundation of I ready and all edtech. So we have summative assessment. And everyone knows summative assessment, that's your big final grade or your big final test. Summative assessment says you can't do anything else. This is it. This is your final snapshot. Go about your life. Okay, formative assessment then are those day in, day out tasks, like maybe the Friday quiz, maybe the question and answer we have in class that just kind of track our learning and growth as we go along. Now, we've known that formative assessment is one of the biggest drivers of learning. The more you do it, the more your kids are going to learn. And we always assumed the reason for that was the feedback that the kids got by continuously kind of assessing you and asking you questions that was feeding back into the student, letting them know where they stand. And that's why it's so powerful. Along comes something like a iready. We can ask kids a bunch of Questions, and we can give them feedback. We're doing formative assessment, right? Wrong. So Dylan William, the guy who largely coined the term formative assessment, he spent 10 years analyzing, why is formative assessment effective? What's actually making it work? And he fully believed, as did we all, it would be the feedback to the student. It wasn't the feedback to the student. It helps. It's not a bad thing. Go to town. But he found that the reason why formative assessment works is because of the feedback it gives the teacher. It's the teacher formulating and asking questions and then seeing how kids respond feeds back into the teacher and allows him or her to adapt on the fly in a way that then guides learning forward. So he no longer calls it formative assessment. He calls it responsive teaching. He says, I wish I never would have used the term form of assessment, because it seems like it's focused on the kid. It's actually focused on us. So this is where, if you were to use the exact same technique, here's a bunch of questions that are going to adapt to whether or not you get it right or wrong with a machine versus a human. The machine can give you the next question, harder or better, and it can tell you if you're right or wrong. It can't tell you why, can't tell you what you thought. It'll just say, yep, right or wrong, let's go harder or easier gives the kid feedback. But that's not what makes that practice powerful. What makes that practice powerful is the teacher getting the feedback and then adapting his or her following questions to what they know is actually going on in the classroom. So the dimensions along which humans can guide their instruction is so much vaster than what machines can do. So, yeah, by all means, if you compared something like I ready to a human instructor doing the same stuff the human instructor would almost always trump.

A (15:39)

Would the Iready salesperson be like, well, our technology is so advanced and we use AI to kind of formulate those questions based off of that feedback. In a better than a teacher could Irrelevant.

B (15:52)

No, because. So one of the first things you'll hear if a school starts to divest from technology and kids start to go writing back on paper, one of the first things you will hear in every single school is teachers go, oh, now I know what my kids are thinking. When you can actually see your kid working on a sheet, you can immediately tell where they're getting stuck, where they kind of stopped their thinking, what are they circling around. When you see the thinking, you know where the Problem is you can diagnose the problem. I Ready is almost all right or wrong. It can't tell you why you thought what you thought. It can only tell you did you get it right or did you get it wrong. And most of I ready in these other programs, the vast majority of what you do on them are going to be multiple choice or single input. Here's a question. Type in your answer. That's irrelevant. I need to know the process that led you to that answer. If I want to meaningfully diagnose where things are going right, where things are going wrong, that's the kind of stuff teachers pick up on. So long as the kids are doing analog work with them. When kids have a computer and are doing I ready and then the teacher gets their report, your kid got 80% correct means nothing to me. I don't know what that cool. I don't know what they were asked. I don't know what they do. It starts to go the other direction. So I think unless these programs start to demand basically the thought process involved in how you achieved your answer, it becomes really difficult for them to offer meaningful feedback in any or guide itself and adapt in any meaningful way other than correct or incorrect.

A (17:23)

I've heard from a lot of teachers and parents and students who are frustrated that they're required to do a certain number of minutes or per day or hours per week on I Ready. Did your research reveal why I Ready is so kind of strict about this time requirement?

B (17:42)

According to their research, they'd say that that's the minimum required to see the benefit they're talking about. But again, that's proprietary research. So what am I supposed to do with that? I gotta take that on faith. So the best I can say is they believe that's useful. Why they believe that, I could never really say. Now, I could make neurological arguments for why 30 minutes would be good if you were talking about training or drilling. Hell, I can make an argument why 10 minutes would be fine, would be probably enough. But I can't tell you why they're making that argument.

A (18:16)

Yeah, it just seems to be very frustrating to teachers, parents and students. And kids do get real frustrated with that process of going through the questions. And if they get it wrong, there is no feedback. There is no help to what they did wrong. And so there's a lot of outbursts and frustration. From what I've heard as.

B (18:34)

As there should be. I mean, one of my favorites. So. So one thing I Ready did do is they hired John Hopkins University to vet their tool. They paid John Hopkins to do research with their tool. And I give them all the credit in the world for doing that. God bless you for at least trying to put your money where your mouth is. The problem with that type of research is it's, it's all white paper. None of it is vetted, none of it is is peer reviewed. It's just basically I'm going to pay a researcher to do it and they're going to give me some data. We're going to publish it as a blog post. What did John Johns Hopkins find at 30 minutes a day at scale when people use it? No real growth on the reading stuff and 0.7% growth in math. That was their big findings. Congratulations. So when you extrapolate in the systems they use and the amount of kids they used, the school system was paying about $550,000 a year for kids to use. I ready to see no change in reading and a 0.7% change in math. That's $100,000 to see 0.1% change in kids math scores. Now if you gave me $100,000 to hire a new teacher and now class sizes are slightly smaller, even by one kid, I promise you you're going to see 0.1% growth in math. Hell, you give me 200 and allow me to hire two teachers, I'll bet I could give you more than 0.2% growth. And that's where it just. Even when they tried to put their money where their mouth is, it started to fail. And it's just these implementations and one of the things John Hopkins did say though is, is implementation is most people simply don't do it the way I ready says they should do it. And maybe that's impacting our results. Cool. If I invent a tool and I say the best way for it to work is you got to spin three times and stand on your head and no one does it. You could say my tool doesn't work because you're not doing it the way I said. Or it's possible that my tool simply doesn't work. So it's, it's, it's a possibility. But that's something research has to answer, not just hope.

A (20:33)

Right? Okay. So based off of your research and in your opinion, does I ready have a place in K through 12 schools?

B (20:41)

Nothing has a place in K 12 schools that doesn't demonstrate efficacy in my book. I, I'm not. So don't take that to mean no ed tech in schools. Take that to just mean that no. If something isn't going to improve learning compared to traditional methods. Why would we ever even debate it being in schools? So that's why, like with your law case, your lawsuit, I love it is we are having all these debates and lawsuits about privacy, about data, about student safety. And I'm 100% on board with all of that. But underpinning all of that is the promise that these tools boost learning. And I think genuinely there are a subset of parents, probably more than we believe that would be totally fine with their kids data being stolen, sold safety issues. They're fine with all that being bad so long as their kid is learning more. Take my kids data. If they're going to get a better grade, it was worth the price. And so that's why I think if we can come in and at least say, by the way, your one final hope that at least you're boosting learning, that doesn't appear to be holding now. What are they standing on? So not only are you stealing my kids data, making their lives less safe, you're also making sure they're not learning any better. So why are you here? Why do you exist? So by all means, if someone invents a tool that boosts learning, we can have a much better discussion. But in this instance, no, I don't see any reason for it to be there. It's just not proven effective.

A (22:01)

And you cut that contract and you have hundreds of thousands of dollars to hire new teachers which will actually help kids learn more.

B (22:07)

Funding and education is a zero sum game. Money that goes one where one place cannot go to another place. And so by all means, the money that goes into tech is money that's not going into teacher development, is not going into infrastructure. And that's where I'd say when I was talking with the school recently, they're like, we want to kind of divest from tech, we want to go back to analog, but where are we going to find the money to print things, to get notebooks, to get textbooks again? And I said, how much are you spending right now on. I ready right. A couple hundred thousand dollars a year. There it is.

A (22:38)

Stop.

B (22:38)

I ready. You get that money back, it comes back into your pocket, it's not gone. You get to, you get it back and you get to move it somewhere else. So by all means, yes. I think I'm on a, I don't know if you're on it. I'm on a chat with a bunch of schools about ed tech and someone was talking about that, how their, their district is about to fire three Teachers, which is going to increase class size to about 38. At the same time, they're going to invest over a million dollars in more digital computers for all their kids. And it's like, well, I mean, that's your choice. But you, you, you've basically told me you have the money to keep the teachers here. You're simply choosing not to because you think you are going to get better results by bringing in tech. And I'm here to say, where's that data behind that decision? I mean, you have to have something to make that decision meaningful. You have to prove that to me, they got nothing.

A (23:25)

Better results meaning what?

B (23:27)

I would assume better grades or test scores, but you won't be getting that from giving every kid an iPad. That's, that's one of the clearest things, is one to one computers ain't gonna boost scores. They're gonna lower scores pretty significantly. So like, that's where if somebody says, we're gonna spend $500,000 on iReady or another tool, I'd say, cool, why prove it? Where's your data? Like, if they can say, well, here we've got this data that shows this investment will lead to this kind of outcome, I'm like, cool, go to town. But no one can do that with I ready. So why are we even having the conversation for it to exist at this stage?

A (24:01)

So the research breakdown that you did is in a substack article, which is really well done. And parents can send that to their schools, teachers can send it to the administrators. I think it does a really good job poking holes in the research that we talked about today. Is there anything else you just recommend that parents or teachers do who are, are concerned about Iready or ed tech in general?

B (24:23)

Two things I just say as you go forward, just always say, where's the data? We're not trying to be buttheads. We're not asking for miracles either. I'm not trying to be a thorn in the side of schools. I'm just trying to say, why are you making the decisions you're making? And in any other field, we have to justify those decisions. And when people offer you platitudes, that's not evidence. When people say, well, kids have got to be future ready, cool. I've got data showing you that giving them a screen makes them less future ready. Do you have any evidence suggesting that it will make them more future ready? No, it just feels right. Well, that sorry feeling doesn't count. We're going to spend money on I ready. Cool. Why? Where's the data? Well, they published a blog saying it helps. Cool. Here are the only four papers ever published about it showing that it doesn't help. Mine trumps yours. Instead of trying to get into the weeds about ideology or what may or may not work, just keep returning to the data and say, what is the evidence? What's the justification for why you're making these choices? And that's where people will always get tripped up because they simply don't have that evidence. Or it's all proprietary and we just got to take it on faith. The other thing I'd say then too, about the whole movement is if you're a parent and you've been breathing in the last two months, something is happening. Yes, there is momentum coming from somewhere. I don't know where it happened, I don't know why it happened, but it's happening. So you are now, more than even three months ago, you are far better poised to speak up and have your voice heard. Because now, literally in January, if you went to your board meeting and said, I want to talk about ed tech, people would be like, some people won't even know what you meant by ed tech.

A (26:08)

Yeah.

B (26:09)

Today you could say in the last week alone, here's an article from the Atlantic, here's something in the New York Yorker, here's something from the New York Times. It is everywhere. So you are recognize you're not alone now. And now is probably the best time to, to stand up if you've got fears, doubts, the moment the ball is behind you on this one. So go for it. Now's the time to push.

A (26:28)

Yeah. And we had the LA USD pass a new resolution this week, limiting screen time, pushing screens completely out of K through 1, which is not enough, but a start. I agree that it's no longer like when I started asking questions a couple years ago and they're like, you're the first person to ever ask us this. Like, that's not where we're at anymore.

B (26:49)

You, you've got the world behind you pushing here. And that's. That's such a good point. The la. Like, I love that they. You're right. K through 1. A big question people ask me is about bans and stuff. Believe it or not. I don't, I don't. I don't remember ever saying the word ban. Except for when it comes to smartphones. I got no problem saying ban the heck out of those. Those things should have never been here anyway. But when it comes to ed tech, I got. I do not remember ever having Said the word ban. I guess I'm not against it, but when it comes to younger kids, I've never heard a valid argument for it. So when they talk about getting rid of K through one, they're like, are you okay with that? I'm like, heck yeah, dude. Before middle school, I have yet to hear a valid argument for why kids would need tech beyond typing and tracking. That's about as far as they need to go with that stuff. I can start to see different arguments beyond there, but that's where, like, a lot of, like, I ready and those programs are now getting younger and younger. They're like, we can teach kindergarteners how to read. So you know who else can teach kindergarteners how to read? Me. Leave me alone. Get out of my classroom. That's my job. I hope I. You would like me to do that as well with them. So that's where. Yeah, I. I'm glad that they're starting to say with the younger ages, it's just off the table, man. It's just not a question.

A (28:01)

No.

B (28:02)

Especially even when PD pediatricians are coming on saying, why would you give your kid a screen at all when they're a kid and then they go to school and they're like, here's a program, kid. Come on.

A (28:11)

Yeah, we all need to get on the same page. And I think finally we're getting close. So as far as supporting parents and everybody around iready, I'm going to include a link in the episode notes to your breakdown on your sub stack. I also created a parent teacher resource about the data privacy issues. I'll include that too, which has steps you can take, including requesting your child's data from Curriculum Associates, the parent company of iReady, which I think is a good first step. But there's lots that we can do in starting conversations with our schools. I know that's our already happening. I'm hearing from people everywhere. So I'm just so grateful to you, Jared, for this side of it. You know, I got the data privacy part handled, but we need to be pushing on the effectiveness of the product. I think those two things work hand in hand. And if it's not helping our kids, then why is it there? And this is such an important conversation to have with your school. And hopefully this conversation and these resources will help.

B (29:05)

Genuinely, thank you. You're doing the work. My side is easy. I just look at numbers. You're. You're on the ground pushing, figuring things out and getting work done. So thank you for what you're doing.

A (29:14)

Apparently it's not very easy because I already fooled schools everywhere with this research, so it.

B (29:20)

It is. It was stunningly easy. It was. The only hard thing was drawing a line in the sand. And after three days saying, okay, I'm not going to find anything more, the hardest part was stopping, because you just kept expecting there to be more and there just wasn't.

A (29:36)

Oh, my God. Okay. It's so important that everybody knows this. So, Jared, thank you, and we'll stay in touch on this and keep everybody posted.

B (29:44)

Thank you. Talk to you soon.