← ASU+GSV Summit Sessions
ASU+GSV Summit Sessions
The Design Gap: Why “Good Enough” Consumer Tech Isn’t Good Enough for Schools
00:40:56
Loading summary
Transcript120 lines
- [00:03]
A
This session was recorded live at the 2026 asu gsv summit in san diego.
- [00:14]
B
Okay. Welcome. It's 10:10. So the rule is we have to start on time and end on time. And as a former teacher, I appreciate that. I might even let you guys out a little early for recess. Joking. Well, good morning. I'm Erin Mote. I'm the CEO of Innovate. Edu. We are a house of brands, not a branded house. So we run large scale, multi stakeholder alliances throughout the K12 and higher ed ecosystem, including the Ed Safe AI Alliance, Project Unicorn, and a number of others. And I'm super pumped to be here today to talk about the clarion call that we are hearing from for evidence for efficacy and really to distinguish between consumer tech and ed tech. I'm gonna have my panelists briefly go down the line and introduce themselves. And when you do, I want you to tell me, what's your favorite ice cream flavor? Just one. Okay, go ahead.
- [01:13]
A
All right. Well, hi, everybody. I'm Jamie Rafel. I'm the Chief Product Officer of clever. Clever is a platform that supports access and integrations and supports software in schools across the US and perhaps more importantly, my favorite ice cream flavor is when you get the really good dark chocolate and there's some chili pepper in there. It's a rare one, but when you find it, it is the best.
- [01:40]
B
Does Talente have one of those? Yeah, I thought so.
- [01:44]
C
That sounds really good. My name is Christian Pantel. I'm Chief product officer at D2L. We are a large learning management system or learning platform. I've been with the company for 11 years now. We are based in Canada, but are present throughout the world. In terms of favorite ice cream, I will go with salted caramel.
- [02:07]
B
Okay. Okay. I need ice cream with both of you. Dan.
- [02:11]
D
Rocky road. Sorry. There's more on that. I'm so sorry.
- [02:14]
B
There is, there is.
- [02:15]
D
That's the important part, though. Yeah. Hey. Dan Meyer. I work at Amplify in an R and D capacity. Former public high math teacher and we do curriculum ed tech at Amplify.
- [02:26]
E
Good morning. I am Kimberly Smith. I serve as Executive Director for the Office of Teaching and Learning in Jackson Public Schools. This is my 27th year in education. From a teacher to an assessment specialist to a building level principal, and now this current role. And my favorite ice cream flavor is by Hachin Dass. And it's Dese de Leche.
- [02:49]
B
Okay, well, I probably made you want ice cream. Good news. There's a Ben and Jerry's over in Seaport Village. So you can hit that up and also do the hack. Have an ice cream meeting at GSV next time you're here and go get Ben and Jerry's. You can have Rocky Road, salted caramel, lots of those. So I'm going to set the stage a little bit before I dig into questions. And I might even break the rule of gsv, which is to let you all ask questions, don't tell. So right now we're seeing a substantial amount of pushback against screen time and edtech. And I would posit to Dan's piece this morning that the sort of disappointment or the sadness of AI not delivering the revolution that it promised at AHGSV just a couple years ago. And so we see 36 billion bills in 19 states that are looking at everything from outright bans of screens and technology for kids K5 to states stepping back and saying we need to develop some guidance and some guardrails to some states coming forward and say we are going to be a champion of evidence and efficacy and we're going to have a statewide registry in Vermont or Utah. And so I think this is a moment of reckoning in the ed tech industry. And for our superintendents and district leaders, they're really having to answer important questions from parents, students and teachers themselves. And so I want to dig in a little bit around, like, why we're here and how we got there and frankly, the need for us to distinguish between consumer tech tools that are in our classrooms and ed tech tools. So, Dan, I'm going to start with you. You ready? Rocky road.
- [04:44]
D
Let's do it. Yeah.
- [04:44]
B
Ready? Rocky road. Okay. You speak a lot about intellectual need as consumer platforms make answers easier to find, especially with AI. How does amplify design for productive struggle a key element of science of learning? And what we know actually gets kids to learning while still maintaining user growth metrics that really are associated with consumer tech.
- [05:10]
D
Yeah, Great question.
- [05:12]
B
I think that I don't only ask good ones.
- [05:14]
D
Yeah, like, learning is hard and a lot of people, when given the opportunity, will opt out of it. Perhaps you have family members like I do, you know, but it is an unwiring and rewiring of your brain. It's difficult. And like most difficult things, it's humans who help other humans do that. Whether that's doing that last rep off the bench or tuning up your soul with a psychologist or a pastor or whoever, it's humans who help humans do hard things. And that's just a core feature of human psychology. And when we bring technology into that so often it undermines what it is that helps humans do hard things. So I'm talking especially about moves like putting kids in front of screens. Like, we've seen teachers and their aspirations for their work, students and their hopes for their social environment for learning. They don't want to be in a call center environment. No teacher signed up to be a call center supervisor. Parents don't want to walk into that class and see their kids in a call center. And so a mistake that a lot of ed tech companies make, I think, is in misunderstanding a teacher's aspiration for their work. They want the work to be easier. They want to feel more powerful. They don't want the work done for them, especially not in a worse way. So attending to that need for humans to help humans do hard things and to slip into the slipstream of teacher aspirations for their work is how we thread the needle of helping kids and maintaining a healthy, growing product and company.
- [06:41]
B
Let me just follow up there. I know you're going to talk about chatbots across the hall. I think in your next panel you want to plug it real quick in case you're there.
- [06:47]
D
Yeah, Seaport something. Rather great.
- [06:49]
B
Great. That was an effective plug.
- [06:52]
D
Seaport F and G. Yeah. Give me one shot. One more shot on that one.
- [06:55]
E
Thanks.
- [06:55]
D
It'll be awesome. How about that?
- [06:57]
B
Just follow Dan as he runs off stage. How about that? But is that why you think that AI chatbots are just not producing the learning outcomes because they remove the struggle? There's not enough, you know, productive friction. There's. Give me, give me your like.
- [07:13]
D
Yeah.
- [07:14]
B
20 cents. And maybe a little from the Codmigo piece this morning that you wrote.
- [07:18]
D
It's like this. Like, I'm in classes every week. I was in class on Monday, 8th grade learning. Learning area of complex shapes. I do this to sharpen my skills. I need to go in and feel my best ideas fall flat often, right? I'm in there and I'm helping kids as a tutor, right? And I'm just saying I helped kids who didn't ask for it. And when kids wanted me to leave, I didn't. And that's the key differentiator between a human tutor and an AI chatbot. The chatbot needs to be summoned in, like, the kid needs to click that thing and say, hey, come help me. And the chatbot will leave whenever it's told to leave. And those are just major differences in how human tutors of even average skill and chatbots treat kids. And key to the whole reason why kids aren't Learning from chatbots the way people hoped three years ago.
- [08:02]
B
Yeah. And I think it's also why we see when young people have a meaningful connection with one adult in the school, their chronic absenteeism level is going, you know, so far down because it is that human to human connection, not just about learning, but about being seen, known, loved. Okay, Christian, over to you. I think you have some mooses and socks in the. Is moose a word? Mooses is not a word. I think you have moose.
- [08:25]
C
Just moose is the plural.
- [08:26]
B
Okay. You have a moose and some socks in the back in case anybody needs those and you have room in your suitcase. I don't. Christian. Learning science suggests that deep thinking requires friction. We just talked a little bit about that. How as a product leader at an lms. Sorry, that's my kids. It's the only thing that rings on my phone. It's important at an lms balance the demand for seamless user experience. So you want people to be delighted, right? You want it to be nice with the need to keep students cognitively engaged?
- [09:03]
C
Yeah, that's a very interesting question. For me, my background actually is in human computer interaction and user experience design.
- [09:10]
B
Amazing.
- [09:11]
C
Have spent a career trying to remove friction from user interfaces to make them very easy to use and very clear in how they're communicating to you and how you can communicate your goals. So designing for great user experience is different than designing for learning.
- [09:27]
B
Say more.
- [09:28]
C
Yeah, so when we design for a great user experience, we want to remove friction. We want people to feel in control of what they're doing and for everything to be intuitive and easy. But learning requires productive struggle. It requires desirable difficulties. You need to have these moments where it isn't easy and that you have to persist and you have to challenge yourself. So in our platform, the idea is to balance that. Right. Is to be able to provide great content creation tools and learning experience tools to educators and to instructional designers so that they can create excellent learning experiences that have things like opportunities for reflection, opportunities for active learning, opportunities for metacognition. These things that we know from learning science actually helps with learning. At the same time, we want the aspects of the user experience to be very easy. It should be easy for me to find out when my assignments are due or to find out, you know, a little bit about my teacher if they. If he or she wants to share details about themselves. All those things in terms of navigation should be super simple. But the learning can't be too simple. It's gotta be challenging because that's how we grow and how we learn.
- [10:43]
B
And what about the idea that there's an opportunity to get content that challenges you? Is that something that you ever think about in designing the learning management experience for the student? That we want to personalize. Personalize, Personalize. But from running a school, sometimes when you give kids something that pushes them to be challenged, to drive something harder, they rise to the opportunity.
- [11:07]
C
Yeah, absolutely. I think when we expect more from people, they do tend to rise to that opportunity. So, yes, I think there's lots of opportunities to provide different paths for learners. You know, some learners who may need extra help can find that extra help. And those who have, you know, who are quick to learn a particular concept can be enriched with more challenging experiences. And I think one of the things that's really important and perhaps a challenge in some edtech platforms is you don't want to to peg a student as a particular kind of student in all domains. You might excel in one area or in one particular unit and struggle in another. So you need to be adaptive to the actual context of what is the student doing today.
- [11:55]
B
Well, I know this as a mom and as an educator, learning is jagged. Right. There's not an average. It is jagged. And so how did we design for jaggedness, I think is something that I think is super interesting. Okay, Jamie, you see thousands of apps at Clever. What are the red flags in design that suggest maybe a tool is not optimized for learning, that it's instead optimized for screen time engagement rather than educational impact? And I want you to dig a little bit into your product design background here.
- [12:32]
D
Yeah.
- [12:33]
A
Also a HCI user experience background. And I have built consumer experiences before I came into education. And so a lot of what you were just talking about in terms of frictionless is a really good idea when it comes to the things that are not part of learning. You don't want to have friction just logging in. You don't want to have friction getting to the things you need. But when I back up, your question was how. What are those red flags? Yeah, to me, it's about intentionality.
- [13:03]
B
Okay, say more.
- [13:05]
A
And a little bit of that awareness of what actually happens in classrooms and how schools need to function. And so a couple examples. One is, if you are building for consumers, you can kind of pick your audience. You can have a focus. We are building this for people who are like this. If you are building for students, you have to serve all students. You have to serve all students regardless of language or ability or all sorts of other factors. That is part of the obligation you have for being in this industry, it's a legal requirement, but more importantly, it's a moral requirement. And so if you don't have that built into your systems from day one and that mentality in your teams, you're going to end up building something that works really well for some students and really not well for others. And I'm also going to emphasize it's not just about the students. And I think this is one of the missing pieces folks talk about when we're having the screen time discussion or other things. It is a learning community. You have a whole sphere of folks around that student. You have the other students, you have the teachers, you have the curriculum leaders, you have the parents and the guardians and the other folks. And when we're talking about software, you have to have that connective tissue of how those people are going to be working together. You have to have that teacher in the classroom who's challenging the students. Do we want the teacher's experience to be full of friction? Probably not. They have a very hard job. Let's make that as easy as possible.
- [14:31]
B
No, can I just say, like we do not want the teacher's job to be full of friction.
- [14:36]
A
So I think that's, it's. When I look at things that are designed intentionally for an educational context, these are the kinds of things you just see built in from the beginning often because maybe they have educators as part of their design teams or their engineering teams, but because they've thought through. Yes. This has to be intentional and about learning and it has to work for every student.
- [14:59]
B
Here's what I hear you saying, that we need to be thinking about what's the learning experiences we're designing and not the schooling experiences we're trying to plug tech into. Yeah, Dan, did you want to chime in there?
- [15:10]
D
I was hearing different and I want more from Jamie. So you told us about the features or what it looks like or the process of creating these tools that work for the ecology of a school that plug into the parent community. But what are the features of those that do or don't? So like we have teachers on our team, let's say great, we check one of Jamie's boxes on process. But just like I selfishly, I want to know more about the ones that do work for the broader community versus a very sub select of, let's say kids for whom every edtech tool will work for no matter what.
- [15:42]
A
So I will give one example because it is an important subject for me, which is around accessibility. And often a lot of software companies treat Accessibility as a kind of a checkbox. It's like a thing that we have to make sure we do, blah, blah, blah, blah, and you do a little audit or whatever. Have you gone into classrooms with students who have different abilities? Maybe they have issues with vision or with hearing? Have you worked with teachers who deal with those communities? Have you worked with teachers who may be vision impaired or have other disabilities? So that's one concrete example of where there's kind of a checkbox version and then there's the version where you have actually gone. And I'm gonna try to get more specific because it comes down to even the subtleties of like your color use, your use of different typefaces, the ability to. I mean, who here. Are there any engineers in the audience? We got a couple. Yay, engineers, Designers, people do with software. Okay, how many of you. I'd love to raise hands, including this audience. How many of you have actually experienced your own software through a screen reader? This is a great audience. Yeah, that is a better, better odds than I usually get when I ask that question.
- [16:58]
B
This is like Wednesday. This is the Wednesday audience.
- [17:01]
E
So.
- [17:02]
A
So one example. I could do more, but we'll go on to other things.
- [17:05]
B
I love that. Can I just. I'm gonna just jump in here too and just say like. Can I just tell you something that really pisses me off?
- [17:12]
D
Can we stop you or no? Is that like an actual question?
- [17:14]
B
It's not. It was a rhetorical question. Dan. Just like I don't stand next to Dan ever, because he's the tallest person I've ever met, I think, in my life, besides Shaquille o'.
- [17:24]
E
Neal.
- [17:24]
B
You're in good company there. But products that don't make it easy to turn accessibility features on. One of the things in one of our alliances that we had to do during the COVID pandemic, when folks all of a sudden had to understand how to take their classroom experience into tech, was they couldn't even figure out how to turn the accessibility features on. If you are designing a product and it is not like instantly easy to do that, go back and fix that now and don't hide it between like, you have to search for it or you have to have different types of crazy permissions to do it. The ability to turn on accessibility features, first of all, it's the law that these are available for a free and accessible public education. And two, remove that friction for teachers. That is non purposeful friction and it hurts kids. All kids, not just students with disabilities. Okay, sorry, I'm gonna Tuck my soapbox under this chair and get to somebody I really admire, Dr. Kimberly Smith, who's at in Jackson schools. Dr. Smith, I knew you from your work on outcomes based contracting and demanding student outcomes matter in the work that you're doing. But I'm going to ask you to really call in the voice of the practitioner here because we've heard a lot from the supply side. You are here to talk about the demand side of this equation. And so I'm going to start with this. I'm a sexy new AI driven tool that promises to revolutionize the classroom. Have you heard that this week?
- [19:05]
E
To some degree, yeah.
- [19:07]
B
What specific evidence or outcomes are you looking for before you sign a contract? How do you, as someone who's in schools, distinguish from the flashy user experience and this user AI from deep learning? I'd love you to take us to church on those points, Dr. Smith.
- [19:28]
E
So I would say in partnership with this work, we do work with our data and accountability department and that's led by Latoya Blackshear. And so she has been a strong forerunner with our OBC contracting. And so when we're looking at products,
- [19:45]
B
OVC is outcomes based contracting. Go ahead. Sorry. Thank you.
- [19:49]
E
No problem.
- [19:49]
B
I'm a drug investor.
- [19:50]
E
Yes. Thank you. So when we look at that, you know, look at products that are being offered to our district, the first thing we look at is our demographics. And so we always ask, like, what is the data for districts like ours? We are the only urban school district in our state. And so we're looking at products that will help for certain unique situations that our students may endure that other students do not. And that's a myriad of different things, a plethora of different challenges and setbacks or even some of the things that we move forward because we also have some specialized schools. So we look at that piece, but we also look at. And my amplifier friend, he really hit on it. We look at products that's going to really help expand the teacher's reach in the classroom and not doing the work for them. And so, you know, we are really big proponents of small group instruction.
- [20:46]
B
Yes.
- [20:46]
E
So trying to manage a classroom of 25, 30 kids and smile group, what does that really look like for a teacher? When the expectation is for the teacher to have a teacher guided group, but then you have four or five other groups. So what support do those products provide? Does it provide scaffolding for those students that need it? Does it provide adaptability? If I'm getting it, I got it. How does it, you know, catapult me even forward, you know, further in the work that I'm expected to know and be able to do. Because one of the things our challenges have been like, once we get our students to proficient and be maintaining that, not allowing them to slide back. And so when you look at our data and probably nationally, most of our teachers teach in the middle. And so how do we. And we have a great opportunity of growing students. But once you get them to that level, they need to be. How do you catapult them forward and keep them there? And so we look at products where. And you talked about it, the accessibility piece, you know, we look at our students with all types of disabilities. Is it functionable for our teachers but as well as our students. So looking at the easability usability of those products and then what's the parent component? Because we have to have our families and communities that are a part of this work. And so a lot of times when we have after school programs, things of that sort, they are looking for those data reports. How can they continue that learning usage on that platform outside of the classroom? And so there are. We use certain rubrics when we're looking for that. But also the professional learning piece, the product is only as good as the person in the classroom that's utilizing it to help that you have to be blended with a product that yes, they're on screen time, but then face to face. So when this report is giving us oh it's a yellow flag, it's a red flag. And now there has to be some face to face so the teacher or the tutor whomever can provide some type of face to face or intervention. Because if I'm at a red, I don't know it. And there's not a whole lot of prompting I can do to a, in a chat box. I need a person to come and assist me. So professional learning is very key in the work that we do because it gets used if the person that's using it understand how it works.
- [23:04]
B
How do you feel about ed tech vendors who make you pay separately for professional learning and not as integrated into their product?
- [23:13]
E
Most of the products that we use is separate.
- [23:16]
B
Yeah.
- [23:17]
E
However that. Because it can't be a one and done type thing.
- [23:20]
B
Okay.
- [23:20]
E
A lot of times the initial professional piece is about understanding the platform. But then as ongoing now you got data reports, now you have other unique challenges that present itself once you've delved into the use of the platform. So now we need some support with that piece. And so we do look for as a key Ongoing support that's within that. So what I've noticed is now that it's not always a face to face. The initial training may be face to face, but then there's virtual 24. 7 support that really helps that and immediate feedback. A lot of times we have discontinued services with certain providers because there's a delay in response time. If I have an issue, you know, I can't wait. 2448. 78.
- [24:07]
B
Yeah, your house is on fire.
- [24:09]
E
I need support now because I got a student that needs support with his work or a teacher that needs support. You know, we're moving forward and so we look at how the response time is from the provider as well as the ongoing support. And virtual support is fine as long as it's not AI assistant, because I can keep asking my questions and I need to get further into what I need support with.
- [24:36]
B
Yeah, I was with the superintendent yesterday who was on a rage machine because he was like, first of all, I put a ticket in. It took them 72 hours to get back. And then the person who got back to me was not a person.
- [24:48]
E
Right.
- [24:48]
B
It was an AI answer interface. And so we don't just need human support in pd. Right. And in classrooms. We need it around operational work. Thank you so much. Okay, I'm going to just open this up to anyone who wants to take it. In the session description, we talked about the blurring line between consumer tech and edtech. Should procurement policy shift from buying features to buying outcomes like Dr. Smith has done? And how do you prepare for that? Looks like Christian is like ready.
- [25:24]
C
I'll jump in on that one.
- [25:25]
B
Yeah, he's got the moose and he's ready.
- [25:29]
C
Yeah. So absolutely. I do think we should. And I think we are seeing a shift towards more outcomes based procurement. Unfortunately, I think there is still a lot of feature evaluation and I think we do that because it's easy perhaps.
- [25:44]
E
Right.
- [25:45]
C
You can create a spreadsheet, you can have a checklist, you can compare feature A to B and just kind of go down a list. By the way, I think that's the same dynamic that leads to us in our personal lives having too many buttons on our remote controls and our microwaves. It's feature creep. So much more important is to be focused on the kinds of outcomes that you are looking for from a platform that you might be bringing into your district, into your school. So how do we prepare for that? Well, I think from a product perspective, whenever we are building features, first of all, they should be built because there is a real need, a Real observed need in our client community, not just
- [26:25]
B
somebody who's sitting at their desk.
- [26:27]
C
Yeah. Not just like, here's a shiny object.
- [26:29]
E
Oh, cool. Yeah.
- [26:30]
C
Yeah, exactly. So, yeah, grounded in real user needs. That would be the first thing. The second thing I think is that when we do build something, we want to make at least a really strong hypothesis that it will move some kind of important needle, whether it's something on driving greater student outcomes, driving greater retention, staff productivity. There's all kinds of different outcomes that we might be trying to achieve. And we want to be very clear at the outset about why we're doing something. And then it's measurement. Right. So we measure as we build things and as our clients adopt solutions, we work with them to have success criteria early on. And these are things that we can measure. Right. So if we intend for something to enable teachers to provide faster feedback to students, that is a measurable thing, we can see whether that has actually changed over time. And then, you know, and then we can have a hypothesis that if students get feedback sooner, they will perform better. That is also measurable. So I think it's trying to build in those measures and be aligned between, you know, all the way from product conception to implementation and, you know, ongoing adoption and support as well. You know, just launching a feature isn't particularly useful. We want to be partnering with our customers to make sure that they are getting the full value out of it, that they know how to use it and that they can grow over time in their successful usage of a. Of a platform.
- [28:01]
E
So I would say also too, it deepens the partnership because it creates a cascade of accountability. Right.
- [28:09]
B
So that I'm stealing. Cascade of accountability.
- [28:11]
E
Just so we're clear, because it creates opportunities when you are buying outcomes, we got to check in more. I'm not just calling you when I need you. We're checking to see did I promise that I deliver on what I said. And then it holds the school system, district accountable as well. Are the students showing up? Are they on? Are they doing these things? So when you create that scorecard and put those measures, those. Those measures in place, then you're holding each person accountable. The company is held accountable for what they said they would do. The district is accountable for what they said they would do. And so then you should see some higher gains there, because it's not like I'm giving you a program or I'm giving you this platform and you got, I'm going to give you some ongoing support, but then you do whatever you want to with it. Then the studies and the research that has said this project is designed to do this, this and this. You don't really know if it's effective or not because the school district has not done anything they need to do with it. So I think the outcomes based contracting just creates a deeper partnership and then it also allows the, the, the provider seat. What more do you need? I can't tell you the countless numbers of partners we've had where we've actually helped them to customize their product to help support our needs. That in turn helped other districts as well. And so when you have a closer knit product partnership of that sort, then it makes both accountable and it makes both entities better.
- [29:37]
B
And you're using the word partner there. You're not saying customer, you, you're using the word partner. I just want to like call that out. Thank you so much. Dan, you want to chime in?
- [29:46]
A
Yeah.
- [29:46]
D
I love that description of the partnership and what it does for us on the provider side as well. I think it creates a really strong, healthy pressure on us to support not just the variability of students that we support, but also and especially teachers that we have a teaching force that has such incredible variety of skills from novice to expert and massive turnover, especially in large urban districts every single year. And it's very easy for a lot of providers to say, you know, like to blame poor outcomes on teacher implementation. But this kind of OBC style relationship says to us we need to find ways to support teachers even past that first year. We need to realize that second year educators in our program have different needs than first years. It creates incentives to look to the consumer tech world and say, you know, they don't have an onboarding webinar for a Figma subscription. They don't have like a, an in person 6 hour PD for a Google Apps sort of integration. Like what can we do to flatten that learning curve so that any teacher, any skill, novice to expert can hop on and then grow with us. Grow together.
- [30:47]
B
Yeah. They also don't have evidence, like I think the thing in the Instructure report that when we released it at south by Southwest, InnovatedU and Instructure just partnered on the 2026 evidence report. Yesterday we released the 150 most used tools in America's classrooms. It's about half consumer tech, half edtech. Edtech tools that are classified. 40% of them have evidence still stacked at the lower tiers. Anyone want to take a guess what the consumer tools were? You cannot say if you were in the room when we announced that. Anyone. Anyone want to take a guess? 40% EdTech has evidence. What do you think? Consumer tools. You said five. Okay. Lower.
- [31:29]
D
Zero.
- [31:30]
B
Two. Two percent. Like, what the hell, 2%? And so that's a. I think it's a clarion call for us to really be thinking about. If you can't articulate what you're doing with student outcomes, why is this tool in America's classrooms? Okay, now we have to talk about AI because it's like obligatory. So we all know that AI is moving faster than traditional peer review cycles. I, you know, you're not going to have an RCT right now, which is a randomized control trial on an AI tool that's been in production for six months. How do we build more rapid evidence? I see our colleagues from Lean Lab were in here. They're doing a lot on rapid evidence frameworks. But how do we think about rapid evidence that tells a district or a teacher that this is working in three months or three weeks rather than three years? Any of you, please chime in.
- [32:30]
E
Well, ask. I kind of start and look at and think about the parameters in which we utilize AI in our district. It is, it's, it's, it's new, but it's not new. So let me say that. And so looking at, first of all, we have as school districts, we have to get our thoughts around what is the use. So when we talk about what's working, what's not working, first of all, how can I use it and then determine if it's working or not? And so even, you know, we are looking at different curriculum now that helps teachers to develop these plans, develop, you know, individualized plans, because that's a lot when you got 25 kids in a room. They're all on 25 different levels for the most part. And you're having to develop these plans and scaffold and all these different things. And so looking at what's working and what's not, I think we're still trying to figure out in our district what it is and what's the use. So we have some AI tools that we utilize to help support that piece. But then you still gotta look at your data. So when you have AI tools that can. Their algorithm helps to support providing that data based on what students are putting in. That really helps us to determine what's working and what's not. So, for example, if a student. Student is working a math problem based on their prompt in that chat box, and then whether it's saying, I don't understand what's the next step, whatever that is, if it can give you, the teacher, a transcript or a summary of what that student is putting into their chat box, then that helps teachers to better understand where they need to intervene, where they need to provide stronger Tier 1 instruction of that sort. So that's what we kind of are right now with what's working and what's not.
- [34:09]
B
So learning process data tied to instructional moves is what I hear you saying. Like you want to understand the process of learning the places where students are struggling, because that can be helpful for the teacher and understanding, you know, what's the next move. When I taught middle school science, I used to do a hinge question. It was like my did. And the hinge question was basically like, did you? It was a question that helped me evaluate in the moment. Did you grok what I said the day before? Right. And that hinge is really, really important because if kids, we know, keeping kids on grade level content is really important, we shouldn't just like deliver a lesson and then move on. It's, do they actually understand it? Anyone else want to chime in? We need more evidence. How do we do it in the age of AI rapid evidence generation. Yeah, yeah.
- [34:56]
C
I mean, I think both can be true. Right. I think we do want to have, you know, rigorous academic research as well over multiple years. I think that'll be true.
- [35:05]
B
And it's not an or.
- [35:06]
C
Exactly. Yeah. And then in terms of like the rapid, I do think we want to have rapid signals. Right. So again, I think if we can have like, you know, a clear assumption about what we want to be measuring, we want to be looking for early indicators of longer term success. So if we can identify what those early indicators are, if we can run controlled pilots, not kind of like random
- [35:29]
B
pilots or random pilots. Well, you can't. That's happening.
- [35:34]
C
Yeah, exactly. Right, yeah. So if we could have more controlled pilots where we can actually learn from them, because otherwise pilots tend to end and things are just too messy. You can't really learn anything from them unless you've really been deliberate about how
- [35:45]
B
you run a pilot and how to make them generalizable. Right. Like, I think you're trying to combat the special snowflake problem. Right. I'm a pilot. I'm a special snowflake that's not going to work in someone else's classroom. I love that. Anyone else?
- [35:59]
D
You can make a huge chop right away just based on usage. Yeah. I want results. But if you look at the usage data on a lot of these tools, they're used by 5% of the kids, 5% of the teachers for who have access to them. That's a great signal right there. Even if you got efficacy out of that 5%, what does that buy you? What is that worth to you when it's the same 5% that seems to succeed based on any resource? Just chop based on usage first.
- [36:24]
B
Chop on usage. That's a good one.
- [36:27]
A
I'm going to answer maybe from a different direction and this is maybe a little bit more as a parent than you're calling. You're calling me in because I think the one piece that makes this all more challenging but also is very exciting when we're thinking about how we learn what's effective. Kids have their own autonomy, especially when you get into those older grades. They are selecting their own tools. I know that's something that we all think about as parents, as teachers, as folks who work in education. And I have certainly watched my. The classrooms that my kids are in have to adapt to the new tools that are available to both students and teachers, regardless of sort of that larger curriculum environment. I think that's where I would start is really that student centric of like if they are being pulled into certain kinds of tools, a some of those may not be a great idea. So we should certainly understand what is happening. But we also can learn those lessons of like what is all the way back to that original topic of like where do we see that productive friction? Where do we see that non productive friction? And so I don't think we always should start with the vendors. Even though that's a lot of us up here. I think sometimes we should start with that classroom environment and what's happening organically to understand where the best learning can happen.
- [37:51]
B
Great. I'm going to hit just a real lightning round here. I think we've distinguished that we have tools that are. The usage point is a really good one. We have tools that are just optimizing for engagement but aren't actually generating learning. We have tools that are not being used. Lots of things to do just we tell superintendents who might be facing ed tech pushback right now. The first thing you need to do is find out what is being used in your classrooms and in your district, put it on the website and say what you're using that for. That discipline of auditing the tools, frankly, like scrapes a lot of tools off. But it also opens the conversation with parents. So for any superintendents and folks who came to this session because you're thinking about how do I continue to have social License to innovate. Transparency is really, really important with our families, our communities and students. So real quick, 30 seconds each. Think about an edtech tool, one you love or one you hate. What is one feature or behavior in that tool that looks like learning but is actually just engagement, is actually just hooking the student, trying to hook the student who wants to go for Dan. Dan's got the mic up like the ice cream. You good? You good?
- [39:04]
D
Yeah, yeah. Personalized learning math software. Kids tank the benchmark. They do below grade level work and it looks productive on the kid's side. Teacher has kids that are docile. No one's actually learning.
- [39:16]
B
Awesome.
- [39:17]
E
I did know that as well. And then the little fun stuff comes out and it's not, not. They're not learning. They're just being engaged and entertained.
- [39:25]
B
Just playing a game.
- [39:26]
E
Just playing a game.
- [39:26]
B
I love that. Okay.
- [39:27]
D
Coolmathgames.com.
- [39:29]
B
what'd you just say?
- [39:30]
D
Coolmathgames.com.
- [39:32]
B
it's like, no, hey, coolmathgames.com. you're on blast.
- [39:36]
C
Yeah, yeah, I'd say like without maybe naming any particular vendors, but you know, there are, I think there are a lot of things where people will just click through, right? They'll just click, click, click. It's kind of mindless. It'll produce a dashboard. It'll look like something has happened, like completion has happened, like learning has happened, but it really hasn't. I think that's a real red flag.
- [40:00]
B
Yeah, red flag.
- [40:03]
A
I'm trying to.
- [40:03]
B
Just one.
- [40:04]
A
Just one. Yeah. There's a lot of talk about gamification and I think just the student to student competition is one that I would watch like a hawk. It could be used well, but I think it often is one you gotta keep an eye on when you're having that kind of leaderboard dynamic. I would be very careful about how I deploy that.
- [40:25]
B
Yeah, I'll just say one. If you have a tool that's using or trying to form a relationship with your student in a chatbot or companion interface, get that out of your classrooms right now. They are bad, there's foreseeable harm, and they are just about keeping a student hooked, not getting a student to learn. Thank you to this panel. You're amazing. Give them a hand.