Episode 347 - Conceptual Learning w/ Dr. Catherine Williams

Summary

---

ABA Inside Track – Episode 347: Conceptual Learning w/ Dr. Catherine Williams

Release Date: June 17, 2026
Host: Robert Perry Crews, with co-hosts Jackie McDonald and Diana Perry Crews
Guest: Dr. Catherine Williams, Assistant Professor at University of North Carolina Wilmington

---

Episode Overview

In this lively and deep-dive episode, the hosts are joined by Dr. Catherine Williams (“Triangle Lab”) to explore the often-overlooked but foundational topic of conceptual learning within behavioral analysis. The conversation dissects key research studies, practical approaches for instruction, and the nuances of teaching complex concepts. Through a mix of technical detail and accessible analogies (expect plenty of triangles!), the episode aims to translate rigorous research findings into actionable strategies for educators and behavior analysts.

---

Main Topics & Discussion Points

1. Defining Conceptual Learning in Behavior Analysis

• What Is a Concept?

  • Catherine: “A concept itself is a group of things that belong together... might have physical characteristics in common, some sort of relation, or a shared reinforcement history.” (05:54)

• Behavioral Importance:

  • Emphasis is not just on what a concept is, but on demonstrating conceptual learning:
    • Generalization: can you apply knowledge to new instances?
    • Discrimination: can you identify what doesn’t fit?

  “We need discrimination, too... most of the times that we’re trying to generalize, it’s not that we truly only want something to be applied everywhere. It’s that we want it to be applied to more than what we’re teaching, but not to everything.” — Catherine (08:16)

2. Historical Frameworks & Influences

• Behavioral research on concepts traces back to studies on pigeons (e.g., Herrnstein et al.), and early instructional design work (e.g., Tiemann & Markle).
• There's a disconnect between existing educational practices and behavior analytic principles regarding concept formation and teaching methods.

3. The Role of Examples and Non-Examples

• Must Have vs Can Have Features:

  • Must Have: Essential defining characteristics
  • Can Have: Extraneous attributes (color, size, etc.)

• Types of Non-examples:

  • Close In: Barely differ from examples, forcing nuanced discrimination
  • Far Out: Drastically different, offer easy discrimination but less robust conceptual learning

  “The closer our S- (S delta) is from the S+ (S delta), the better that discrimination is.” — Catherine (24:03)

4. Research Overview: Six Experiments on Conceptual Learning

General Procedure (29:08)

• College students (often for extra credit)
• Presented with arbitrary visual stimuli (mostly complex triangles with various features).
• Pretest (to ensure no pre-existing concept), feedback-driven training (correct = $$, wrong = buzzer + loss), and posttest with novel stimuli to assess conceptual learning (i.e., generalization & discrimination).

Details of Stimuli: “Crazy Triangles” (32:49)

• Deliberate manipulation of features to create both examples and nuanced non-examples.
• Each experiment evolved to test different permutations or variables, such as:
  • Motivation (reinforcement magnitude)
  • Feature arrangement and similarity
  • Impact of “real” vs. arbitrary concepts

“Six experiments... I love lots of little studies that lead, tell me a story.” — Rob (41:52)

Sequential Research Progression (46:23)

• Exp 1: Close in vs. far out non-examples with arbitrary triangles; mixed results suggest more than just feedback is at play.

• Exp 2: Same prep but with real biological concepts (cell types); more robust conceptual learning, suggesting 'realness' and context matter.

• Exp 3-6: Dissected reasons for variability:

  • Manipulated motivation (money), feature arrangements, and teaching order.
  • Demonstrated that close in non-examples paired with explicit concept analysis leads to the strongest, most generalizable learning.

  “It’s not just inherently that these [concepts] are real and these aren’t... it’s that there’s different feature arrangements that matter.” — Catherine (50:31)

5. Instructional Takeaways: Rules, Feedback, and Example Selection

• Providing Rules

  • Implementing explicit verbal instructions (Rule-EG – rule, example, non-example format) accelerates learning for most. (See second study)
    • “If I provide rules that just lay out the things I need to know... does that do all this for them, right?” — Catherine (55:24)
  • For some learners, rule-based instruction may obviate the need for lengthy trial-and-error practice — but only if rules are well-constructed (incorporate must/can have features, plus clear examples and non-examples).

• Application to BST

  • Raises the question whether all four components of behavioral skills training (BST) are needed if robust rules/examples/non-examples are used.

6. Practical Applications: Designing Slam Dunk Lessons

• Key Steps
  1. Do a Conceptual Analysis — Identify must and can-have features.
  2. Select two maximally different examples and several close-in non-examples.
  3. Use clear rules and structured feedback (the Rule-EG sequence).
  4. Advanced: Start lessons with overly simplistic stimuli (only must-have features), then gradually introduce more complex can-have features.

“Just think more about your stimuli. That’s it. That's what it boils down to.” — Catherine (69:02)

• Student Involvement
  • Have learners generate their own examples/non-examples to boost motivation and deepen understanding.

7. Future Directions

• Expand research to diverse learning contexts and learners (early childhood to adult, lab to classroom).
• More examination of motivation, instructional sequence, and direct vs. discovery methods.

---

Memorable Quotes & Moments

• “Embrace the crazy triangles. You’re going to hear the word triangle more today than you probably did any other day this week.” — Catherine (02:09)
• “It is, sort of, like a giant...Sudoku game...enough degrees of freedom to have different stimuli, but also they all fit together and they show different patterns.” — Catherine (34:18)
• “If you want to sound smart, just talk about how the examples and non-examples are miscategorized... No, no, no, conceptual learning is happening." — Rob (16:54)
• On Rule-EG instruction:
  “You don’t actually always need the practice and the feedback to get learning, if you have these rules and examples set up this way.” — Catherine (59:00)
• On textbooks and real-world teaching:
  “Even textbooks are a great example... they never explicitly say, ‘this isn’t that,’ but it’s something different... you would probably learn it a lot faster.” — Catherine (61:14)

---

Notable Segment Timestamps

• 00:13 – Episode intro & guest introduction
• 05:54 – Behavioral definition of a concept and conceptual learning
• 13:05 – Historical context and prior research
• 20:48 – Examples vs. non-examples, must-have/can-have features, close-in/far-out definitions
• 29:08 – Overview of experiment procedures
• 32:49 – Stimulus design: "Crazy triangles" and analogies to the Spot It! game
• 38:10 – Discussion of multiple experiments within one paper
• 46:23 – Iterative progression of experiments and findings
• 53:32 – Key takeaways from the research
• 55:24 – Effect of rules (Rule-EG) on learning concepts
• 66:45 – Practical steps for creating effective conceptual lessons
• 69:16 – Future directions and research possibilities
• 73:18 – "Pairings": Related topics & “snack pairing” (Doritos as example concept!)

---

Practical Insights for Listeners

• Don’t skip conceptual analysis — It underpins effective concept instruction.
• Use close-in non-examples — They are more effective than far-out non-examples for teaching nuanced discrimination.
• Combine rules, examples, and non-examples — For learners with verbal skills, this can be a shortcut to concept mastery.
• Start simple, then layer complexity — Progress from core features only to more naturalistic variations.
• Involve learners in creating stimuli — Boosts motivation, relevance, and insight into feature control.
• For curriculum developers: Apply behavior analytic concept instruction to diverse content, not just “behavior-analytic” curricula!

---

Further Resources and Contact

• Contact Dr. Williams:
  williamscluncw.edu
  Open for collaboration and discussion on applying/expanding this style of conceptual learning research!

• Recommended Episodes (Pairings):

  • Ep. 143 – Stimulus equivalence
  • Ep. 168 – Emergent relations
  • Ep. 31 – Problem-solving
  • Ep. 73 – General case analysis
  • Ep. 223 – Generality and generalization

• Snack Pairing: Doritos — “Crazy triangles” in the chip world; perfect for contemplating must-haves and can-haves of a concept!

---

This summary covers the essential academic and practical substance of the episode (excluding ads, extra plugs, and non-content banter) and preserves the show’s playful tone and educational intent.

Transcript

[A]

Foreign.

[B]

Hey, everybody. Welcome to ABA Inside Track, the podcast that's like reading in your car, but safer. I'm your host, Robert Perry Crews, and with me, as always, are my fabulous co hosts.

[A]

Oh, hey there, Robert Perry Crews. It's me, your friend, Jackie McDonald.

[C]

And it's me, Diana Perry Cruz. Don't have to be a friend because I'm your wife.

[B]

Ouch. Wow. Thanks for sharing about our marriage secrets. Don't be friends with your partner. Just make a podcast. It's all going to work out just fine. All right, so, you know, you probably already figured out this ain't a podcast about marriage tips. It's a podcast about behavior analysis and behavior analytic research where every week we pick a topic and discuss some relevant research articles. And sometimes we pick a topic that is, it makes sense. But also there's like a lot of in depth, deep experimental research that needs to go on. And we wave the white flag of jab and call in a special guest. And that's what we have done today because we're going to be talking all about conceptual learning. And if it were just us, the episode would end right there. But fortunately, we have been joined by Dr. Catherine Williams, who's going to walk us gently through all the experiments on conceptual learning.

[C]

Dictionary defines conceptual.

[B]

Catherine, thank you so much for, for coming here and sharing about your like again. I don't know, it's like a jab thing. It's like the reason when you're reading it, you're like, oh, yeah, no, the perfect makes sense. And then the second you try to explain it to somebody else, you're like, there's these shapes and they're all over the place. And you sound kind of crazy. So I'm so glad you're here to help us out with, with the concept and the research of conceptual learning.

[D]

Well, thank you so much for having me. I'm very excited to be here. And it's funny that you. You've already kind of called it what I call it when we refer to them in my lab as crazy triangles. Like, that's what they are.

[C]

They're.

[D]

They're the crazy triangles. So it's great. Embrace the crazy triangles. You're going to hear the word triangle more today than you probably did any other day this week.

[B]

Excellent. Very excited about it. But, you know, we'll go beyond just the triangle as a shape. We're going to be talking about it as a concept, the concept of the triangle, our conceptual triangles here. So, Katherine, would you mind telling our listeners a little bit about yourself? And sort of how you came to become the Triangle Lab.

[D]

Yeah, it's a convoluted tale. No, actually, so I, I started out in behavior analysis. I think I kind of like many stumbled upon it during undergrad. Just took a really, really great class actually in eab. I think that's sort of unusual. I started out in the basic research and then I found the implied world. Ended up doing some internship stuff at the Marcus Autism center and severe behavior and decided that this was gonna be my life. So I went to grad school, got my master's and PhD at WVU West Virginia University, went back to the Marcus Autism center for postdoc, and now I'm at the University of North Carolina Wilmington as an assistant professor. That's the story.

[B]

Excellent. That's the story. And where did the idea of conceptual learning come about? Was it like you took a tangent into learning theory? You were like, I want to program something in an experiment and that just was the quickest one you could get your program going. Where'd it come from?

[D]

No, it was not the path of least resistance, that's for sure. It came from, I think, actually my very first year of grad school. Well, back up. I knew going into grad school that I wanted to get into how to teach better and in particular how to teach complex skills better. So getting beyond just like simple things and going into problem solving or multi step learning or concepts as it turned out. And so a really cool opportunity came up to go to the Morningside Summer Academy, which is a great program. If anybody's interested as students, you can kind of apply to go for free three, essentially three weeks over the summer. And they sort of introduced me to these ideas of how to teach concepts and how to teach complex learning that are great. But I noticed you don't really see them much in our everyday literature and they don't come up in most of our classes or anything like that. And I think part of the reason is there's not a lot of recent empirical basis for them. And so I wanted to work on that empirical basis to start doing research, to sort of be like, this sounds really, really great, it seems to work, but can I provide experimental evidence that it works? And really the best way to use it?

[B]

Okay. So I'm fascinated that you kind of came at it from a teaching perspective because certainly as someone who's primary, I went to graduate school for special education before becoming a behavior analyst. When we talk about how to teach, the work you did in your studies, even though they were translational research and basic research. And it did really feel very close to that idea of how do I teach something new? Like, what are the mechanisms at play? And even though it wasn't necessarily the total goal, I think of the study, I'm definitely going to be, you know, wrapping us up with some like, okay, but really how do we teach things now going forward? Because your research so directly points to that idea of the. The concept as something to be learned. Which brings us to kind of the question, when we talk about a concept, what. What are we actually talking about? Like, I know Jackie would probably define it as a mind file that has a lot of examples in the papers. And, you know, you open the fine mind file and you pull out some examples or whatever, you know, that's it. But we all know that's a silly way to think about learning what exactly is a concept in behavioral terms.

[D]

Yeah, and I was going to say that's the first thing is it depends on who you're talking to. So I do think, like, if you're going out into your everyday world and everyday environment, like, be aware that the way I'm about to talk about concepts is not how most fields outside of behavior analysis talk about concepts, but really a concept itself is a group of things that belong together, which sounds very, very big, but just meaning that they might have some sort of physical characteristics in common. They have some sort of relation. So, like, the concept bigger something is always larger than something else. Or they can also just have a shared reinforcement history. Thinking about, like equivalence classes, that kind of world can also be thought of as concepts. And so, like, that's what a concept is. But really, I think from a behavioral perspective, what's more important is thinking about, like, what demonstrates conceptual learning. So, like, how do we know that somebody's treating something like a concept? And that's all about behavior that shows that you can go beyond the things I taught you to identify new instances of what fits with that concept and then also just as importantly, identify what isn't a fit for that concept. So the technical terms there, right? Generalization, applying it to things you haven't seen and saying that is a concept, and then also discrimination, applying it to what you've seen and saying, no, that's not an example of what I'm talking about here.

[B]

Okay, so that combination of knowing what you're, you know, these examples of non examples of as well, both aren't there. We're not talking about a concept. You're talking about maybe like a rule or you're talking about just like A set response versus learning a concept.

[D]

Exactly. And going. Yeah, I think the rule, the set response is like, I'm going to give you all of the things. You're going to demonstrate it in front of all of the things versus the concept is I'm going to give you some of the things, but I realistically can't give you all of them. And I want to see, hopefully, that you can do it with anything that you encounter in the real world.

[C]

Yeah. So there's like differentially reinforced boundaries, basically, of some.

[D]

Exactly.

[C]

Name. Right. So like, dogs are generally furry, they generally have four legs, they generally have a tail, but they don't go meow. And that's how we know the cats are not dogs.

[D]

Most dots don't go.

[C]

Yeah, most.

[D]

Right, exactly. But yeah, exactly. And I think one thing I like to really emphasize here, we talk a lot about generalization training for generalization. Generalization, generalization, generalization. But we need discrimination, too. And I think most of the times that we're trying to generalize, it's not that we truly only want something to be applied everywhere at all times to everything. It's that we want it to be applied to more than what we're teaching, but also not to everything. And so I think discrimination doesn't get the paper time that it should.

[B]

Well, it certainly does in your papers. And we have two papers we'll be discussing. One we'll probably be discussing the majority of the time. The other one was a little bit shorter and kind of, you know, to the point, like, here's what we learned. I mean, I think I will say, Katherine, kudos to you. You had a pretty long paper, and in the end you were able to boil down all the results. And when I say all the results, there's a lot of results to read, folks. To boil it down into, like, five general principles. So even if you were crunched for time, you could probably read your discussion and be like, I think I learned something about conceptual learning that I did.

[D]

Yeah. In fact, let's. People want to do this research, which. Great. Love that. Please email me. Let's do it. But I would say, yeah, skip another method. Skip. Maybe like skim the first method just so you have an idea of what's going on. Skip to the discussion. That's where the meat and potatoes is.

[B]

Okay, but what are those in case people do not want to take your advice and they want to, like us, go through all the paper so they could generate all the. All these questions about these concepts and conceptual learning?

[C]

Is that my cue to tell you what the articles are. So as Rob said, we have two articles to discuss. They are using must have and can have features to improve conceptual Learning by Williams, St. Peter, Peron, Aguilar, Cederberg, Gregerson and Richardson. And that was published in jab 2025. Also, instruction consisting of a rule and set of examples and non examples reliably teaches concepts by Williams and Roop. And that was also in jab 2026, I think, maybe is it when it finally came out. 2025. Two in 2025. Wow, what a year. Okay, that's what we're doing. Sorry.

[B]

Okay, that should be a little better.

[D]

You were really busy in 2025. Well, no, I was really, really busy from 2020 to 2025. And then they finally got accepted in 2025. No, I know, I'm totally joking.

[A]

I love when people are like, wow, did you just get those done in one year? And you're like, no, it's been multiple

[D]

years in the making.

[B]

Yeah, I mean, you all know how it is. I mean, you're all, you're all, well published researcher. You just kind of bang out a study real fast, you know, boom, boom, boom, Irb, what do you think? And they all say thumbs up. And then you're done in a week. And then, you know, write it and it's gone. Published immediately. Oh no, folks.

[C]

I always think it's going to be like that somehow. I still think you do you still think that.

[B]

Keep hope alive. It's the only reason you keep publishing. It's like, maybe this time.

[D]

Well, there's also always the papers where you're like, this one's gonna take a while that are like totally fine. And then the ones are like this one, this one's a no brainer that

[C]

take, why take a long time. But that first one, you have a great reason why, because it is really long. And it's actually secretly multiple studies all jammed into one paper.

[B]

No.

[C]

All right.

[B]

Well, no, before we. No, it is before we get kind of deeper into the studies themselves. Catherine, I'm kind of curious because certainly hearing your sort of reference to sort of like teaching, like the study of education, study of teaching skills. When I hear, you know, conceptual learning. When I think about conceptual learning as sort of the behavioral definition, I sort of think of other research in education around teaching, which tends to be more, at least in the behavior analysis field. Like maybe something like personalized systems of instruction. Maybe something like Engelman's direct instruction with examples, non examples that you're talking about that idea of generalization, discrimination, More than discrimination, I would think for the most part, for a lot of those. And then a lot of other studies from education that sort of have, you know, weaker experimental designs. They're either like group studies that really don't necessarily tell us a lot other than this curriculum is not terrible. But does it, why does it work? Why doesn't it work? But that's probably only one corner of it. When you were sort of looking at and getting into conceptual learning. Do you mind sharing a little bit about kind of what do we, what do we have already prior to 2025 in terms of experimental behavior studying this concept? We're going to say concept and conceptual like a thousand times. And triangle. We're going to say all these words so many times.

[D]

Yeah, these will be great. So I think there's so much out there and there's so much out there not just in behavior analysis, but also like in the educational literature and beyond. Like anybody who teaches anything is usually has some idea of what a concept is and what conceptual learning is. And so I'm not about to do this literature based justice. There's a lot out there. There's a lot of different perspectives from the behavior kind of analytic history. This study is sort of a merge of two, I would say two areas of research. One is the conceptual learning literature or the term that's more commonly used, I would say in this literature is concept formation. If you really want another one. I kind of steer away from that one because I think it makes it sound like a concept is a thing when really it's just a behavioral pattern. And so I like conceptual learning, but I think this goes all the way back. I think Mechner is actually the first one to say the definition is that it generalizes to new instances and discriminates from other instances. And then there's that seminal study by Herrnstein, Lovelace and table where they demonstrate sort of conceptual learning with pigeons. And I think that was the first moment where we looked at concepts using behavior analytics strategies and approaches and boiled down to the behavior that we talk about as conceptual learning. And then ultimately that line of research is alive and well. It's led to a lot more about stimulus control, what conditions produce this and like kind of the scope of things we can use to teach this and that we can do this with humans and non humans. And I think that debate is even still very alive and well about the degree to which conceptual learning occurs outside of humans. So if you want to get basic and you want to get nerdy with It. There's a whole group of great researchers in that world. And there's even a special issue actually in jab, if you really just want to embrace jab. I think I want to say 2002, early 2000s special issue on jab that is just all about concept formation, conceptual learning. And it does a great job of just going all the way through that history and providing a nice crash course. It's a really nice special issue. So that's one end. I warned that it's. It's kind of emerging. So the other end is more of the behavior analytic instructional design literature, a lot of which is fairly old, like early 70s. All the way, kind of through the 90s, I think was sort of the peak of this literature. Right. Where like psi, like he mentioned earlier is really, really common. Direct instruction especially starting to come become more prevalent. And this is actually by one of the books is like by Tiemann and Markle, which I believe Markle was a student of Sker. So ultimately this does converge back to Skinner. But basically they provided a book called Analyzing Instructional Content that goes through how to do a concept analysis and then describes the way that that analysis can be used to inform the instruction, which is identical to the instructional approach that I used. So it started there and that there was fairly little research that then followed up on that. There are some more education level group designs like the ones you mentioned before that appeared sort of right when it came out, but it doesn't seem to really have emerged to the groups who could use it the most. So that's sort of long term. What I'd love to see is that everybody's using this approach. It's something that becomes really regular and behavior analytic training.

[B]

Okay, so robust history, but not necessarily one that, you know, the average BCBA or behavior analyst is as fluent in discussing out there.

[D]

Not yet. Yes.

[B]

Not no. Yeah, exactly. Not yet. I am a huge fan of anything that gives me a better way to sound smart. When I tell someone this curriculum you're using, it's awful and it's the cause of all the problems because usually someone spend a lot of money on that. And if that's all I got, it sounds like I'm just a crank or I have a, you know, an opinion that I want to share. But being able to talk about like, well, I mean, your. Your examples and non examples are, you know, widely miscategorized. No, no, no. Conceptual learning is happening like that. Whoa, whoa. So your role this curriculum might be bad. We should listen to this guy. Clearly knows he's talking about, except those are the only terms I know how to use. I don't know how effective it goes beyond that. No follow up questions, please.

[D]

Well, and you know, I could do a whole diatribe on the way our curriculum gets selected and the way that our curriculum gets designed and the science or lack thereof that goes into that. But I do want to like point out too, I think one thing that I commonly hit when I try to get people to use this, especially when I'm talking to instructors in higher education, but also honestly in secondary education too, is the idea that we can use the same approaches across areas. If I go to, and I've had this conversation, if I go to somebody who teaches math and I say, hey, I know something about how to teach math based on behavior analysis, they say, no, you don't. You need to be an expert in math and math curriculum design to teach that way. So I think overlying all of this too is sort of the need for evidence that you can boil down any topic you're teaching to behavioral principles and use the same principles to then do instruction.

[B]

Well, that's always music to my ears. I mean, that's why I continue to come back to behavior analysis day in and day out of. I want someone to explain the rule that I'll be able to use all over the place. You know, for the most part at least, you know, with some, no exceptions, but you know, with some idiosyncrasies here

[C]

and there, what's great about having our conceptually systematic vocabulary allows us to interpret widely the behavioral phenomenon out there?

[B]

I want to answer.

[D]

I also want to take this moment too to point out that just because we're about to talk about this in terms of college students and more advanced learning, we still teach concepts to our itty bittys too, teaching them to respond to the letter A. That's a concept. You want them to find every instance of an A in the world, even when it's in cursive, even when it's up on the road sign. So we teach lots of concepts at all different levels.

[C]

Yeah.

[B]

All right, well, let's, let's dig deep into your studies, which are going to get into the kind of the basic mechanisms of that discrimination and generalization that comes into conceptual learning. So you've already mentioned, Katherine, we can't talk about how to teach a concept with examples only. We need some non examples. And in your paper you describe two different types of non examples that you did a really great job sort of manipulating orders and details and Reinforcement and features of. But we can't talk about non examples unless we talk about the close in non examples and the Super 70s far out non examples.

[C]

It's groovy, man.

[B]

Where I, I know at the end of the day these terms sound way more exciting than I think in terms of. Their definition is pretty, pretty standard as to what they are.

[C]

Just pop that balloon right now.

[B]

I know, I didn't know how far I wanted to go into like the hippie far outness, but you know, we, we got, we got a bunch of studies to go through, so.

[C]

No, well, I, they. These triangles are groovy for sure.

[B]

They're very groovy.

[C]

Yeah.

[B]

But, but what, what is the difference between the types of non examples that you used in your study?

[D]

Yeah, why do we, why are we

[C]

talking about examples and non examples? Can we like back up there?

[D]

Yeah, we can definitely back up to there. Example means it belongs in the class or thinking about it behavior analytically. If you respond to it when asked to find an example of the concept, that response will be reinforced. And we can define an example some of the time in terms of physical features. For example, if you're teaching triangle like just a normal triangle, not the crazy ones, the normal ones, you can say it has three sides and they all intersect with each other. Right? And so those are the features that define the triangle. And then so that's, anything that has those three features is going to be an example and anything that doesn't is going to be a non example.

[C]

Okay, great.

[D]

So those are what those features that I'm saying, the ones that you have to have, they've been referred to as must have features. Great name.

[C]

Right?

[D]

Makes perfect sense. Must have features and then it's contrasted with can have features. So these are features that can change and don't impact whether it is an example. So for example, if we're thinking about our triangles, those three sides got to touch. That's my must have feature. But my can have feature is what color is my triangle? Is it on a computer? Is it physical? Is it on a sheet of paper? Is it, you know, a dashed line? Or is it a filled triangle? Is it a three dimensional triangle? Which I guess we could get in the weeds about whether that's a triangle. But so, and you get the point, all the things theoretically don't change whether it's a triangle.

[C]

Is it equilateral? Is it isosceles? Right.

[B]

I think when it was upside down it blew my mind. I couldn't, I couldn't handle your paper once you started turning them upside down and saying this. I was like, I don't know.

[C]

I'm going to do it anymore.

[B]

In my defense, I did not get as many practice opportunities as the, as the participants. I just had like nine.

[D]

And honestly, truly bless the reviewers and bless the associate editors because they also provided so much feedback to me along the way about how to talk about this better. Because it is really, really hard to talk about in part because we don't have a huge amount of research on this. And so there's not a very standardized literature about how to talk about these things. But concepts, we've got those features that make examples and non examples. And identifying those is actually what's called a concept analysis. Taking the time to go through and make sure that you're getting all of the must have features and can't have features listed so that you can go through. So then you're creating your examples and non examples and you've got to think about what am I going to change for my example to make it a non example? And the key there is that you're going to remove some of those must have features. So thinking back to just a normal triangle, right? If I've got to have three sides and they're all touching, one of my non examples might just have two sides. Or one of my non examples might have three lines, but they're not touching each other or one of them's not touching, right? And so how different that example is from that non example? There's actually two terms that are used. A far out non example is one that's way different. Far out. They're way out there. Yeah. And if we're teaching a triangle, a far out non example would be like a hippopotamus. It's like, this is a triangle.

[B]

Whoa, man, are you seeing what I'm seeing?

[D]

This hippopotamus is far out, far out there. And then a close in one would be like, this is a triangle and this is a triangle where one of the sides is like slightly off so it doesn't intersect. They're just barely different from each other. And so the idea is, in theory, if you use those non examples that are just barely different, you're probably teaching the concept better than if you use those on examples that are really, really far out. Which I think has a lot of face validity to it. It makes sense. We know with the very basic discrimination research, the closer our S minus or S delta is from the S plus, the better that discrimination is. It makes sense. But there wasn't a lot of research to support it, why we did that. And so the key terms here must have features, can have features, close in non example and far out non example. And then I'll throw in one more and that is about the relation between our can have features. So we talked about our must have features between the examples and non. Examples can be different, but our can have features may also be different if they're not differing. We call them matched. So a matched example and a matched to non example have the same can have features but differ in terms of the must have features. Unmatched, everything is different. So we have different can have features and different must have features. All right, that's like a class. I'm sorry. If you're listening to this rewind, take a sip.

[A]

Hi. Do you want to be a bcba, also known as a Board Certified Behavior Analyst?

[C]

Sure. We all do.

[A]

Now you can come to Regis College in Weston, Massachusetts to get your graduate degree just minutes outside of Boston.

[C]

Choose from any one of these Masters

[A]

of Science in Applied Behavior Analysis, Master's

[C]

of Science in Special Education, dual degree

[A]

in Special Ed and aba, or be eligible for your Postmaster Certificate.

[C]

You can complete your degree and be

[A]

ready to sit for the exam in two years. And our 2022 graduates had a 92% pass rate on the BACB exam.

[C]

Come enjoy approved fieldwork placements, ethics, mini handbooks, PhD levels, professors, small class sizes, and a service trip to Iceland if interested.

[A]

And don't forget, our program is accredited by the association of Behavior Analysis International or ABAI as a Tier 1 master's degree program.

[C]

Don't delay. Supplies are limited. Learn more at regiscollege. Edu.

[A]

Again, that's www.regiscollege.edu regiscollege.edu one more time www.regiscollege.edu see you there.

[B]

Hey everybody. Sorry to pause our conversation, but I just want to remind you that ABA insidetrack is ACE and KWABA approved. By listening to this episode, you're able to earn one learning credit. All you need to do is finish listening to the episode, then go to our website abainsidetrack.com or click the link in your podcast player to be whisked away. Once there, you're going to be able to find links to all of the articles that we discussed today. You're going to be able to find a bio of Dr. Williams transcripts, some follow along notes, and a quiz you're going to need to take. Now, some of the quiz questions will require you to have demonstrated learning around our learning objectives, but Two of them are going to require that you know the two secret codes that we've hidden in this episode. And the first one is Jenna. J E N N A. It could be one of Dr. Williams cats or it could be a reference to Office actress Jenna Fisher. Only we know the truth, but you just need to know that the code word was Jenna. All right, let's get back to our discussion of conceptual learning with Dr. Katherine Williams. Maybe before we come back to the candid must have examples. Because I almost just want to talk about in terms of the like logistics of, of examples. Let's take a, let's take a, take a detour to say, all right, so having all this information, you have your concept analysis, you have what it is you're trying to sort of, you get at the bottom of in terms of the experimental design. Can you just talk generally about what was this experiment? Because it was kind of the same experiment over and over and over that you were iterating on, but it was generally the same type of procedures. Just what features of the concept analysis were you most interested in? Or was it something unrelated to the concept analysis, more about say, reinforcement or motivation? So if I came into the lab and you said, welcome, I'm so excited to have you here, what, what would I then do for this experiment? Like what, what did it look like?

[D]

As a participant?

[B]

Yeah, as a participant, yeah.

[D]

So basically we have college students, they're showing up, they in most cases just sign up for extra credit. So they don't know what they're getting into. And what we tell them is we're going to teach you something, might be something real, it might be something made up. It's up to your job to figure out how to select the correct response and we want to see what you do. We're kind of intentional about not telling them a lot because we know if we accidentally say the wrong thing, it might completely disrael what they're learning. So we want to make sure that they don't know a lot going in. We're just challenging them to figure out how to respond correctly. And so then what goes on. And this is all presented using a touchscreen computer. So they're sitting in there, we're observing them either through a one way mirror or through a camera, sort of depending on which lab we were in, and making sure they stay on task but otherwise not really involved. The computer presents stimuli and they touch to select one and it provides some feedback. They do that for, in most cases, three hours or potentially more, depending on which experiment we're looking at. This is a pretty long study. Although they do get breaks. Then the way that the session is set up is at the very beginning. They have a pretest. They're just responding. They. There is no way that they can respond correctly just because we made this up. So unless they're really, really good at picking up on patterns, which we did see a little bit, there's no way they can respond correctly.

[A]

Those people you're bringing home, you're like, what?

[D]

Yeah.

[A]

What is wrong with you that you

[D]

can pick this up?

[A]

So you're like, we're going to do a whole new study on you. All right. Yeah, you break them up.

[D]

I love that. No, it was pretty impressive. Honestly. I did not expect anybody to be able to figure it out. So that was really cool. Then we do training or teaching. So basically just showing them ones and they get feedback. If they got them right, they earned a little bit of money and they got a nice little trumpet sound. If they got it wrong, they lost some money and they got a nice little buzzer sound now. I'll let you know. They can't go in debt. We have it set up. Everybody's like, when they lose money, what happens? You have to pay you. The IRB would not allow that.

[C]

Right.

[D]

And then when they're responding, I spend

[C]

my money in the triangle experiment.

[D]

Yeah, yeah. It's a gambling problem in triangle. And when they learn to respond correctly, they're doing it. They get a post test which is just like the pre test. No more feedback. And hopefully what we see is that they continue to respond correctly, but now they're getting kind of new stimuli. And so this is why it's evidence of conceptual learning. They've never seen this before. It's just like driving down the road and pointing at a sign that you've never seen before. Same thing here.

[B]

All right, so, so there, there's the gist of it, but the big difference is going to be in what are they seeing in the corners of their screen. And this is where we get our examples. Our non examples are close in non examples. Our far out examples are untrained exam. That's where you start seeing all the fancy triangles. So how the, how the heck did you do all this? Because just reading the pages of here is what they look like. This is an example with a can have feature that's matched not. I was like, I. You know what, I'm so glad Catherine's coming on the show because I'll just kind of, just kind of let that one go. Over the head, check out the results and the discussion at the end, and then I'll ask you what's up with all these examples? How did you make them keep them separate? Know which was happening when? It was like Herculean task.

[D]

A far out fever dream. Yeah. So there were a lot of stimuli and honestly, they've always made sense to me. The struggle has not been organizing them for myself. It's been organizing them in a way where anybody else can understand them.

[C]

I love.

[D]

But basically, I mean, this is kind of the fun, I think of like translational research is really like, creativity is the limit. Like, you get to. You get to go out there. There's no limitation on like, you have to be anything. Like anything that's out there. You pick something that suits what you need for that experiment. And so in this case, what I needed for this experiment was something where I had enough features that they weren't going to learn it super, super fast. Because if they pick it up really, really quick, then I'm not going to see any difference between my conditions.

[A]

Right.

[D]

It needed to be one where I could also create enough different examples for the purpose of doing, practice and training. So it couldn't just be like, if, if it had very few features, there was going to be a very small limit to the number of stimuli I could make and therefore the number of things I could look at. I wanted them to be equally difficult. This is always the fun of doing, like skill acquisition research is you want to have conditions that are independent of each other but also equally challenging. So in this case, the way you could do that is by having kind of three different features that we use in each condition. Three is sort of arbitrary too.

[C]

Right.

[D]

It needed to be something where I could show a difference between close in and far out. So I think one or two wouldn't work for that. So three was the magic number. Again, like some of these decisions you just make and if they work, you stick with it. If they don't work, you change them. And this is not the. The triangles that were published were not the first iteration either. I just want to point that out, right? Like, there was a lot of trial and error that went into the creation of these, some manipulation of the features, but basically that's the logic. It's sort of like a giant. One of my grad students called it a giant Sudoku game where basically you need to have enough degrees of freedom to have different stimuli, but also they all fit together and they show different patterns.

[A]

That's like a. That's like A mind death for me to be on.

[C]

You reminded me of Spot it. You guys played Spot it before?

[A]

Yes.

[B]

Oh, yeah, with the cards.

[A]

Catherine.

[C]

I'm not playing Spot. It's Spot it. There's a. It's. It's simpler than, than your thing, but the little. They're little circular cards and they have probably seven or eight different symbols on them. It's like a frog and a tree and boa. And then you put out two cards, and each card in the whole deck will have one similar symbol as each other card. I don't know how that. They do figure that out how to do it, because there's probably like 20 total symbols. So when the two cards are out, you have to look at both of them and find the same feature across the two cards. And then it's a kid's game, so whoever does it first gets the card or whatever. But as I was attempting to look at the different symbols and so the symbols are like, it's a triangle, but then the triangle has like a, like a line going through it. Is the line connecting? Is it. Is it floating? Is it outside of the triangle? That's number one. Then it has like some little symbols in it. There is a little rectangle in one, but then in another one there might not be a little rectangle there. Instead it's a little triangle.

[D]

Right.

[C]

And then is there a line going through some other shape in there? So there's like all of these features that you're trying to identify, some of which are going to be, you know, putting it into the positive category of. Of. Yes. I don't know. I don't think the. The correct triangles didn't have like their own name, right?

[D]

No, nothing. Nothing that did. We tried to stay away from naming.

[C]

Right. I want to name everything. So. So either those little indicators put it into the yes column or some of those features automatically put it into the new column. So it was very. As I'm trying to like compare across all these triangles, I'm like, this is just like spot it. Like which of these P is the feature that is in both. That is the piece I need to attend to. So it's a little bit like that game.

[B]

So Spotted is identical matching. I think the only difference is orientation or size. It's all the must have features. So you need. Although now I'm thinking, do we want to go into business to make the conceptual learning Spot it game? That feels like. I know, Catherine, with your mind for examples. Non examples. And our knowledge of the game. Spotted, I guess is all we're bringing to.

[C]

This is what we're bringing.

[D]

Like the dairy. Little side hustle going on.

[C]

It's love.

[D]

Spot it.

[A]

I think, actually, Diana, you guys gave me spot it.

[C]

It's possible.

[B]

I have an extra in the closet upstairs.

[C]

We even have world's tiniest spot it. It's like the size of a quarter. You gotta wear your glasses for that one.

[D]

Yeah, that sounds horrible. What do you call it? Concept it.

[C]

Yeah.

[D]

No.

[B]

Yeah, There we go. Yep.

[C]

I like it. I like it. Let's take this convo offline.

[B]

Yeah. Edit this out. This is a great idea. Gonna make billions.

[A]

Nobody market this.

[B]

So. So gather. We have all these different examples. They have examples, not examples. Must have features. Can't have features. You like you described like Dave, this guy's lots of triangles. Lines in triangles, lines outside triangles.

[D]

Right.

[B]

And then you ran six different experiments with variations on these things. Now, I already mentioned. I mentioned before we started recording. I am always T. T, T. When people put more than one study in their study and. And six. Like, I don't. That's. That's a lot of. That's a lot. That's asking a lot of your reader. But, you know, why.

[A]

Why did you pick six? Like, why didn't you decide to publish them separately?

[D]

Great question. You know, my. My tenure documentation would be a lot better off if it's for six applications over one.

[B]

Everybody wins, but every tiny experiment gets its own paper.

[D]

Yeah, it is sort of interesting. So just to be transparent, this started out as two studies. The first two studies, study one and study two, were my dissertation. Great done. Dissertation check. Let's write it up. Let's write it up and let's submit it to JM And I did that. I already knew what studies I was going to do. Afterwards, I start writing some additional studies based on Lou's original findings and jab reviewers, as they often do. And I've been this reviewer too, to others said, this is great. It's a good design. It seems to be getting at the right thing. But there's all of these other variables that could go into play, and you don't really look at them. And you stop short of a complete analysis is a common line. Right. Because.

[C]

Well.

[D]

And you say, oh. But I say like, oh. Every time I read those reviews, even though they were like, do more work, they got me excited because I'm like, you're right.

[C]

Right.

[D]

There's a lot more things that I can do. And I'm glad you're excited and you're motivated for me to do them. And so really studies 3, 4, 5 and 6 were in the works and were follow ups on those first two because ultimately the different results we got between those. So in one of those we got like really, really robust conceptual learning. It seemed to be like not just happening for everybody, but happening and then facilitating the next condition, which is bad from an experimental control point of view. From a. From a learning point of view, it's like, holy cow, if we can teach this concept so that the next one's easier to learn, we're set. Because eventually they're going to be learning these concepts like this. But to figure out what differences between those studies made that kind of difference in outcomes, you have to isolate them and look at them one by one. That's what studies 3 through 6 did, is isolate different ones of those variables and look at them one by one. It wasn't that I sat down and I said I'm going to do six experiments and this is what they're going to be. I sat down and I said I was going to do two. And they kind of led me to do the rest of them. And I think that's where the experimental analysis of behavior is awesome is. Your data tell you where to go next. And eventually you get enough data that you have a full story that will then be useful for people who just want it to work.

[B]

You know, I'm going to say the organism is always right and letter to the editor always leave them wanting more. I want to read experiment one and then be like, I can't wait for the sequel. Then I get that one. And then when you're done with the whole series, it comes to a satisfying conclusion, then fine, then we can publish them all and like the deluxe version, you know, the greatest hit, like the hardbound copy of, you know, the conceptual learning study by Dr. Katherine Williams and

[C]

signed by the author.

[B]

Yes, signed by the authors, special edition. You know, that's what I want. So I'm just, I'm going to write

[C]

them that this way you have to

[B]

get butts and seats for, for jab.

[C]

You know, Williams et al. 2025. Like that's going to like encompass a lot of this research in one citation.

[B]

But you can also Williams and all A. Williams and all B. Williams that you next level stuff. But anyway, that's my personal preference for studies. I love lots of little studies that lead. Tell me a story. But you know what? Let's just pretend this was the deluxe special edition and I'll get over it.

[D]

And I do challenge you if you could Only be one study. What study would it be? Right? Do you feel any one of these would leave you satisfied? Like how you were satisfied when you left the six?

[B]

Yes, yes. I got to. I got to remind myself of which one. No, there was 100% I want to see not there. Honestly, I think it was. I think it was four or the arbitrary stimuli. B. I think those are the ones with the. Where you sort of saw the interplay between the order of the close in non examples and the far out non examples and the order in which they were presented and the pros and cons of teaching that way. I think I. I'm going through my notes real fast. I think that's. I think that was number four. Yeah, that was. That. That was the best one, I thought. I don't. I don't want to say you peaked with. With number four, but it sort of did feel like the Marvel movies. Like, you got to Avengers Endgame, and it was like Billion Dollar Idea. And then you had some others. Like, these were good. But I don't know. I still remember Experiment four. Man, that was the one epic battle.

[D]

Until you've tried everything else, you know.

[B]

That's true. But I still nostalgia remember rose colored glasses. Experiment four is where I was really.

[C]

This.

[B]

This is. This is my jam for conceptual learning. And then 5 and 6. Like, these were good. These were good. They kind of tell me a little bit more about what I already learned, though. That high. High.

[C]

Come to know and love.

[B]

You couldn't quite see it again.

[D]

So what I'm hearing is that you did four studies.

[B]

If you got into four and then been like, you know, boom. It was like. I'll be honest with you, Ken. It was a real crescendo. It was like a real crescendo of study design. And then I don't.

[A]

I don't.

[B]

I feel bad being like. And then five and six. What a letdown. Like, oh, let's. Let's get back to basics.

[C]

You're tying up some loose ends.

[B]

Well, yeah, it was a lot of loose ends. It was sort of like an app, like, you know, for completionists only. It's like, I read the Lord of the Rings and then do you want to read J.R.R. tolkien's like, seven other books about, like, nitty gritty details of hobbits and junk? Like, you know what? They're cool. They add more. But if you just stop to four, you'd be like, good to go. And then the discussion put it all together.

[C]

So see, I. I liked experiment two, where we went into. With the paramecium and the plant cells and everything. But I think I liked it because I was able to, like, like, identify the. The concepts much easier, which I think was potentially limited.

[D]

Part of the problem.

[C]

Yeah.

[B]

I think Experiment two was a sequel. It gave you something you didn't know you wanted. It was really an audience pleaser. But I got to tell everyone, you better make it to four. Maybe you won't feel the same way as I did, but that's. That is my. My recommendation. And then if you're like, I love conceptual learning, you go. You go on Do 5 and 6, you add the rule egg study later on, you know, you got it all. Then you're. Then you're having a good time.

[D]

You can honestly skip three, too.

[B]

Okay. I was. I was good. I didn't want to say anything. I feel like I've already been, like, doing this metaphor where I'm like, your experiment. I didn't like that. Like, And I'm so sorry. Like, we. You're a guest on our show. I'd be a kind of a jerk.

[D]

I mean, I started this off, though, by saying people could read the discussion and nothing else, and I would be fine with it.

[B]

I know. I. And I'm gonna chow. I'm gonna challenge you back. Catherine, if you don't read Experiment four and just read the discussion, I think you're missing out. I think you are Ms. Study. I think that that was. I think. I think if you want to go 1, 2, 4, and then maybe come back to 3, and then 5 and 6 if you're, like, loving it.

[D]

Okay, that's starting to sound like Star Wars.

[B]

I mean, you know, everything comes back to Star Wars.

[C]

I'm.

[B]

I'm of a certain age where everything's a Star wars reference. But why don't. For folks who haven't read the study yet, or maybe they read it and they're like, I kind of need a little more background because.

[D]

Or maybe at this point, don't want to.

[B]

Maybe they don't want to. I'd love to just hear generally about, like, we don't have to go into, like, elaborate detail for each one because, like, like you said, we can just go to the discussion at the end. But I'd love to hear just a little bit about, like, what was the iterative process as you were building through the six studies? Because I was not.

[C]

Joe.

[B]

I mean, I was really not joking with the. There is a crescendo of, like, organized study throughout. Throughout the paper, which is very, very Cool to see. It's like you, you see the thought process. But I kind of love to hear from you how you went from experiment one to two to three. Like the broad strokes of what you were looking for in each experiment.

[D]

Definitely. And full disclosure, they are not in the order that they occurred. So I will. And I think that's often the case with articles with multiple studies is you arrange them to tell a story that you wish you had thought of at the start because they just flow better.

[B]

So I don't want to pull us back to Star wars, but George Lucas himself will tell you that every movie is made in the editing room. That's how you get the idea. So you did it. You brought us back to Star Wars.

[D]

Experiment one was just asking the question of can I, using controlled environment with arbitrary stimuli, demonstrate that close in non examples work better to teach concepts than far out non examples? The answer was it did great for two participants. And then the other four are messy, so you stop and you study it. Right. Why did those two work and the other four didn't? And one reason that we think is like motivation. Right. I'm asking you to do this weird thing. And even though in that case I was paying them for responding correctly, still probably, you know, they're just getting paid basically minimum wage. Like it's not that big of a deal. You don't necessarily want to work that hard to learn these made up things you're never going to use again. So what if we try it with real concepts? That's study two, where basically we just mapped it onto different kinds of cells, biological cells, which if you're wondering why cells? Why out of all of the things, first of all, they're visual. The reason visual stimuli are so nice is we don't have to worry about prerequisite skills as much. There's a wide range of reading skills, reading comprehension skills with undergraduate students. So we don't have to be concerned about that if we're just doing visual stimuli. Also, this was in 2020, so Covid was very on my mind. Cells were very on my mind. So that's why. Cells, right. What she thought about doing a con. One of the conditions was Covid, but we thought that would at that point would be a little too insensitive to our.

[B]

You wanted to kill motivation to learn anymore. I guess I don't want to do this experiment. Keep your 1440 or whatever.

[D]

It was definitely that study. Basically we got really robust conceptual learning. We got more conceptual learning with the close in non examples and the far out but we got a lot of conceptual learning across the board. We lost a little bit of the experimental control. We were like, okay, back up a step. Why did experiment two work so well and experiment one didn't? Experiment three through six were looking at different possibilities. Experiment three, I'm making sure sometimes I mix up three and four, so if I flip them, I'm sorry, but experiment three was looking at how different motivational attributes could play in. So just basically, if we arrange so that they're earning more money for responding correctly during the tests, if we set this up a little bit better, does this work better? It didn't matter. It didn't just seem to be a motivational issue. Experiment four was when we started to mess with the stimuli even more. So we made. I won't go too much into the details of how we changed the stimuli because I think when you can't see them, it just doesn't make sense. But if you're curious, go and look at them. But basically we made some changes to the stimuli saw for that may, in theory, that would make the stimuli more similar between the triangles and the biological ones. Just in terms of the number of features and the way the features were changing. It did not work like worst outcomes. Those poor students did not learn those concepts. Experiment five, we made different changes to the stimuli. Again, just trying to outline what is different between those cells and what is different between those triangles, other than being cells and being triangles. And that one worked beautifully. We got results that were a lot more similar to what we got with the cells, which is really cool because it showed it's not just inherently that these are real and these are aren't, and things that are real are easier to learn than things that are fake. It's that there's different feature arrangements that matter when we're teaching concepts and we need to think about them and we need to program our instruction accordingly. And then experiment six is just then trying to see if by teaching with those ones that worked really, really well, they could pass the test on the ones that worked really, really bad. Basically, seeing if by teaching you the must have features using those easier stimuli, can you tell the difference between the must have and can have features with those stimuli that didn't work? The short answer is a little, but not really. We need something more. There we go. That's the story is that hopefully that sort of sees. I mean, it's logical. They do flow into each other. Each one just sort of led to a different question. But you need each of those pieces to really understand all the moving parts.

[B]

So just having feedback was not really any sort of a predictor, like, how much money, little money, whether you knew you were gaining money, whether you banked the money and could keep it for next. Like, nothing really impacted learning. No.

[D]

And I mean, we probably could have, like, the feedback was not informative other than like, I mean, it was basically just like the feedback you would get in a rat lab. Right. Like, you get the food or you don't get the food, you get the money or you don't get the money, you get the buzzer or you don't get the buzzer. It was not meant to help directly teach anything conceptual, except to just differentially reinforce things that worked and things that didn't. And I think that would be, you know, anybody wants to go out there, do research on this, like, that's a whole area that people could go into is like, how could you provide feedback in a better way to facilitate the learning of concepts?

[B]

Yeah, I can kind of see a combination of the rule and the feedback of, like, almost. But this one was missing blank, blank features as facilitating learning versus, like, money, no money, loud noise, different loud noise, kind of kind of feedback. So at the end of the day, skip it to the discussion a little bit. The close in non examples. If you presented the close in non examples and had practice with the, you know, your examples and your close in non examples, while that led to a lot of errors, it would lead to a little bit longer time. Not always, but. But often would lead to some, like, longer practice sessions. The close and non examples were kind of the magic sauce of if you learned the examples and close and non examples, you could discriminate those. You would learn the concept. You demonstrate learning of the concept, the discrimination, the generalization. But it wasn't like full proof in that regard. Like, I know some participants would have weird patterns of like, did they develop their own bizarre superstitious rule? Maybe their responses were better than, but not at a mastery level. Do you mind going through some of the key findings overall when this was done in terms of what do we now know about conceptual learning that we didn't before?

[D]

I think the biggest key for this, just because it is fairly a new line of research, is we need to think more about concepts when we're programming our instruction. Be it for early learners, be it for older learners, be it for something simple, complex, we need to think through what these features are that are important and use them in our instruction. I think this paper provides some guidance for that. Close and non examples seem to Work better then a. There's a lot of other variables that we're picking during our instruction that need to be looked at in just the same systematic way so that we can create ideally this instruction where we very quickly, very reliably can create something after a concept analysis and it works for most learners. Then we teach those discriminations so so fast and so so quickly and so so well for our learners. Concepts are everywhere. So that would. That kind of precision in our instruction would really benefit a lot and also just I think expedite a lot of the procedures that we want. If we have rules about how to set that up versus Just, just do your best until they reach master criteria. Yeah.

[B]

And well, speaking of rules, I know we probably won't give it certainly as much time as you know, the larger paper, the longer paper, but where did the study on like that rule leg system. So providing rules of what the examples are. So basically taking the experiment you'd already done and saying hey everybody, here's what we're looking for. Take a look at this triangle. We want this and this and this in it. If it doesn't have that, it's not the right thing. Don't click on it. Right. It was kind of that next thing was that happening in between the other six studies. Was that something that came later as sort of I wonder if rules kind of just add to this.

[D]

I think it happened after, I think after where it might have now I'm pretty sure it happened after. I'm sticking to it. So. And the reason is I wanted to make this study as basic as possible in the sense that actually originally study two was going to be with non humans. I realized that that was going to be beyond the scope of what I wanted to tie my graduation to and sort of pivoted a little. But I wanted to keep this very very tied to as close to a basic laboratory non human experiment as possible. But the reality is verbal behavior is amazing. We know that individuals will behave according to rules very very well until those rules are no longer in alignment with the contingencies. And so basically I just wanted to see okay, if I provide rules that just lay out the things I need to know. So what the can have features are what the must have features are and then talk them through an example and a non example, does that do all this forum right? Like does that work out? Because if so it's super super practical. And although not all of our learners are going to be able to pick up multi step rules like that super super fast. So it's good to know how to do this just with like practice and differential reinforcement. A lot of them can. And so that's why we did the second study or the second paper, I guess.

[B]

So rules good.

[A]

You.

[B]

It was kind of some of the, some of the results there.

[D]

Yeah. Try not to completely spoil it.

[B]

I did really like how you sort of set up the experience so you'd have your like practice first, then rules and then rules first and practice and then, then the practice. And certainly, you know, we already let the cat out of the bag if you do rules. And you, you described the kind of. The PowerPoint. It was like the equivalent of looking at the first, the first six experiments page of stimuli and being like, hey, dumb, dumb, check this out. Like, this is what you need to pay attention to. This isn't what you need to pay attention to. I don't know if those are the language you use. Probably it was nicer. And then you sort of went through no. And then you do your post test or oh no, there's the mid test. You sort of your mid test to see did this impact learning and then the post test after if it didn't. But it was interesting to note and I'd love to see if this were true for like an experiment two type of like a real learning rather than like an arbitrary stimuli learning test of even wasting people's time by being like, try this and see how it goes. And they're failing over and over. Didn't make it impossible. For then when they finally did learn the rules to produce learning of the concept, which I guess that was good. I mean, you could have had just a bunch of angry people who are like, I'm not doing this experiment anymore once we get to the rules. Although I guess you then you brought up the question and I also had the thought of why would you want to ever teach like this if you know that the rules work? Why would you want to be like, I'm going to let you suffer for a bit. And then maybe you did. You were gentle about like, maybe it increases motivation to know that there are rules afterwards. I don't know, guy. But you know, I don't know if I believe that.

[D]

Yeah. And I think just to be. Just to be specific too, it's not any rules. I think part of the major contribution of that paper is saying if you have rules set up this way. So like with the rule egg sequence, which is just an acronym that combines rule with egg, which is short, for example. So the rule egg sequence, not egg like the food, but Egg like the figures.

[C]

Like e. G. Right?

[D]

Yeah, like eg.

[C]

Yeah.

[D]

And really it's rule, egg, neg. So example and non example. But I think it is important, like, you still need to do the concept analysis that we talked about earlier. You still need to be really thoughtful about what rules you're giving, and you need to give that example a non example. But I think it's also sort of cool because if you think about it, it also goes against what we think of as bst. So behavior skills training. The very first step is instructions and you give it an example that's ruling. And this makes an argument passively. I never explicitly said this in the paper. In fact, this was a point that was made by some of my colleagues at a seminar recently. But this points out that you don't actually always need the practice and the feedback to get learning if you have these rules and examples set up this way. So I also think there's potential research out there where you look at bst. If you do BST with rules like this, do you really need to go through the practice and the feedback to get.

[B]

That actually brings up a good point there, Katherine. And I feel like we've seen that a little bit in. I'm blanking on what the term was. But when instruction manuals have like a visual component as well, so there's sort of those like step by step and then there's an example, non examples. Usually it's just examples. So that. That might not always work because as we know, we do need some non examples. But you know, you think about. You see a lot like to dos, like computer to dos, like, here's what it looks like on the screen and here's where you click. And I was always sort of struck by the idea of like, but BST is so amazing. Why the hell would this. Like every other manual of how to do things doesn't seem to lead to changes in responses or mastery of a skill. But when they add kind of the example pictures and visuals, that those do seem to get better results from the trainees. So maybe, maybe that's kind of that sweet spot. But again, I don't think those usually have non examples. So what's the mechanism there? That in this case examples plus rules works pretty good. But in other case, maybe it's not concept formation in that regard. We're just teaching a task analysis so it's a simpler set of steps.

[D]

Well, you're probably still teaching. You're probably actually teaching a conceptual chain, right? So like if this, then do that. So you have to make a discrimination and then based on that discrimination, you know what to do next. Usually most like task analyses are discriminative chain unless there's just no decision points at all. But it is an interesting point about, you have the instruction, you add the visual, you add the example, but you don't have the non example, I imagine, right. Like you said, it probably works better than not having an example at all, but not as well as if you had that non example there. And even textbooks are a great example of this, right? Like usually you spend a paragraph talking about a concept, you get lots of examples, and then later on you sort of passive aggressively get some non examples because you talk about a different concept. I love that it is, it's like it's there and they sneak it in and it's like they never explicitly say, like this isn't that, but it's something different. And so it is sort of interesting. Like even with textbooks, I feel like if you actually put these things side by side and we're like, this is this because of this? And this is this because of this. And if this had this, it wouldn't be that. You would probably learn it a lot faster. And in fact, a lot of students learn to create those visuals for themselves. Or we guide them to do concept maps or logic maps that allow them to create that for themselves. But why we don't just give them it that in that format? I don't know. I need to write some letters to some textbook companies.

[B]

You know what, that's a, that's a great point, Catherine. I feel like so many concepts, especially when we're talking about the more, you know, fine grained concepts, even in behavior analysis, I know I spend a lot of time, to this day spend some time doing the. Yeah, but how come this one does that? How come this one isn't an example? Because, you know, you'll bring up a phrase, you'll talk, this is an example. Oh, it has these features. And then everything feels like it fits under that category until no, well, that doesn't count because. And then they give you the re. And then, and once you start hearing all the reasons that whatever you suggested is a bad answer, then, oh, now I understand the concept. But you had to go through those extra steps. And if I had just kept my mouth shut and not tried to like ask, you know, to, to clarify, I probably would have gone through life being, well, everything counts as a, an example of, you know, discriminative stimulus or, you know, what, whatever concept it might be.

[D]

So you're providing your own close and non examples.

[B]

But if you. But I didn't do it, then I wouldn't have my close and non examples. I would just have examples right until I took the test. And then I'd have some, you know, far out or close to non examples on my, like, F paper. Like, here's where you fail. Which again, talk about motivation. I don't think we added that into the, into experiment seven of like. But you, you're a failure. Here's how do you feel about learning more now? Like, that's not fun.

[D]

Well, and, but they're thinking of multiple choice questions, right? Those are great examples of how stimulus control can get wonky because like, think about all those times you knew the correct thing to select in a multiple choice question for some reason other than the reason that was intended. Right? So like, you see those and you're like, oh, I'm picking between this one and this one. This is the only one with a period though. So it's probably that one or this is the longest one. So it's probably that one or the Right. Like it's not. We've all done this and that. It's the same principles though. So then we have some issue. Or there's a feature that's telling us what to select that's other than the one we want to be that feature. So you're right. It can both point out something that you didn't learn or that you should have, but it can also teach you something that you did learn that you shouldn't have.

[C]

Well, all right.

[B]

You got to compare conceptual learning using these techniques and some sort of vibes based learning program and see which one comes out on top. Yeah, I'm guessing it's. I'm guessing it's yours.

[C]

Well, we gotta pull in.

[B]

This feels like a good time. Yeah, I mean, we're kind of already jumping into it, but let's, let's kind of come back, summarize a little in dissemination.

[C]

Station trains pulling out. We're having two running behind it. We're throwing our luggage on.

[B]

We're talking about our favorite experiments like Star wars movies, which, which experiment was the Empire Strikes Back of.

[C]

All of these made Catherine talk about so many nerdy things.

[A]

She loves it.

[B]

We almost missed the train. Let me finish my. Yeah, we almost missed the train. We're so in ration in this conversation. We're arguing over the rankings. You know, when it comes down to it, Catherine, I, you know, you mentioned in the discussion, close in on examples. Boom, there's our magic. Add some rules, Bam. Now you've got yourself a magic way to teach all the conceptual learning. However, it's not necessarily that simple. Like you discussed, we need conceptual analysis as well. So you know, when it comes down to if you knowing what information is out there now you were asked to based on these sort of pillars of these are going to be the key concepts. What would you say if you wanted to develop a lesson that was going to be a slam dunk? Everyone learns a conceptual lesson. What do we sort of know right now has to be a part of it and what do we feel? Our vibes based instructor maybe tells us you might want to put some of this stuff in like what do we kind of know should be a part of those slam dunk lessons, but maybe not fully know.

[D]

Yeah, and I'll throw in my answer here. Is also going to be informed by two additional papers. One is under review and the other one is we're still writing it up. So we've done more research looking at other variables beyond these. So this is throwing those in there too. But I think the number one thing honestly is just if you're teaching a concept, take a moment to acknowledge that you're teaching a concept and then do a conceptual analysis. Because even if you don't put that in your instruction yet, it's going to impact the way you talk about it with your students. It's going to impact the way you think about just changes the way that you teach. I think if you're like, I need a 10 minute solution, not a 10 hour solution. Do a conceptual analysis or think through a rough conceptual analysis and then go from there. If you're ready for it and you want to design all your instruction this way, you're like, I'm sold. What do I do next? I think use your conceptual analysis to create two examples that differ from each other as much as possible and a couple of non examples. With those non examples, you want them to be as similar to those examples as possible, except for the things that matter. And then use those in your instruction. And as much as you can use those in whatever practice you do, whatever test that should be your gold standard is can you tell the difference when there's only one thing that's different, do you know which one of these is the case? Then bonus, bonus, bonus points if you just want to go all in. Most recently we found that students actually do better if you gradually add in those can have features. So rather than starting with just the whole shebang, this is what it looks like in the real world, we're going to look at some examples and non examples. Start with overly simplistic examples and overly simplistic non examples that only have the must have features. And then once they can do that work in the can have features until you're getting to the okay, this is what it looks like in the real world and that helps. So I think I gave like three levels of implementation there. Mix and match as you feel, but just think more about your stimuli. That's. That's it. That's what it boils down to.

[C]

Good advice.

[B]

That's awesome. So Katherine, you have, you have the two studies coming up that will hopefully be. Be out in publication in the near future. Kind of what's next for conceptual learning? Like where do you want to see people putting their, their research muscles over the next couple years?

[D]

I feel like we have a good formula. I want to see it used. I think more research and application. So try this out. Do it in a classroom, do it in a lab space. Do it with early learners, do it with adult learners. See where this works, where it doesn't, and see where, when it doesn't work. Let's look at why. What is happening there. Is it that the conceptual analysis is really hard to do? Well, is it that we're ignoring motivation? Because in case you, you didn't notice, right? I'm pretty much ignoring motivation here. So, like how we motivate students to respond accurately and try it out. I think demonstrating the scope and the span of this is kind of the next step. And we're doing a little bit of that work coming up. In my lab, I have some undergraduates who are sort of doing honors theses where they're just going to try it out for their favorite class and go through it.

[C]

All right.

[B]

It's free labor for their professors, I'm sure as well. I love that if it works, I

[D]

don't know if it's free. A lot of writing, editing goes for free.

[B]

A lot of learning that goes into the process. You can benefit. It's always nice when you have a product at the end of any sort of thesis or hard work. What can I use again in the future?

[D]

Well, but honestly, students are better at creating and this is a whole nother thing. But I think if you can incorporate students in the creation of the examples and the non examples, they're going to help you out because they're going to pick topics that are interesting to them, that are culturally relevant to them, and they're going to point out in that kind of creation, what they understand and what they don't. So I think like, there's a whole nother level of like, does actually creating these examples and non examples further facilitate conceptual learning and motivation and everything like that?

[B]

That's, that's true. I'm always a huge fan of. I like, I like any sort of team or having a large number of people because when someone asks me to make examples or non examples, I can come up with two off the top of my head. And then I'm like, that's all I got. I guess that's, that's the whole concept until, you know, I'm testing people out of them, like, oh, I didn't teach this very well. More ideas, the better the examples. But you got to be smart about how you use them.

[D]

And I think it's also just cool if you want to get metal with it to like show students that learning and teaching is a science. It is data based. The way we do it is intentional. And so the more we can incorporate that into our teaching, the better too.

[C]

Awesome.

[B]

Well, Dr. Kathryn Williams, thank you so much for coming on the show. Thank you so much for telling. Talking to us all about conceptual learning, sharing about the process of doing all of these different experiments, listening to various Star wars references. I didn't even ask if you wanted to talk about Star wars at all. It just kind of came up. So sorry, that's my bad, My bad on the hosting part.

[D]

I mean, it doesn't hurt. It is like not far from May 4th. So it was about this.

[B]

You know what? This is an evergreen episode. Bring it back every May the fourth. You could just listen. This is the one. This is the ab inside track for you.

[D]

Yeah, you can have sound effects too.

[B]

I don't know if our editor adds in some lightsabers or, you know, that'll

[D]

be study beyond study 2. As mentioned, you just start hearing Darth Vader at an increasingly loud volume.

[B]

Catherine, if folks want to reach out to sort of, you know, ask questions, talk about maybe their own examples, non examples, and, you know, share thoughts or hear more from you, how can they

[D]

do so please email me. I'd love to talk about any and all of those things. Or if you're just interested in trying to apply this in a systematic way too, I'd love to collaborate. My email is williamscluncw.edu. so Williams spelled normally the letter C as in cat, L as in I don't know long UNCW Eduardo, once more,

[B]

really big thanks to Dr. Catherine Williams for discussing her research, discussing Conceptual learning and for letting us get a little bit silly with the concept. I don't know, I feel like something about jab that always brings a couple more jokes out of me. I don't know why, but we really appreciated her time today. And now it's time for the last segment of our show, pairings.

[C]

All right, it's time for pairings. Pairings is part of the show where I tell you about past episodes you might want to check out. We have never talked directly about conceptual learning, but we've talked about a few things that might be related. Those episodes include number 143. We talked about stimulus equivalence. Number 168, emergent relations with Drs. Leslie Schaller and Brian Blair. Episode 31, problem solving with Dr. Judah. A. Episode 73, general case analysis. And episode 223, generality and generalization. I also like to recommend a snack to go with the episode. The snack today is Doritos.

[B]

Oh.

[C]

Do you know why?

[B]

Because they are examples and non examples of Doritos.

[C]

Kind of. Yeah. Well, they're those crazy triangles, first of all. But your Doritos, you could have the nacho cheese Doritos, you could have cool ranch Doritos, you can have whatever the

[D]

purple ones are called.

[B]

Some sort of spicy. Yeah, hot and spicy or something sweet.

[C]

I'm about maybe sweet spice chili. Chili spice chili.

[B]

Yeah, something like that.

[C]

It's something like that. Right. So, yeah, there's variations in there, but they all still hang together as a Dorito category.

[B]

So what are the examples of Doritos? Because you might say triangle. But what if I got one of those Doritos, it's sort of folded over itself. Well, it's got to be made of corn.

[C]

If you were to flatten it, it would be a triangle.

[B]

Okay. If you flattened it, it would be a triangle. Yeah, it's a round edge fried corn

[C]

chip and it's a delicious.

[B]

And it has some sort of a dusting on it.

[C]

It's edible.

[B]

It has to have dust on it, though.

[C]

Yeah, dust.

[B]

Because if it just were a corn chip, it would be a corn chip.

[C]

Right.

[B]

Gotta have a dusting. What kind of dusting? Any kind of dusting.

[C]

Cool ranch, nacho cheese or purple.

[B]

Yeah, all of those are fine dustings. Those are can. Those are. Can have examples.

[C]

Exactly. But.

[B]

Well, there must have examples. But the color and the taste is a can have.

[C]

Yes, exactly. That's right.

[A]

Yes.

[B]

So.

[C]

So they're a variation. But like Puffy Cheeto. That's a non example. It's not the Right shape. It does have a dusting.

[B]

Not the right shape, but it is orange. It is the same color exactly.

[C]

Yeah. Okay. And that was pairings. Please enjoy.

[B]

I wish we just made Dr. Williams listen to us talk about snack foods and whether they're examples or non examples of that snack food. Now we'll get around. We'll get around for the sequel bonus episode. All right, Just eat snacks. We'll just talk about snacks and examples and not examples of those next. All right. Well again, thank you so much to Dr. Catherine Williams for joining us on the show. Thank you all so much for listening. If you have not, please rate and subscribe to ABA Inside Track wherever you like to get your podcast. If you are so inclined, we would love to have you join us on Our Patreon page, patreon.com ABA Inside Track where you can subscribe at the free level to just get episodes sent to your podcast player of choice. We would love that as well. Also gives you an easier time to get in touch with us though. You can always email [e-mail] but if you would like your episodes a week ahead of time. If you like discounts on CES and if you'd like access to all of our polls and access to our various listener choice and book club episodes as soon as they are released, well, you can join us at the $5 level for those quarterly listener choice episodes where you get a free CE for listening or at the $10 and up level where you get access to our book club episodes the moment they are released. Three of those are only available for patrons for an entire year. And the two CES you get for any of our book clubs, totally gratis from us as a big thank you for joining us on patreon.com ABA Inside Track. Finally, if you are interested in getting CES, you want that second secret code word. It's Maggie. M A G G I E Again, could be a cat, could be actress Maggie Gyllenhaal. There's just no way to know. But as long as you put Maggie, you're gonna get what you need out of it, which is passing your ce.

[C]

Dame Maggie Smith.

[B]

Yes, it could be. Who knows? Maggie. All right, so some final thanks. Again thanks to Dr. Catherine Williams, thanks to Dr. Jim Carr for recording our intro and outro music, thanks to Kyle Sturry for our interstitial music, and thanks to Dan Thabit of the podcast Doctors for his amazing editing work. We'll be back next week with another fun filled episode. Until then, keep responding. Bye.