← Insights Unlocked
Insights Unlocked
Building a customer research panel with BECU
00:38:28
Learn how BECU built a high-impact member research panel with UX leader Jessica Schultz. Tips on engagement, recruitment, and scaling insights.
Loading summary
Transcript39 lines
- [00:02]
Nathan Isaacs
Welcome back to the Insights Unlocked podcast. In this episode, we're joined by Jessica Schultz from BECU who shares how her team built a thriving customer research panel from the ground up. From starting as a teller to leading Experience Research, Jessica walks us through the process, the wins, and the lessons learned along the way. If you're thinking about creating your own customer panel, you don't want to miss this episode. Enjoy the show. Welcome to Insights Unlocked, an original podcast from User Testing where we bring you candid conversations and stories with the thinkers, doers and builders behind some of the most successful digital products and experiences in the world, from concept to execution. Welcome to the Insights Unlocked podcast. I'm Nathan Isaacs, Principal Content Marketing Manager at User Testing and our host today is Becky Wright, senior staff Product Researcher here at User Testing. Welcome, Becky.
- [01:02]
Becky Wright
Hi everyone.
- [01:05]
Nathan Isaacs
And our guest today is Jessica Schultz. Jessica is the senior Manager of experience research at BECU, a member owned credit union with more than 1.5 million members and 29.4 billion in assets. BECU is the largest not for profit credit union in Washington state and one of the top four financial cooperatives in the U.S. welcome to the show, Jessica. Hi, thank you for having me.
- [01:33]
Becky Wright
So, Jessica, you've had a really impressive career at BECU and in Experience Research. So could you share a little bit with us about your journey and how it shaped your approach to research?
- [01:47]
Nathan Isaacs
So I started at the ECU as a teller working in our, you know, the very front line dispensing cash to members for their financial transactions. And at, at that point, one of my big joys was to, you know, figure out problems and look at, look at ways that I could help craft solutions either for our members or for our employees to help make processes more efficient. And I was started to work my way into different roles within the credit union where I was able to help with process efficiency and project rollouts. And I had, you know, some success, some failures. And it was about 15 years ago where I was at a user conference and I was in a session where I was being taught about Personas and journeys. And this was the first time I was really hearing about this codified method. And I felt kind of like a hobbit where I was like, wait, this comes in pints. There's like a career in ux. This is a thing. And I turned to my boss at the time and I was like, we need UX designers and UX researchers at becu and I want to be one. So I'm going to go figure out how to get my master's in this and then let's do it. And she was incredibly supportive. And I was able to go to the University of Washington and complete their Human Centered Design and Engineering master's program that really set me up with the. The foundational steps of Human Centered design and user research. The academic side that I was lacking, where I could then look back at projects and say, oh, I understand why this one worked really well. And here are some of those systemic, we didn't know unconscious errors. And to be able to really start to queue up and start to codify how we continue to involve our employees and our members in the projects and the processes that we're doing at BECU and then help to grow out the UX research practice as well as our product design. And so I've been at the company for over 20 years and been able to just grow and really found my passion in understanding users of software, users of systems, to really understand those deep needs and then be able to translate those into actionable findings so that we can work to make things better. So that's really been my approach is that empathy, that deep understanding and then the constant curiosity. That's probably my biggest weakness is I am so curious that any kind of interesting problem is like, oh, I'd like to find out about that.
- [04:34]
Becky Wright
Yeah, I feel you kind of get pulled in all these different directions. But yeah, oh, I love hearing about how everybody, different people get into user research, experience, research. Everyone's got a very different story. So we're here today to talk about one of my favorite topics, and I think it's one of yours too, which is around customer panels. So you and BECU have been successful in creating a member panel. So let's start at the beginning. What inspired you to build your own panel over other recruitment options?
- [05:07]
Nathan Isaacs
Yeah, so we've been a user testing customer for a while and when we first started out, while we are a very, we're a large credit union and large in Washington state. When you think about like the panel within user testing, we were finding as we tried to create screeners and identify BECU members that there was an opportunity to get more of our members to kind of come to the panel. And so we started looking, working with our account reps and looking at the custom network and thinking, gosh, this might be a good avenue for us to partner and to create an opportunity so that it would be easier for members to provide feedback and for us to identify and talk to members in moderated and unmoderated studies. And so that was what we were Looking at, we had done quite a bit of successful direct to member recruiting and we, we realized that we'd have a pretty big efficiency play to be able to take that recruiting from, you know, two to three weeks down to, you know, two to three hours by, by investing in creating a, creating a panel and then inviting our members to come in and provide feedback.
- [06:31]
Becky Wright
Amazing. Yeah, as I say, it's that kind of, you know, difficulty when you have that kind of slightly more niche recruit or finding those customers and getting out to them. What were some of the really early decisions that you had to make perhaps around who to include, how you were going to start recruiting those members and say perhaps any other tools you were going to use.
- [06:54]
Nathan Isaacs
So one of the earliest pieces that we needed to do is to really work to make sure that we understood exactly how the custom network worked, how that building out of the member panel would look like, what was the invite process going to look like, the signup process going to look like, and then what were the different, you know, the different steps along the way? Because we needed to work with our legal and our security areas to really ensure that this was something that we felt comfortable doing, that we could invite our members to participate in. And so that was the first piece was to make sure we really understood how it worked, get all the documentation. We actually created mural journey walkthroughs, journey maps of what was this recruitment process going to look like, what was that experience going to look like. And then we did a lot of competitive analysis of other firms that have similar types of panels that people can sign up for to really try to understand what information did we need to make readily available to our, to our members so that as they considered the invitation of joining the panel, they would feel equipped and informed and have their questions answered about how would we be dealing with their data, with security and trust. And so we did quite a few design exercises leveraging the journey map that we created of what we wanted that ideal state to be like and then leveraging the competitive analysis that we've done of the other companies sites to really identify, okay, what does this landing page that the invites are going to direct our members to need to contain. And so we built that out and it's on our public website and people can take a look at it. And we did some user testing of the page as well to just make sure that our hypotheses of what was going to be important, that it did resonate and had some iterations as people reacted to what was on the page and what they would want to see. So once we had that ready, we had all of the approvals, then it was the time to start thinking about, okay, how do we do the recruitment? And this was a powerful partnership with our marketing department because we realized we wanted to have our panel open to all of our members. We want to hear from all of the diversity of voices and life stages and people, businesses, consumers. We wanted to hear from all of those voices. And so when we were setting up our panel, we, we did a lot of work to figure out what kind of information do we want to collect to allow us to filter. And so some of that, some of that work was. Was important because we knew we were going to invite the entire membership. So setting up our, our, our filter settings so that the, the survey that the members fill out when they join to collect that information so that we could, we could filter on. On them. And so that was, I mean that was some of the stuff that we, that we did. And then once we had that ready, then pages built turned on and then we were able to start inviting members. And we did we do that through direct links in our member newsletter. We'll have Calendar times about 3 times per year that will include information on our. On our homepage to let people know that the panel exists. We do a fair bit of social media marketing as well, letting members know who follow us on our different social channels that this is an opportunity. And then we work with our frontline staff team members. We work with our frontline team members in both our branch locations and our contact center that if and when they're talking to a member who has a lot of feedback to provide that we also let them know that the panel is something that could sign up and join that we would welcome their voice to be part of that conversation.
- [11:12]
Becky Wright
Yeah, this is bringing back lots of good memories of just the amount of effort and the scope of all the different teams that you have to get involved. It really is kind of like a massive effort. And I guess sometimes it's underestimating how much time it does take to even get to the point of. Of launch.
- [11:35]
Nathan Isaacs
Yeah, we ran it like a full project. We had our kanban boards and kind of figuring out what needs to be done. It took us about six months just based on availability and the partnership to get it built, tested and unlaunched.
- [11:56]
Becky Wright
Yeah, it's definitely a labor of love, I would say.
- [12:01]
Nathan Isaacs
So.
- [12:01]
Becky Wright
Yeah. Could you walk us through? I'm interested to hear this in particular is how do you keep the member panel engaged kind of over time? Are There any things that have worked really well for you? Anything that perhaps hasn't worked so well.
- [12:16]
Nathan Isaacs
So one of the biggest things we do is to keep launching studies, keep launching studies that are available for the members, for the members to take so that they can be providing feedback about items that they care about. And so that's probably our biggest thing that we do. We have our panel. When you set up panels, you have two choices. One is to have the PII accessible and one is to not. For privacy and security, we chose to not have the member emails visible. So our, our way that we communicate with the people in the panel is by launching studies. And one of the things that we needed to do, we found out at the beginning, is that with custom panel members, what we needed to do is help teach and coach them on how to take unmoderated studies. It isn't a, any of us who've taken one who've been a participant on the receiving side, especially that very first time, it's a little bit like, what am I doing here? And we noticed that when we first launched and we were running some studies, we noticed that it was like, oh, we need to write these studies much differently than the way we had become just kind of comfortably accustomed to launching studies to people who've taken more than one. And so we really had to help engage the panel by providing super clear instructions we have in our screener to take a study. We make sure that they know that they're going to be on camera and that we're going to be asking them to talk because a lot of people are thinking it's, oh, it's just going to be a survey which you're not going to be on camera and you don't have to talk. And so we put that, we put that again in the instructions that we're going to ask you to speak aloud. We describe what speak aloud is. And then in every single question, you know, if it's a verbal response question, we coach them like, please speak your answer. You know, please write your answer. And that we found from again, engagement has been really helpful. And then when we have members, you know, complete studies where you'll potentially, it's a little challenging to pull out what they were trying to share. We'll reach out to them to the, through the platform and provide them some coaching, coaching notes of like, hey, you thanking them for the time that they've invested. And then in the next study, you know, usually it's the reminders to think aloud, which is just not a natural. It's not a natural action. Anyways, as researchers when we do moderated studies we're often needing to remind people to, you know, what are you thinking? So those are the ways we've kept the members engaged. And then on our, on our website we keep a, we have a little spotlight where we talk about the different projects that we've launched, where the members voices have been instrumental in, in the, in the product launch and, and we, we, we test pretty much anything that we're going to test. We test it with a member panel first and then oftentimes we'll also be testing with you know, what we would consider prospective members that aren't members yet. And so we look to include their voice in all of our studies.
- [15:46]
Becky Wright
Amazing. Yeah, as I say, I guess for a lot of these members this might be the only kind of panel they're a part of which is say, is so different from perhaps some of the other people in the rest of the user testing contributor network. But yeah, I love the idea of them being able to see the impact as well. And generally as I say, people just want to do really well and help you. So I love the extra coaching. I'm sure they're that, you know, they're really happy to receive that feedback.
- [16:15]
Nathan Isaacs
And the unexpected benefit is that changing the way we write our protocols to be more instructive on what are we asking people to do has actually improved the output of the studies that we run with non becu member panel people. So it was a win win where we didn't think about being more articulate and, and very specific in that this is the task and so it's been helpful.
- [16:44]
Becky Wright
Amazing. Yeah, I think as well, like it's that participant experience is kind of slightly removed from all of us and you know, you test it yourself but you're so used to your questions and your tasks at that point. So yeah, I love it when we kind of put the participant kind of front and center. What were some strategies that you've used to perhaps ensure that the panel is representative of your kind of member base?
- [17:12]
Nathan Isaacs
So like I said, we continuously recruit and we have it open to all of, all of our membership and then what we, you know, when we are reviewing studies and we look to see, you know, how has the age spread of people who've responded, how does that look, how does the gender identification, how does that spread, how is income looking? And look to see how balanced of a sample can we get. Our biggest strategy to have it representative is to keep inviting, keep that invite up and running right now we're in the middle of one of our campaigns to invite members to come and join and I checked this morning and as of right now, we're at 558 members that are part of that panel. Now, as we said at the beginning, you know, that's a small, small microcosm of what we have. But that, that's our, you know, our, our current, our current, you know, best way to, to, to. It is the more we can increase the panel, the larger the representation and making sure we're inviting all. And then oftentimes, if we need to balance out, then we'll use prospects from the user testing panel. If we're, if we have, you know, if it was weighted towards younger BECU participants than older participants and we want to look at that. So in our studies we'll, we'll do that. But our biggest strategy, to keep the panel representative, invite everybody and, you know, keep inviting and hopefully it continues to, to grow.
- [18:54]
Becky Wright
Yeah, it's definitely not like a one and done like as soon as you've launched it. It's definitely a lot of extra work to keep it going. And yeah, I find it addictive seeing the numbers go up. It's, it brings me joy when you see it keep going up. So we're wondering, how do you balance transparency and privacy whilst managing participant data and consent? It's obviously a big, big tricky one to make sure that you get it right.
- [19:26]
Nathan Isaacs
Yeah. And as I shared, one of the choices we made at the very beginning as we were setting up our panel was to not have personally identified viable information visible to us on the BECU side. And so the information that we are looking at and that we are collecting as they set up the first survey to join is, is more on the demographic side but not tied to any personal information. And then when we think about running studies, we're really careful in how we word our studies that we're doing. We make sure that we're always testing with prototypes, that it's test data and we're careful in the way we structure questions as well, because our intent is not to, to not to have members disclose any sensitive information and we have protocols in place to delete that if it was inadvertently shared. And so we make sure that our members, that they're very aware as they join the panel that they're going to user testing as our partner site from our BECU page and that they're agreeing to the privacy policies that are tied to the member panel with user testing. And we make sure, like on our landing page Some of those questions are this is a partnership with user testing so it's very clear that members understand how, how that, how that works.
- [21:03]
Becky Wright
Perfect. Yeah. So kind of that upfront information. But yeah, again you can never kind of remind people too much, I think so amazing. You have this member panel. How does your team use these insights that you're getting from the panel to inform the decision making across bec product or experience teams?
- [21:27]
Nathan Isaacs
So often it depends, it depends on the project and what stage we're in and where we're at. But we often do generative moderated interviews with members out of the panel. We were exploring some concepts just a bit ago where you know, having the, having the product owners be able to observe and sit in on the sessions where these are exact users of the products that they're looking to continue to enhance and improve. To be able to hear, you know, how, how they're using it today, what are the opportunities, you know, kind of their wish list, what's that magic wand. And so you know, then you're, then the product owners are able to take that away and inform the roadmap. Okay, this over that. Because with product roadmaps it's always a prioritization exercise because you can't do everything all at once kind of thing. Lots of good ideas out there. So we definitely use it to inform prioritization. And then the other piece is, especially when we're looking at changing elements of our digital banking, our online banking or our mobile banking, it's really critical. We can run a basic usability study with a participant who just banks. But to understand the criticality of how is this change to a platform that you're used to using daily, weekly going to affect you. It's been super helpful to be able to run usability studies with members that are using that platform and to be able to figure out what, what works and what doesn't and how we can make sure that our changes are positive. So yeah, that's, you know, that's what we kind of use. Like I said, we have a policy right now that any study we run we look first to recruit out of our member panel and then we fill in with prospects as needed.
- [23:38]
Becky Wright
Amazing. Yeah, I guess they're probably. People get very excited when they're watching actual actual like members using, using the products I guess related to that. Have you also faced any challenges around internal buy in for I guess the investment in the member panel and also using the panel and if so how did you address them?
- [24:06]
Nathan Isaacs
So this is where I think we're really, we're really lucky as a credit union, we're owned by our members and we are, we're a member obsessed culture. And so that idea of what do members think, what do members want, what do members need? That desire to have that deep empathy is pervasive throughout the organization from the top to the bottom. So truly didn't have any challenges. Probably the, the biggest thing, and I kind of addressed this at the beginning was that we needed to make sure we really understood how it worked and have all of that documented so that as we went to the different groups to get the approval, they felt like they had all the information to be able to make the decisions. And I wouldn't say that that's a challenge because you should do that anyways, but that was the, everybody thought it was a good idea and then it was like, okay, well exactly how does this work? And so that was our biggest opportunity was to make sure that we had it all flowed out. We had captures of all the screens that everybody's going to look at. Like I said, we had that journey map of what was that experience going to be from if they got a social push when they followed it, what would that look like to the engaging and taking that first test, receiving that first honorarium payment. You know, what, what did all of, what did all of that look like? And so crystal clear documentation was what we, what we needed to do. But the ideal, the, the like, hey, let's invest in it. We, you know, we want to hear our members voices. That wasn't a challenge.
- [25:52]
Becky Wright
Amazing. I guess probably a lot of people are quite envious of that position. With, with the buy in, did you find it any, any challenges I guess with moving to using the custom network versus I guess previously using more of the user testing contributor network or again did people kind of take to that Quite naturally.
- [26:12]
Nathan Isaacs
The, the, the first concern was how long is it going to take to fill this study? Like, like hey, I hear you. I agree in theory, but I'm working on a very tight sprint cadence. You know, I need to, I need to fill this study quickly so that I can, I got the scheduled date of when I'm going to be synthesizing. So that was a concern. And what we found is that as we started running studies they fill pretty quickly. Now of course the exception to that is if it's a more niche product type usage platform that sometimes those, those studies can take a little, take a little longer to fill. And so at the beginning it was really kind of that test and learn. Okay, let's launch this study. Let's see how long does it take. And now that we've. And then continuing to add more members to the panel, you know, that helped it as well. And now typically 24 hours. And our studies have filled from the member panel participants and so it's slightly longer than what you might see if you were using general population. But we can account for it in our schedules and it hasn't been a problem. But at first it was kind of learning like how long are these going to take? And then sometimes where there was really tight timelines, we would then launch to two audiences, both to the member panel and to the general population so that we could do synthesis on the general population while we waited for a member panel to come in. And then it's nice because you have that comparison to say, okay, general population usability. What did we learn? Member panel, what did we learn? What if anything are the differences when it's someone who's much more familiar with using that platform.
- [28:15]
Becky Wright
Yeah, as I say, I guess with any kind of new process it is like managing people through that change. And I can very much relate to that being a concern about, you know, I need the data. Like I don't want to wait. And it's kind of. Yeah, it's a little bit scary at that first bit. I guess with you, you need, you need to kind of test it out to know what the SLAs are so you can keep people involved. But it's amazing that it's getting to like 24 hours. And I love, I love the idea of doing it kind of with both the general population and your kind of member panel to I guess alleviate any fears and as you say, get that extra kind of segment and look at the data.
- [28:58]
Nathan Isaacs
Really early on we were able to have a huge success where we needed to rather quickly pull members in for some moderated interviews. And we were able to identify and schedule recruit members to be in moderated interviews within 24 hours. And previously when we needed to do recruiting outside with sending email invites that it just wouldn't, that would not have been possible. And so that real quick win and be able to share that so that people were aware that also helped alleviate that concern of how long would it take exactly.
- [29:40]
Becky Wright
Yeah, when you get those wins, you have to just keep shouting about it, which if you're not used to showing off a bit, it's a bit more difficult. But yeah, we do something similar with like this was the fill rate, this was how long time it took and you kind of, you know, get people to on your side and trust it. Yeah, that's a big win, I guess related to that and a big one that people I guess think about when they embark on a big project like this is around. What particular metrics or signals did you decide to look at to then evaluate the success of the member panel?
- [30:19]
Nathan Isaacs
So our first metric is just number of members that have signed up. Then within the panel we look at number of active participants, how many tests, how's each of the panel members been able to take. We also look at how quickly are the tests filling. And then our final measure that we look at is I talked about the filters that we have that kind of talk about the who from a product makeup is in our panel. Then making sure that that's balanced and growing is also a piece that we look to see. Like how successful is, how successful is the panel? How representative is it of our member base of who we've gotten to sign up? So that's what we look at and we measure to evaluate it.
- [31:19]
Becky Wright
Amazing. And do you get to kind of shout about these metrics? Have you found a good way to kind of share or is it kind of more within the team?
- [31:29]
Nathan Isaacs
We do report out on the, on the member panel. We usually do that about, you know, once a quarter as far as like, how is it growing? And it's really tied to the when. When are we actively recruiting members to be part of it? If we're looking at, you know, how is it growing? Because we don't, we don't have a megaphone on the opportunity all the time, people would just become, become noise. So it is, it's more tied to once the campaign is over then looking to see, okay, what was the impact, what is that looking like? And to be able to report on the overall growth and health of the panel.
- [32:07]
Becky Wright
Amazing. And for those that are listening and we're thinking or are thinking about starting their own panel, what would be a few must dos, perhaps? What are some things that you think they should definitely avoid?
- [32:24]
Nathan Isaacs
So the first thing I would start out with is that, that same process that we used of really make sure you really understand how the, how the, how the custom panel works. Do the work to map it out that you understand what that flow is, what that process is. Look to see what should your journey look like? Really, really think about what are your, your, what are your customers, what are your members going to need, what are going to be their questions and concerns and how do you package up the information so that it's available to them? Those are all key pieces before you turn it on. And then that think really critically about how, what, what demographic data do you want to capture so that when you're sending out studies and you have the screener that you're not having to filter as many people out through the screeners because it can be kind of deflating if you get an invite. Hey, there's a study. Oh, you don't qualify. Hey, there's a study. Oh, you don't qualify. When we, you know, we think about engagement and kind of talking about that a couple questions earlier, that's probably the easiest way to start to disengage people because it's like I click on your link a couple times and you tell me I'm not, you know, you're not the one I'm looking for. All right, well, I'm probably going to either opt out, unsubscribe, or not click. And so that, that other piece too is especially with your, your studies is really look to see like how broad of an audience can I look at? Because we, you do want that participant experience to be positive and to look at how you're launching studies too so that if you do need to have one that's super niche. Did you put in your demographic filters so that you can do that filtering on that end so that the invite doesn't even go out to people that are not who you're looking for? That I think is really important. So we spent quite a bit of time on the demographics because that was our goal, is that when we invite people we want to have to use screener questions as little as possible to, to ensure that they're, they're able to, you know, to have a positive experience and be able to participate in the study because they obviously want to. They clicked on the link. So we don't want to, we don't want to turn them away. And so that's the, that's, that, that's probably what I would avoid is especially at the beginning is having some, you like super niche studies. One of the things we, we, we, we've done too is that we have a, we have a general study that pretty much anybody who signed up for the panel would be able to take and that one is kind of always on. And the goal is that people again have an immediate positive experience. Being able to provide some general high level feedback as well as gets them into that first study where we start to condition them so that they understand, oh, I got to talk aloud. Because that's just awkward. Is so awkward. Like it's Me and the computer and you're having me talk aloud. It's just not a natural. It's not something natural. So having that kind of practice is also helpful.
- [35:50]
Becky Wright
Amazing. Yeah, I think as well like having that kind of, you know, as small a window as possible from kind of signing up to then getting an invite, you know, is a great idea, I guess for having that engagement. And yeah, I don't, I remember we spent a very long time as well thinking about our kind of screener survey and you know, it's that balance between asking too much and too little. But I 100% agree with kind of trying to make sure that you can screen as little as possible and try and do that up front. You know, these people are, you know, really important and putting them first is, you know, should be front and center. And it sounds like it is at becu. So I think that's kind of all of it for the questions. I really enjoyed our conversation today. How does somebody who wants to learn more about yourself and the team at Becu, how do they, how do they find that out?
- [36:55]
Nathan Isaacs
Well, if you want to learn more about me, I do have a LinkedIn profile and you can, you can, you can look there and you're, you're welcome to direct, you know, direct message me and I am happy to, happy to answer any questions. As far as becu, I would say definitely check out our website. We are a growing, growing team, growing practice and definitely would welcome anybody to take a look, especially if that idea of being member obsessed and really looking for ways that we can focus on the financial well being of our members in our communities. And that's what we're out to do.
- [37:39]
Becky Wright
Amazing. Well, thank you so much for being on the show.
- [37:43]
Nathan Isaacs
Thank you for having me. It's been a delightful conversation this morning. Want to keep the conversation going? You can find the show notes@usertesting.com podcast if you haven't already, don't forget to follow us on Apple Podcasts, Spotify, Overcast or Google Play so you never miss an episode. And if you enjoyed today's show, please share it with a friend or leave us a rating and review on Apple Podcasts. And until next time, this is Insights Unlocked, an original podcast from User Testing.