← AI to ROI
Loading summary
Transcript51 lines
- [00:00]
A
Foreign. Welcome to today's episode of the AI to ROI podcast. Today I am joined by Jim Piazza, the Chief AI Officer at EnSona. I'll be covering four main topics with Jim today. First, the role of a Chief AI Officer and a managed service provider. Second, how to measure the impact of AI investments in an msp. Third, two customer success stories of indie ROI of Ensono customers. And fourth, Jim's lessons learned as a Chief AI officer. So with that, Jim, could you please take a moment to give a brief overview of your journey to becoming a guest on the AI to ROI podcast? Yeah.
- [00:52]
B
Thanks so much for having me on, Ray. I really appreciate it. My journey is a little bit different, I think, than some. I started off as a programmer way, way, way back in the early 90s, and eventually found my way into data centers and network and all kinds of things. Eventually I sort of found my way and found that I really just liked connecting things. At the end of the day, I wanted System A to talk to System B, and I wanted to make sure it was fast and efficient and secure. And so I spent a good almost 10 years at Facebook now meta, working in data center operations, where we started off with a number of systems in place with a number of people, and we had our vice President of Infrastructure come to us and say, hey, we really need to scale operations. Can you help me do that? And we used machine learning back then to do it now, what we all love and appreciate with AI, and so we were able to 13x scale the number of devices we're able to support while we only scaled the number of people by 4x. And so it was really an interesting journey in doing all that, and that's sort of what led me to Ensono, is that they have the same sort of aspirations from a managed services provider. And that's what we're building here today.
- [02:17]
A
Okay, well, deep background in operations and data centers. So. So tell me a little bit about what are the top responsibilities of the Chief AI Officer and an msp, and maybe a little bit about how the role came to be at Ensono.
- [02:34]
B
Sure, sure, yeah, absolutely. So I started off as VP of Predictive Systems and Machine Learning here at Ensono, really small team. And when we were able to demonstrate some of the capabilities and things that we were able to do, we decided to expand the role and make the team a little bit larger. To your first question, though, around sort of, you know, what is the role of a Chief AI officer? You know, it's funny that when people ask me That I say it's a little bit of this and a little bit of that. It's a little bit of sort of the Chief Digital Officer kind of role, a little bit of the Chief Information Officer role and a little bit of the Chief Technology Officer role kind of all rolled into one with, you know, an AI slants to it. So, you know, the role in an MSP of the Chief AI Officer is really about connecting AI strategy and to like operational reality. It's not often, you know, I should say this, it's not often enough, you know, just to ask, what can AI do? The better question is where can AI improve service delivery, customer outcomes, financial performance? So a big part of the job is just prioritization, sort of separating ideas from the valuable ones, quite honestly.
- [03:53]
A
Well, this is the AI to ROI podcast, so I want to try to segment this conversation into two different perspectives. So first of all, the internal use of AI at in Sono and how you're measuring it. So maybe you can talk about one or two examples of how you're leveraging AI inside of your own company, not customer use cases and. And how do you measure the benefit?
- [04:20]
B
Great question. I will tell you that one of the overarching lessons that I think we realized early on, number one, you need to decide what your value metrics are before you put hand on keyboard one. That's really, really important. And it's really important also that you have the alignment with each of the business units that you're working with before you try to do any type of technology implementation. And that might be a little Captain Obvious, but I think it's really important to sort of reinforce that as I think about sort of how we're leveraging AI internally to your question is we decided on those value metrics almost immediately and we sort of understood what they were. The biggest thing I'll say on that is you have to align them back to core business metrics. And we'll talk more about that for sure later. The first example I can give you from an internally facing project is we said we want to be able to predict failures before they occur. So we built a system called Envision Predictive Engine. And what that does is it does exactly that. It looks at sort of all the data coming in from a variety of different sources and says what is about to go bump in the night next? So that's one of the internal use cases another, and you'll sense my operational background here is how do we help make engineers more efficient? They have to go sift through mounds of Data in order to get systems back online or optimize systems performance. And so we built another system called Diagnose now that does that. It will look at any specific incident and say, here are the things that are wrong, here are the steps that you should go take, and here are some things that I would recommend. And so all of those things, both of those things, have proven to provide great operational value.
- [06:07]
A
Wow. Let's double click into that predictive one. We're anticipating issues before they happen. And one of the things I guess I would like to ask you is, you know, what are the leading indicators to say, hey, we identified X number of potential issues before they happen, and then how do you actually correlate that to financial impact of not going down?
- [06:33]
B
Great question. So if you think about systems, right, they send certain amount of telemetry somewhere. And I've always been a big proponent of capturing as much value per byte as you possibly can. And so as we look at the heuristics and the data streams that are coming in, there are patterns in the data. And the problem is that humans are good at recognizing a certain amount of patterns across a certain amount of data, whereas machine learning systems can analyze treasure troves of data that's just impossible for humans to be able to capture and sort of report out on. So we identify what the streams are. We do what's called feature engineering to identify which streams and which components are the most valuable to look at. And then we build machine learning models that interpret those signals and say, hey, this combination of events is likely to have some level of business impact downstream. And so that's sort of in a nutshell, how Predictive Engine does. What it does is looking at those vast treasure troves of data from an impact perspective. There's a couple of things. Number one is if you can prevent or predict an incident from actually occurring, obviously you know, your customer satisfaction is going to go up because you're solving problems before they become problems. You have a much lower risk of, you know, SLA and financial penalties. And overall, it just, you know, helps to build more trust, I think, in the relationship overall. The other interesting thing that I think plays into this is that as you're able to get better and better and get further in advance and further in advance, you know, some of the models we're working on right now are beginning to show signs that we can predict an hour in advance. That gives us time to sort of decide what is the best course of action and how do we want to handle this. So those are Just some of the benefits and some of the things that we have built so far.
- [08:29]
A
Yeah, one of the things we talked about was having this central kind of office of AI or AI center of Excellence. How do you work with the functional leaders to try to help identify a process to use AI to improve the efficiency and efficacy? And what is your role in the office of AI versus the business unit or department Executive lead?
- [08:54]
B
Oh, this is, this is something that we were really intentional about too. So we identified one to two subject matter experts that really knew the business of the business unit. Take hr, finance, legal operations, of course, et cetera. And they have to really sort of know their business end to end and sort of know what all the core processes are. And then what we did is we sat down and we did a value stream mapping to say, all right, where are the biggest chunks of work today that are cumbersome, that take a lot of time, that are prone to human error? And let's map those against sort of the value stream to understand where do we start in our journey? And so we did just that. We had several in person sessions where we sat down with the business unit leaders as well as the subject matter experts to drive those conversations to closure. And we were really excited. The camaraderie, the sort of getting in a room and whiteboarding stuff out, you know, that stuff is definitely better done in person than virtual. Virtual can work, but you have a much better experience doing that in person. But that's the process that we went through.
- [10:03]
A
Right Jim? So when you do that, you might do that with multiple departments, the marketing group, the operations group, etc. Development team, how do you go about identifying the leading signals or early indicators that there's actual business impact here? And how do you use that potential business value to rank stack which projects get priority?
- [10:32]
B
So there's a couple of ways to do it. The way that we chose to do it was to look at what are the things that are going to improve our customer satisfaction the most, what are the things that are going to drive value for our company? And that could either be through increasing revenue, decreasing costs. Efficiency metrics. Think about like all the sort of core business metrics that we look at today. We said, okay, so for this value stream, what are the things that we can hit? And what you're going to find when you go through that exercise is that in a lot of cases, when you identify a candidate for something that you want to utilize AI to help go solve is that they don't just hit one of them, they hit several of them at the same time. So you might find one that drives, you know, meantime to repair down. But that also affects customer satisfaction because if we're reducing the time that it takes for us to close incidents or requests, then you know, they're going to have a better experience as a customer. At the same time, you're going to also be driving your efficiency numbers in the right direction.
- [11:36]
A
So you're telling me that right up front, as you do that kind of value stream analysis, you actually identify and define the performance measurements and how it's going to impact either revenue or expenses, correct?
- [11:51]
B
Right, yes.
- [11:52]
A
And is there a particular project, like if I gave you a billboard on Times Square that you wanted to shout out, say this had such high roi, was there a specific one beyond the predictive identifying of incidents?
- [12:08]
B
I would just say that Diagnose now, I think has that capability to be on the. I'll give you some statistics here diagnosed now, again, putting the right information in front of the engineers at the right time and helping them collaborate. We have seen up to 66% reduction in meantime to repair when that tool is used. So we did some A B testing initially to sort of understand what that looks like. And I can tell you that reviewing the data, it's sort of almost unfathomable how much of a delta that can make when you're, when you're really intentional about what are we, what, what are we doing with the data we have and how do we get it to the right place at the right time.
- [12:51]
A
Now, is this a benefit you drove internally or is this benefit you actually drove for your customers?
- [12:58]
B
Both, actually. So it does drive benefit, obviously for Ensono, but it does also drive benefit for our clients. In fact, we've, we demoed Diagnose now to I think about 25 customers at this point. And I can tell you our hit rates 100%. Everybody has said, hey, for the stuff that we don't post with you, can I get that for my environment? Can you port that over to mine? And so we are doing some POCs with customers right now on that.
- [13:27]
A
Interesting. So when you provide Diagnose now to your customers, who's your primary economic beneficiary? Is it the cio?
- [13:38]
B
Great question. You know, it's a mix, I would say it's a little bit of CIO and a lot of cto.
- [13:45]
A
And is it typically, are they able to measure inside of their own company how kind of shrinking that median time to resolution or repair, do they usually have the infrastructure in place to really tie that to economic benefit or is that something you help them with?
- [14:04]
B
Well, I think it's a journey for all of us. And you know, most IT shops today have some form of MTTR measurement that they use, whether it's for their internal IT shop or whether it's for their, their platform that's supporting their customers. I think, and this is one of the benefits of tying back to sort of industry standard metrics is that if you don't see movement in your sort of core business metrics, you're probably not focused on the right thing when it comes to AI. So most customers have that capability today. How they measure IT and getting the direct financials, sometimes that's a little bit of a journey. But understanding at least the core metric is an important component.
- [14:46]
A
Jim, I was at an event yesterday and one of the topics that came up and we were discussing is the importance of change management and any AI initiative. So can you tell me about any lessons learned and what you do to ensure that the people who are either benefiting from or receiving insights from your predictive engine and AI, how that needs to be coupled with change management discipline?
- [15:12]
B
Yeah, Ray, thanks for that. We actually have another system that we built called Change Guardian. As we all know, being in the IT industry, change equals risk. And so we don't like risk. Right. We like things to be stable. So how do we take risk out of the equation? So we built and it's in beta right now internally within Sono Change Guardian. And what Change Guardian does is it looks across a set of sort of all the available data, including the insights that it gets from Envision, Predictive Engine and so on, to say how risky is the change that you're about to make? And IT categorizes that across multiple dimensions. Like, hey, how busy is the group that's about to make the change? What's been their past success rate? How good is the documentation quality associated with it? Many more to come up with a risk score that we evaluate. And we say the risk is either low, medium, high or extreme. And if it's high or extreme, it's an immediate stop in the process, which is an important component. Right. Because if you're going to use AI within your business to sort of help you work faster, smarter, better, with higher quality, you have to be willing to accept that your processes as a company are going to have to change and be incorporated into this whole advancement. So that's what we did. We said, if it's higher extreme, we're going to stop the change immediately until we're able to mitigate some of the risk. That's one of the benefits of Change Guardian and another, quite honestly is, you know, it takes in some cases for really complex changes. You know, it may take hours to generate a method of procedure. So what we did as part of Change Guardian is we said, look, we do 8,000 plus changes per month. We've been doing changes for 10 years. Let's go mine all that data, understand which ones were successful, and rather than have to build an MLP from scratch, let's leverage what we know across all those changes and automatically build an MLP that we know has been successful in the past that saves a ton of time and improves quality and de risks the change altogether.
- [17:16]
A
Now, do you do that both on your own internal data and also for customer specific data?
- [17:23]
B
So we do that for a customer where we are supporting customers. All of that telemetry that I mentioned before, as well as all their prior changes, we utilize all that data as part of the process.
- [17:34]
A
Anything else you want to share about any customer success stories? Because I'm going to pivot back to kind of lessons learned as a caio.
- [17:43]
B
I'll say this just real quick. Obviously can't mention customer names specifically, but I will say this. You know, we have seen in the last year and a half that we've, we've been sort of full steam ahead on this journey where we had a very large logistics company that most people would know where we host a significant portion of their mainframe and distributed systems, and we were able to detect a problem 144 minutes prior to there being business impact. It wasn't even in our scope of responsibility. We just happened to pick it up through the telemetry and called the customer and said, hey, we think you have a problem that's brewing. They looked into it and said, oh my gosh, you're right, and was able to fix the problem without any downtime occurring. So this stuff is becoming real and I would really encourage like this, this is going to change the way in which we think about sort of managed services providers and the performance they deliver for customers. So just wanted to give a customer anecdote there. Right?
- [18:45]
A
That's a great story. Even outside of your scope of responsibility. Let me ask you this question because you have both your internal responsibilities and your external customer facing. First question is about the internal role. First time you've had a chief AI officer at Indesano, what are your lessons learned and what advice would you give to people who are taking that chief AI officer? And it's the first time it's existed in their company.
- [19:11]
B
That's a really good question because you're right, the Chief AI Officer title is something that is sort of new across the industry. I would say this, I would say start with the business model, understand where you want to sort of make the most impact. Listen for what the pain points are, listen to where you think the most benefit will be. Don't start with the technology stack. Like the technology is not the hard part. It's really understanding the business. In an MSP as an example, my job is to improve the delivery economics and the customer outcomes at the same time. And so we want to pick use cases where the pain is somewhat obvious and the data is available and it can be measured. Right. So that goes back to the measurement piece we talked about before. I think the, the other big lessons learned and where I've seen people fail in the past is they try to do 10 things at once, they try to do 10 pilots at the same time and they go nowhere. Pick one or two, you don't need to go and boil the ocean all at once. Pick something that you think has the highest probability of success, where your data is in the best shape possible and sort of start there. Build your credibility through that and solving those real problems and then the sort of rewards will come thereafter where you can sort of expand what you're doing across the business. So those would be a couple of things that come to mind for me.
- [20:29]
A
You know, I hadn't thought about this before, but as an msp, right, the more cost effectively you can deliver your services, the higher your margins. So was that one of the business metrics that you were held responsible for? How do I deliver our service more efficiently using AI and thus increase my gross margin levels?
- [20:48]
B
I will say this, and I'm going to call this out because you said accountability. I think that it is a shared accountability between sort of a number of folks. So when we talk about operational efficiency, the COO of Ensono and I talk very frequently about how we're measuring it. What are we seeing? How do we want to change? Where do we need to adapt? Because at the end of the day, when accountability is shared across a small number of people, you get a whole lot of buy in and you get more collaboration through that rather than somebody saying, oh, just chuck it over the fence and it's that person's problem. So I think partnering up on accountability, especially on the business unit owners, is really important.
- [21:31]
A
Yeah, we didn't discuss this, so it might be an unfair question, but I gotta ask it because a lot of our larger company audience members, they have a Chief Information Officer and we kind of know what a CIO is responsible for, but now they bring in a chief AI officer and their partners. Right. Any kind of key differences you see between the role of the Chief AI Officer versus the Chief Information Officer, where they have both roles?
- [21:59]
B
Yeah, I do. And I will say, you know, in all honesty, like, it's going to be a bit of a bumpy path to figure out sort of where the lines get drawn and sort of who's responsible with that. I'm really lucky at Ansono because both the CTO and CIO and I are get along really well and we work really well together. The way that we've chosen to sort of delineate the work is the CTO sort of owns the platforms themselves, of which sometimes I'm a consumer, sometimes I'm a builder for the cio. He's the data owner, right? So he owns the sort of data structure and the quality of the data. And again, I'm a consumer of that. And so we all sort of have a vested interest together in making sure that the outcomes of the business are what we want. But we sort of know, like, oh, geez, you know, I really wish we had this data set where this data set really needs to be cleaned up, that I can go to the CIO and be like, hey, I'm seeing this and this is what it's preventing us from doing. Can we work together on getting this in the best possible space? Or maybe there's a new data set that needs to be ingested and cleaned. And so he's really good about understanding the business need and so on. And so we all get together. In fact, we have an internal team chance, the three musketeers, to help make sure that we're able to work together in near real time to solve these problems.
- [23:23]
A
So when you identify a project that you want to apply AI to, whether that's machine learning or generative AI, how's the decision made? What's your role versus the CIO and CTO role for the AI tools and models you use?
- [23:38]
B
Sometimes we always evaluate sort of a build versus buy approach. Right. So is there something that's out in the market today that's proven that it can actually work? And I say that sort of tongue in cheek because there's a lot of stuff out there that doesn't, so beware. But for the stuff that is out there that is vetted and works well, you know, is it worth our time to go sort of build it or should we just buy it? So we make that decision. Regardless of whichever path we go down, the data still got to be in a good place. So when the cio, CTO and I get together and we say, hey, this is a business outcome we want to drive, here's the data that we need, here's the stuff that's available on the market, here's how long it will take us to build it internally, and we look at all that together. We sort of each know what our responsibilities are. If we decide to go buy it, it's probably going to end up on the CTO's desk to make sure the platform gets built. If the data is not available, the CIO is going to go work with his team to sort of go get the data in the right place and shape. And at the end of the day, if it's something we're going to go build, you know, my team's going to get involved and we're going to go build it to take advantage of that. That's sort of in a very, you know, 50,000 foot view, how we look at it and how we do it.
- [24:49]
A
Got you good insight because this is the first time on the AI and RI podcast we've spoken to an msp and it's interesting that you have all three roles and how clearly you've defined and delineated three responsibilities. But like you said, the collaboration is a three musketeers approach. And let's give the audience a chance to get to know you on a personal basis a little bit more with three rapid fire questions. Okay, what are the key variables that have to be in place to ensure your CFO and co can really see the return on investment from AI? And this may replicate a little bit of what you said, but what are those key variables?
- [25:26]
B
So I'd say the first one is clear use, case and scope, good workflow, fit, access to usable data, strong adoption by the people that are doing the work, management discipline at the start. And honestly, willingness to sort of kill the weak, use cases early. Not every idea that comes through is going to be, you know, have a long, dramatic life. So be willing to sort of fail fast and kill things that aren't. That aren't working.
- [25:54]
A
Okay. And for AI initiatives that are led by you as a chief AI officer, who's responsible for measuring and reporting the actual business impact?
- [26:07]
B
Well, the business has to own sort of the outcome and the reporting of it. You know, if I step in and try to say, well, this is What I think the business impact is, you know, that's, that's my perspective. But the business really needs to sort of own the outcome there. The technology can enable that capability, but the business has to own the value realization.
- [26:25]
A
Interesting. So I understand that. And who owns the budget?
- [26:28]
B
So the company budget itself.
- [26:30]
A
Oh, for the AI, let's say we had to spend a half million dollars to implement this AI initiative for that business function. Who owns that budget?
- [26:39]
B
I own that budget.
- [26:40]
A
You own the budget and then you collaborate with the business unit leader to report the benefits. But he or she owns that primarily.
- [26:47]
B
Yep, that's right.
- [26:49]
A
Cool. Okay, last question. We do have some early career professionals out there that are like, you know, how do I really ensure I know what about AI should be learning to enhance my career path early on. What advice do you give to them?
- [27:04]
B
I would say we have, we have a few interns on my team as well because we want to help the next generation come into this. So couple couple thoughts on that is number one, learn how to work with AI, not compete with it. You know, when people say, well, AI is going to take my job, that's not, that's not right. The people that are going to take your job are the engineers and people using AI are the people that are going to take your job. So that's one, I think. Build skills in problem framing, critical thinking, communication and domain understanding. Those are super valuable. Get comfortable using AI to sort of accelerate your research, writing, analysis and experimentation. You know, AI is going to reward people who compare technical fluency with really solid, good business judgment.
- [27:50]
A
I like the idea it's the business judgment as it as much as it is by understanding the technology itself. Jim Piazza, Chief AI Officer at Insano, thank you so much for being my guest here.
- [28:02]
B
Thanks so much, Ray. I really appreciate it. Had a great time.
- [28:05]
A
And to the listening audience, if you're enjoying and finding value from conversations like we had with Jim, Chief AI Officer, go ahead and subscribe to the AI MRI podcast on your favorite podcasting application. Go ahead and give us that five star rating and let us know if you have someone you think should be a guest here on the podcast. Thanks everyone. Thanks, Jim. Sam.