← Embracing Digital Transformation
Loading summary
Transcript94 lines
- [00:01]
A
Why do I want. I don't want to contaminate, I don't want it to hallucinogen. I literally want that. And if I can do that with a smaller subset, so I start getting deeper with a smaller subset and slm. Here we go.
- [00:18]
B
Welcome to Embracing Digital Transformation, where we investigate effective change, leveraging people, process and technology. This is Darren Pulsford, chief solution architect, author, and most importantly, your host. On this episode, embracing the power of small language models with CEO of a proio, Chris Carter. Chris, welcome to the show.
- [00:45]
A
Thank you, sir. It's great to be here. It's an. It's an actual honor. I'm. I've been following you for years with intel and so. So it is. This is truly an honor for me.
- [00:53]
B
Oh, Chris, don't. Yeah, yeah, you'll be disappointed at the end.
- [00:57]
A
So.
- [00:57]
B
So I'll just give you that up front. But hey, before we dive into the use of small language models in this whole new way of thinking about things, instead of Big Goliath, you know, we'll talk about in a second. Anyone that listens to my show knows that I only have superheroes on my show and every superhero has a background story. So, Chris, what's your origin story?
- [01:23]
A
Oh, my origin story started on a Commodore Vic 20 in 1986 or 1984. I got my Commodore Vic 20 after I traded it in from when I was at Radio Shack. Got rid of that, got the Vic 20, got the Apple IIe. I was very fortunate. I went off to Georgia Tech and I tried to learn as much as I could about computers and technologies. And I got my first, first paying position at Coca Cola next door to our campus as an intern, dealing with punch cards and dealing with monolithic. And by monolithic, I mean in their basement, monolithic mainframes, they had an old SAP environment. And so I started playing on that with nowadays, thank God they're not there anymore. But I was playing, I was learning as much as I could. I leveraged them and lo and behold, all those years later, I got into cloud computing. I created the first SAP cloud for a client back in the day. I started playing with AI back in the 1990s. Already people didn't know what it was back then, but really it was available and people were starting to leverage it and use it. It wasn't called AI back then, but now all of a sudden everybody's calling it artificial intelligence and AI. But yeah, I really started back in the 1980s and just had fun programming and learning. And that's for me so for you.
- [02:55]
B
For you software development was a passion.
- [02:59]
A
Yes sir.
- [03:00]
B
Oh yeah, that's the same, same with me. Yeah, see I was like, I could not get my hands on enough stuff and I luckily had some opportunities to get some really cool machines really early on when I was in high school.
- [03:15]
A
Yes sir.
- [03:15]
B
Starting with a Vic20 like you did. It's the only way to start, man. With the reel. The real cassette tapes where I started.
- [03:23]
A
Remember the cassette tapes? I remember my girlfriend yelling at me so many times, what is on here? Oh wait, that's my program. Don't do anything with that.
- [03:32]
B
Don't do was not a tape I, it was not my love song tape that I gave to you.
- [03:37]
A
Right, that's exactly what it was. I remember getting in so much trouble because I couldn't find a non program tape. So I used her, she used to make me these cute little tapes.
- [03:50]
B
Well of course, yeah, sorry sweetie, that's.
- [03:53]
A
What I had to use back then.
- [03:55]
B
Oh, the good old days. If our, if our grandkids are listening to this. Yeah, that's, yeah, that's pretty funny. So this whole thing around generative AI, really, I mean you and I both know AI has been around since the 1960s actually when they first came up with some of the theories and ran some of the things but it really kind of just pittered along because the hardware couldn't really keep up with the theory and things like that. And even when the hardware could keep up, the lift to actually program in AI was pretty substantial, absolutely massive.
- [04:36]
A
I don't know of any other companies other than your former firm. Some of the Google stuff that they were doing when they pre started trying to get all that data. That lift was massive in the beginning.
- [04:48]
B
But now ChatGPT launches on the world and democratizes AI and all of a sudden people are like talking, oh that, that model has a billion parameters. Oh a billion, that's a lot. Now we're talking trillion parameter, big, large language models. And now it's like, well maybe I don't need to go that big.
- [05:15]
A
That's exactly where my mind went. I was like, I don't need a boiling ocean. I've been working in this little area called SAP, very micro focused and I look at it, I started thinking I don't need to boil the ocean, I just need to boil that database. That's all I need is I need that SQL database or I need that Oracle and now HANA database. Why do I want, I don't want to contaminate, I don't want it to hallucinate I, I literally want that. And if I can do that with a smaller subset, so I start getting deeper with a smaller subset and slm. Here we go.
- [05:54]
B
So it's, it's really interesting the direction that this is headed, which is not the direction I thought it was going to head. When was it? Two and a half, three years ago? Coming up on three years in November. Or is it four years in November? It was November of 2022. Two. Yeah. Or 2021. Yeah. So, but that's really interesting that now we're going back to. Hey, wait. Some of those first models that I can now run on a laptop.
- [06:28]
A
Yes.
- [06:29]
B
May have some really, really valuable content in there.
- [06:34]
A
You just look at some of the tools that we get to play with now instead of having to worry about all the Nvidia chips and all these big super processing and all this, you know, distilbert, you do that or Tiny Burt and you're playing with that just from a text clarification. So you're literally taking different tool sets. You're. And the great thing in my opinion is you are not dumbing down. You are just simply consolidating on what you need to focus on. And I, I say this to my daughters when they're doing their education and other. Why don't you just focus on what you need to work on? You don't need to worry about everything else. Just focus. And it's amazing how we can do that now. With us, it's fun.
- [07:27]
B
I like how you said that because it all comes to constraining the model. I don't have to re. I don't have to fine tune it or anything. It's just, I tell it, hey, I want you just to focus on this little bit of text. I don't need, I don't need you to go look everywhere and come up with creative ways of doing accounting. No, no, no. I just need you to take this text field and tell me what it means in this context. Right. I mean, that's simple.
- [07:57]
A
That's. It's that simple. Plus, you can do that with, for your code generation. You can do that for your offline AI. You could do that for Document Q. And you can do that with so many different things. But it's literally you and I, you're so. You and I are so on the same. It's just this piece. Just focus. I don't care what you're using with one of the new tools, focus on that component. I don't want and I don't want. I, I Tell this to people that I work with all the time. I don't want everybody else's garbage. There's enough garbage in that database that.
- [08:30]
B
We haven't cleaned or they haven't cleaned.
- [08:32]
A
Or they haven't done anything for. Let's just have our own garbage, clean it and we can use that.
- [08:38]
B
All right, so let's go over some of the use cases, specifically in SAP. If I'm going to use a small language model basically as a natural language processor, because that's kind of what I see a small language model is, hey, you can understand text.
- [08:56]
A
Yes.
- [08:57]
B
And I can. And I can ask you to put it into a form. Form that now a computer can better understand or that I can use programmatically. So to me that's the sweet spot of small language models. What can I do with that? Now where would you use that in like an SAP environment?
- [09:18]
A
So the first thing that we would do with an SAP environment is we break it up into the two sides. You've got technical and functional. Technical. When you're going through code generation, I want to understand and I want to make sure it's on the code that's being utilized from that landscape. If I start, like I made reference a couple minutes ago, if I start using everybody's code and I'm going through the code process from a code generation, I'm going to get things that are not applicable to that landscape, not applicable to that application and it's going to really cause havoc. Especially the fact that no SAP system is exactly the same, including what's going on when them with SAP forcing everybody to go up to the S4. Hana, right now, even some of those are on different levels. And if I'm playing and I'm utilizing a CO generation for an ABAP code or Java and I start leveraging that and I'm getting others, I'm really going to do some damage to a landscape. And when 87% of all financial transactions in this world go through an SAP environment, if I start screwing up my environment or my customer or somebody else that we were working with, that's going to cause a lot of headache. So I look at that from a technical side. Stay, stay from a CO generation side. But then I look at it from the individual end user. So let's say I'm working with somebody and they're in the HR department and they have to leverage what's going on. It's funny, I was just talking with somebody from SAP on this, this go forward strategy just a couple of weeks ago when I was down in Orlando. You look at the HR side, one individual probably tracks all of your time off, how many days you've got left, how much you've accrued, how you're going to accrue that if you haven't used it by the end of year, is it a use it or lose it? Now you're starting to build up a solution that gives you, from an agent standpoint, to be able to go through and grab that entire HR database and start understanding who's doing what in what department. Because I can't have two people off at the same time here, I can't leverage this component. And now it gives you some flexibility. You're digging into the data, you're leveraging it, and you're making recommendations to that user that, hey, I've got two people that went off on the same day. Even though this one asked off earlier and this one asked off later, there's an overlap, we can't do that.
- [11:53]
B
Yeah. So that's an interesting use case, one I hadn't thought of before because normally I would hire some consultants to come and write a little app that would check my databases. I would give it the constraints. So as a programmer, I would interview, you know, the HR people or the company and say, all right, what constraints do you want on having people taking vacation time? And. And then we would code that all up. Right. That's how we would do it in the past and it would take two or three months and some testing and then I'd have to maintain that code as they changed their ideas. But now with a generative AI, I could, if I set it up right, a small language model, I could ask it, hey, can you make sure that no one's taking that this, that I have coverage on all of my departments when people are asking for vacation time and I don't need to write any code at all?
- [12:54]
A
Nope. And you're basically leveraging the generative processes that are there. You're building it with a 365 calendar and you're leveraging everything inside the database. So that individual, all they have to do is ask it the question and.
- [13:14]
B
It would be, can I take time off?
- [13:17]
A
Can they take time off? And you could deploy that if you wanted to, to each one of the individuals within your company. That's part of what SAP is doing with their jewel, which is their built their in house AI toolset. Now we're trying to take that a little bit further and we're trying to see how we can go a little bit deeper and Be a little bit more granular with those activities. They see it as an overarching tool that can help everybody. I see it as something that can really be a benefit to those end, end users who literally have to do the day to day and still give executives that beautiful overview of every painted glass and all that kind of stuff that everybody likes.
- [13:58]
B
So.
- [14:03]
A
Foreign.
- [14:09]
B
So this is really fascinating because we talked about code generation, we talked about not needing code at all, right. Because now I, but the, the hardness is in the details.
- [14:22]
A
Oh yes, yes.
- [14:23]
B
I, I can have a small language model sitting there, but if I can't get data to it or any action items to it, then you know, it's, it's just a natural language processor with no way of communicating.
- [14:38]
A
Oh yes. And I, I'm firmly believe that individuals and organizations need to do a couple of things, especially if they're going to be transitioning with simply their data. If they're going to go out and they're going to use the chat hubs and all that stuff. Okay. That's getting outside the, the world of their in case. And most enterprise companies want to keep it within their walls. They want to only use their data. So you've got to start cleansing that data. I call it, I like to say, fine tune your data. If you have, if you've got a, if you've got a six terabyte database, something's probably wrong. Start cleaning your data. Fine tune that data. That's where all the little jewels and the nuggets and that's why people go out and they pan for gold. It's that goal that gives it once it's fine tuned and cleansed. But then you've got to be able to understand how to deploy that properly and to be able to use those nuggets to give you the understanding of what's going to happen and how that's going to happen. But then at the end of the day, if you're not continuously keeping that up, I kid you not, it's amazing. People say, oh yeah, we, we did a data cleansing project five years ago.
- [15:52]
B
Okay.
- [15:55]
A
Even since. And how are you going to lose that? You can't use that data because you haven't done anything with it for those five years. And then it literally just comes back to what.
- [16:07]
B
I'm glad, I'm glad you said that because I think people forget that. Right. You have to constantly be cleaning your data and it's not a one time and done. Right. Because there's this law called the law of entropy.
- [16:21]
A
Yes sir.
- [16:22]
B
That's out there and chaos. Just everything moves to a state of chaos. Whether you think it does or not. It does. Right. I have noticed that in my own life. Just in, just in my house, right?
- [16:35]
A
Yes.
- [16:36]
B
We have to clean it every week. Right. It's not like I cleaned it five years ago. That's good enough. Right.
- [16:45]
A
So I don't have, I don't have any more children in my home. I'm sure you've got a couple more left in the house.
- [16:52]
B
Just for two more months. Just for two more months and then they're all off to college. It'd be awesome.
- [16:58]
A
I have to deal with dogs. And they're messy. They're messy. Like messier than databases.
- [17:04]
B
Well, and, and that's a good point too, that, that clean. The cleansing of the data, is it. It also goes into training people that are generating the data and, and using the data. So do you see any place for like a small language model on the front end of data generation from human interaction so that maybe when the data is coming in, it's clean before it comes in? Like taking your shoes off at the beginning at the entry of your house, right?
- [17:36]
A
Well, I take my shoes off at the entry of my house. And it's amazing when you start to find those individuals that you talk to at companies who start to, they start to get it, they understand, oh my God, I got to keep this in. They'll start working through processes of making sure that data and systems from an SLM standpoint, that they can be able to make sure that they're getting better response times, that they're getting more control of their model behaviors, that they're getting more refined components. They're really taking on that initiative up front. So on the back end and through the processes, they're going to be able to leverage it because they, they've seen how a small language model can benefit over a large and how fast and how, how the cost and they take on that responsibility. I, I love doing that with some of my staff because I get to see what they're doing and how beneficial it is. When you take care of it up front, you. It's cleanliness in is cleanliness out. Garbage in is garbage out.
- [18:43]
B
And it's so, so let's walk through this a little bit. You're. You're kind of an evangelist for small language models in, inside your group and things like that. What does that look like from. I don't want to say institutionalizing it, but I do eventually have to say, hey, I've got a bunch of tinkers. You're a tinkerer. So am I. I'm going to use small language models, get my work done more effectively. How do I now tell my co workers that maybe aren't on the bleeding edge like we are? Hey, this is how you do this. And this is you can start creating your own.
- [19:23]
A
I'm a show and tell guy. I go around and I, I'll bring my laptop, I'll bring some components, I'll sit there in meetings, I do lunch and learns and I want to show people, hey, you may not be playing with this yet, but this is something that may help you out. And I'm not trying. None of the team that I have, and I'm very fortunate to have a great team, about 170 people. And I will sit there and I'll just, hey, guys, I was playing with this the other day. Give me your thoughts on this and I'll create a gamma site or I'll create some. I'll say, take a look at this and let's do a luncheon. Learn. Give me your ideas. Some people just aren't going to get into it. And I get it. But then there's those ones that you start bringing over to our side and you're, you just have start more and more conversations and all of a sudden you see that light bulb kick on. And for me, I love being able to help educate younger individuals. And it's probably why you're a professor. You're. You love to help educate people.
- [20:21]
B
Yeah, yeah, absolutely.
- [20:22]
A
When I'm educating them and all of a sudden that light bulb kicks in and you see their eyes and they.
- [20:26]
B
Start smiling or like it is. It makes you feel good, doesn it? Yeah, it does.
- [20:31]
A
It feels so good.
- [20:33]
B
So, okay, so let's talk, let's talk practice now.
- [20:37]
A
All right.
- [20:37]
B
Practical, right? You're running large language models on laptops or small language models on laptops?
- [20:43]
A
On laptops and smaller x86 environments.
- [20:46]
B
And yeah, like, yeah, yeah, yeah. I mean, everyone needs to go out and buy an AI PC in only Intel's lunar lake. Of course.
- [20:54]
A
Of course.
- [20:56]
B
I need that stock to go up, people. Come on, let's go.
- [20:59]
A
Yeah, I need the stock prices to go up. Let's go, people.
- [21:02]
B
That's right. So, but what, how. What models should I be using? What. What have you found as the most reliable models? Or does it really matter? Because I am constraining the models more and more. I'm, I'm doing some profound, you know, prompt engineering. I'm hooking rags up to these Things I'm, I'm really constraining my data that's going into these prompts.
- [21:28]
A
Yeah.
- [21:28]
B
How do I get started?
- [21:38]
A
You know, there's so many different ways that an individual can get started, but the first thing is of course, get started. So I'll, I'll ask them right off the bat. Okay, do you want to like playing with text? Do you want to do some code generation? Do you want to work with some different, what document Q&As, what are you going to play with? And then I start recommending some different SLMs that we like to play with. You know the silver, we'll do five, two, we'll do what mobile birds and stuff like that if we're going to play on some cell phone. And then I want to start asking them some additional questions, why it's going to work, what it's going to work. You know, These beat chat GPT4, these are better to run on these types of systems. These are going to be more local and private, but you can customize it even greater and if you want to get into that realm. So I started asking a lot of questions, being able to really dig into those from a use case perspective and I let my team members, before they even start playing with, start playing in their own use cases, start having the ideas, start getting into it and then let's start figuring out what type of challenges and performance gaps that we're going to attack. Are we going to attack something for one of our customers? Are you going to attack something for yourself that you want to play with, that you're finding out? How are you going to do that from a technical perspective? How are you going to do that from a performance perspective? How are you going to do that from a data requirements perspective? And then I just literally allow them, until they get into a customer engagement to really start going through. And then I always want to make sure they're auditing all. I always want to make sure that what other targeted SLMs, what are their deployment strategies? There's so many different things that I ask them to start to investigate, but it's all about being investigative. I want them to be inquisitive, I want them to start growing some of that knowledge and understanding that. And because it's not just about cost savings, it's about how you can get granular and deeper at a more cost efficient price at a milliseconds that you're dealing with and being able to bring something to actually tangible to the table. It's not just saying, oh, here's some data. Good. Enjoy it.
- [23:54]
B
I really like this approach because what you've done is you've given them another tool. Instead of institutionalizing and saying we need to go buy this big old huge license with OpenAI or with Google or we need to buy a $5 million server with a bunch of GPUs and host our own, you're saying, here's a new tool that does different things. I want you to explore the boundaries of it, see how we can leverage it in our customer sites in it, but still within the, the bounds of doing good software engineering and deployment and all that.
- [24:31]
A
I love you and I probably fine tuned Alpaca a million times, but have they?
- [24:37]
B
No. Yeah, yeah.
- [24:38]
A
If they can fine tune, you know, get it down, get the data they need, deal with it in, you know, 90 milliseconds, 100 milliseconds, how is that compared to you're. You're dealing it and just gives them the ability to do the things and to learn and to take that step, to start moving them forward. And I find that to be it not only gratifying for the company and for who we are and what we're doing with them, but it gives them that ability to be able to say, hey, I know I can do this now. I want to try to do something else and I want to do something else and I want to do something else.
- [25:21]
B
Right, right.
- [25:22]
A
Moving them forward in that process. And I find that to be a wonderful challenge for us to be able to do with these individuals that we do that with now.
- [25:32]
B
This is awesome. So how do you move then from, you know, this whole investigative type or experimental into institutionalized or operational? You guys have the luxury of, of working with clients and things like that. They're used to a lot of custom type code and integration with SAP, but do you see some reusability here? And then how do I institutionalize these agents or these small language models with the stuff around them into something that is reusable? Because I don't want to create this new every single time.
- [26:10]
A
You're right. Because the future of AI isn't necessarily about being bigger, in my opinion, it's about being smarter. So what I want to do is I want to stop paying for generality and I really want to start building with precision. So the tools that we, that we think we use from an SLM perspective is going to be more precise and it's going to give customers when, when we work with them on a use case or multiple use cases, try to limit it to a use case. Up front. And we start to go through that process of the dependencies and then what are the targets and how to deploy and what are the savings and what you're going to find from a benefit. It's all about the precision activity, because we'll leverage the one rather than the many. And for us, we find that that works very well because organizations see that, that instantaneous gratification, hey, I just paid you X dollar amount for all this. And I got this. I got exactly what I wanted to. Now let's go even bigger. How can we do this? And how can we deploy other pieces that my HR team depends on, or my finance team, or these parts of my organization? Or from a global perspective, how can I leverage some of these components? That's where I see the precision and the accuracy of those SLMs really benefits. And it's all reusable. I'm not taking their data. We're building the components that allow us to extract that data at their facility, but at another facility that's a completely different.
- [27:45]
B
Yeah.
- [27:46]
A
Organization.
- [27:47]
B
So this is really it, because I still need software engineers. I still need these guys. We've just given them another tool so they can move faster. And it's a different kind of processing that I'm going to do with this small language model than I've been able to do in the past.
- [28:02]
A
Because in my world, I'm not necessarily hiring junior developers anymore. I'm hiring that next level up, those individuals that can think more about what I need to build, how I need to build, rather than just keyboarding something.
- [28:21]
B
So that brings up a nut. That brings up an interesting point, which is, where do I find people like that? Can I find people like that right out of school? I have schools already adjusted to creating workers that can think at that higher level or because I'm teaching it. I'm teaching at Vanderbilt this fall, and I'm having this question myself because I'm teaching a cloud class and I'm. I'm teaching a Introduction to Computer Science class. And I'm like, what am I gonna do? Right?
- [28:56]
A
It's difficult. I. I just came back from Georgia Tech. I was. I was down in Atlanta and Chattanooga this past week, and I stopped on campus and I met with a couple of folks. We started having conversations, and some people seemed like it was a deer in headlights to them, while a couple of them literally gravitated to as well. What if you started doing this and how if you built this and what if you transition this act? And I was like, okay, you three got it. Over here. And the other ones were looking at them going, well, they, we haven't learned anything.
- [29:27]
B
They're just going to use ChatGPT to cheat. I run into that all the time. So you, you want, you want system thinkers, you don't want coders anymore?
- [29:37]
A
Correct. I need thinkers who are tinkerers. And just like I need the Walt Disney folks, I need those people who are going to think about it and then start tinkering with it. You know, it's funny, I was at Georgia Tech this week and I had to give a speech and I made the recommendation, said, I want you to let every student in this school have chatgpt. I want them to have everything at their nose. I can't have them just knowing the Encyclopedia Britannica. I need them to know the globe. I need every piece of information. But then I also need those individuals. This is why you still need a human in the AI world. I need them to review what that has come up with and bring out the best of it. If as you start to see and you start to re. Engage the, the mechanism, start building out those, the components of that more and more. Yeah. And that component, they're doing LLMs and it's all about chat, GPT and Google and blah.
- [30:43]
B
Yeah.
- [30:44]
A
But as the human mind starts tinkering with it, they're thinking about what they're seeing on the screen and now they're going to start playing with. That's where the gold comes in. That's where those little nuggets that I love so much.
- [30:57]
B
Hey, Chris, this has been wonderful. Right. Because you and I are cut from the same cloth. Right. Starting with our Vic 20s and all the way up to AI today. Thank you for coming on the show. This has been very insightful. I know my guests have learned a lot today about small language models and our future. Right. It's not just software engineers that are going to be leveraging these types of things. Things everyone is. And we gotta, it doesn't make us dumber. We actually have to become smarter. So I, I think this is awesome. Thanks again, Chris for coming.
- [31:34]
A
It's been an honor to talk with you. I appreciate it.
- [31:41]
B
Thank you for listening to Embracing Digital Transformation today. If you enjoyed our podcast, give it five stars on your favorite podcasting site or YouTube channel. You can find out more information about embracing Digital transformation@embracingdigital.org Until next time, go out and embrace the digital revolution.