← The Agile Brand with Greg Kihlström® | What CMOs Need to Know About Marketing Technology, AI & CX
The Agile Brand with Greg Kihlström® | What CMOs Need to Know About Marketing Technology, AI & CX
#574: Building enterprise-grade Gen AI applications with Sumeet Agrawal, Informatica
00:29:32
Loading summary
Transcript37 lines
- [00:01]
Audi Q3 Commercial Voice
With the all new Audi Q3, the answer's always yes. Yes to adventure, yes to escape. Yes to right now. The all new Audi Q3 made for the yes Life
- [00:16]
Strayer University Announcer
At Strayer University, we help students like you. Go from Will I to why not? For over 130 years we've been innovating higher education to make it more affordable, accessible and attainable so you can reach your goals. Go from thinking Can I? To Yes I Can and keep striving. Visit Strayer. Edu to learn more. Strayer University is certified to operate in Virginia by Chev and its many campuses, including at 2121 15th Street north in Arlington, Virginia. The agile brand.
- [00:56]
Greg Kilstrom
Welcome to Season six of the Agile Brand where we discuss marketing technology and customer experience, trends and insights and ideas with enterprise and technology platform leaders. We focus on the people, processes, data and platforms that make brands successful, scalable, customer focused and sustainable. This is what makes an agile brand. I'm your host Greg Kilstrom, advising Fortune 1000 brands on martech, marketing operations and CX. Best selling author and speaker. The Agile Brand podcast is brought to you by Tech Systems, an industry leader in full stack technology services, talent services and real world application. For more information go to teksystems.com now let's get on to the show.
- [01:42]
Greg Kilstrom (Interviewer)
It's hard to escape talk of AI these days, but not all AI is the same and not all of it is safe for large organizations to use. Today we're diving into the evolving world
- [01:52]
Greg Kilstrom
of generative AI for the enterprise with
- [01:54]
Greg Kilstrom (Interviewer)
Sumit Agrawal, VP of Product Management at Informatica. We're going to discuss strategies and considerations for building robust enterprise grade generative AI applications.
- [02:05]
Greg Kilstrom
Sumit, welcome to the show.
- [02:07]
Sumit Agrawal
Hey Greg, thank you so much for having me.
- [02:09]
Greg Kilstrom (Interviewer)
Yeah, looking forward to talking about this with you. Definitely a topic on a lot of people's minds. Like I mentioned, you know, we've talked about this from a few different angles on the show already, but it's, you know, just so quickly evolving that it's, you know, I think there's just always, always more to to talk about here. Before we dive in though, why don't we start with you giving a little background on yourself and your role at Informatica?
- [02:34]
Sumit Agrawal
No, absolutely, Greg. And thanks again for having me here. And my name is Sameet Agrawal. I lead the Product management team at Informatica and I'm responsible for mostly of their AI analytics IPAs product lines. I do a lot of work with Gen AI and I have a lot of experience working with enterprises on Genai so very excited to join this podcast and share my point of view.
- [02:59]
Greg Kilstrom (Interviewer)
Great, great. Yeah, definitely you're the right person to be talking about with this topic. So looking forward to it. So why don't we start with just give us an overview of what you're seeing. What's the current state of generative AI and enterprises today?
- [03:15]
Sumit Agrawal
No, absolutely right. So I talked to a lot of enterprises and most of these enterprises are either implementing something with gen AI, piloting something with gen AI or something has already been in production. In fact the number which was just 2% or 3% earlier in 2023, now that number has gone to 55%. When I say 55%. So these are the JDI projects which are either in pilot or in production and that is just the start of last year. And if I fast forward there is a big prediction that by 2026 80% of enterprises have either implemented gen AI or they will be in production or they will be at least piloting with gen. So gen AI is real. It is something top of mind for many executive. In fact it's a board level initiative and recently there was a survey done and there were around 89% of execs have ranked AI engine AI as one of the top priority for their organization. So it's a big thing in many big enterprises. We have also seen enterprises either trying to enhance their existing use case for better productivity with Genai and many of them are also exploring new use cases. Cases. So it's pretty real.
- [04:40]
Greg Kilstrom (Interviewer)
Yeah, yeah. And yeah, 80% adoption, that's, that's pretty prevalent and it certainly tracks with, you know, everything I'm hearing. I know and I'm sure you know what, what's what others are hearing as well. You wrote a blog post recently that talks about the necessity of moving beyond traditional data science tools. I mean you know, let's just say pre chatgpt because I know everyone's familiar with that. You know the this realm was pretty much relegated to data scientists and AI was kind of synonymous with the data scientist and data science tools out there and stuff like that. But you wrote a post recently talking about why we need to not scrap them altogether, but move beyond solely using data science tools when developing generative AI apps. What's the reason for this? What are some limitations and why look at this in this way?
- [05:37]
Sumit Agrawal
That's a good question, Greg. Right. In fact when I last checked I saw that there are more than 17,000 gen AI companies or startups which has come up just since Last December of 2020. 3 so you can imagine the kind of innovation and the kind of vendors which are coming in into this space. It's changing every day as we speak, it's changing. The new best practices are coming up and it's extremely hard for enterprises to keep track of what's going on in the market. And my main reason of saying that you need to look beyond these traditional data science tool is because especially when you are deploying any technology in enterprises, it's not just about deploying the tech, you also need to deploy it in production with the enterprise grade standard. You need the same standard like operationalization, security, governance. You need to be wary of cost, you need to know how easily it can scale, how it can be deployed in a different runtime environment. So all these things are sometimes an afterthought in many of these newer companies or many of these initial, I would say early science tools. And that's where we have seen many of the organizations struggle. They start with something which is shiny and they take this newer tools and very soon they realize that they need all these functionality and that's where it becomes hard for them to come out of that and really put that in production. So in my opinion these are some good tools for prototyping and experimenting. But what you need is not a science tool. You need something which can be deployed in production. So just be wary of that.
- [07:24]
Greg Kilstrom (Interviewer)
Yeah. And building on that, you know you as you mentioned you work with a lot of enterprise orgs and you know where, where do you see the biggest obstacles and you know is it related to kind of the, the tooling and you know, but where, you know where, where are you seeing some of the biggest obs in kind of adopting gen AI?
- [07:45]
Sumit Agrawal
Yeah, I think it's definitely tooling is one of the big aspect but there are other issues, right? Like for example if you are building a gen projects within the enterprises the first thing I see most of the enterprises struggle is the skill set gap. As I was saying as I was sharing you before, right. There are 17,200 startup which just came out how hard it would be for enterprises to really keep up with all the newer things which comes up in Genai, all the new best practices and everything. In fact McKinsey did a survey recently and they found out that there are around 60 to 80% of enterprises find it extremely difficult to hire the right talent for gen AI. And even if they are able to hire them, it is extremely difficult for them to retain these talent because these talents are rare and they don't tend to stick around. So that's I would say is the number one obstacle I see with enterprises is the skill set gap and hiring the right talent. The other big issue I see is the data issue. Because your Genai is as good as the data you are using it for. And here, especially when you're building any Genai projects, you're looking at not just the traditional structural data, you're looking at unstructured data, you're looking at data coming from different latency, real time, batch and especially for CDOs. Having the right data strategy is crucial for Genai value and many of the organizations do struggle to get the right data needed for Genai. Whether it's getting the access of the data or even to prepare your data for Genai is extremely hard. And the third thing I would say is the similar problem we face, right? Like many of the organizations kind of pick up smaller tools or a science tool and they start facing issues with enterprise wide adoption where we see that there is a survey which was done like done recently which says that 80% of your projects will fail, it won't go beyond PoC. And that is because of the lack of enterprise wide standards. Right. You don't have a good standard for model building, training, deployment, monitoring, all that is the key reason why you kind of fail the Genai project. So, so these are the three reasons I generally see what companies are struggling with.
- [10:13]
Greg Kilstrom (Interviewer)
Yeah, and so what do we do about this then? Are there lessons to be learned? I mean we've certainly been through plenty of technology disruptions before even the beginnings of the Internet. And for enterprise, big data cloud, can enterprises learn from some of these things and if so, what are some of the strategies to use to overcome some of these things?
- [10:40]
Sumit Agrawal
Yeah, it's a great question. In fact I would say that for big enterprises, first of all you need to start small. I mean don't have a very large goal where you can't measure that goal. I would say start small, have a very focused goal in your mind with the business outcome in your mind and you should build a team which is cross functional team which can achieve that goal for you easily. And the goal should be such that it is measurable and especially when you are using Gen AI. The other big thing, which I didn't, I missed in the last point is the cost. Right. So be wary of the cost of deploying gen AI solutions. Remember some of these large language models runs on GPUs and your cloud budget will shoot very soon. So I think be wary of that and make sure that you do show the initial success to your stakeholders and then go on to the other use cases. So I mean, that's what we have done when, even whenever the new technology came, whether it's big data, cloud, it's always starting small, showing the progress, showing the success to your business and then moving on to the other big things.
- [12:03]
Greg Kilstrom (Interviewer)
And so, you know, one of the things that I think not only has made generative AI so popular so quickly, but also I think one of the powerful things about it is that it is, it is kind of a democratizing technology. I mean, just about anybody can, you know, go on to ChatGPT or Google, Gemini or, or this or similar platforms and do something with it, right? So, you know, consumers, in other words, can, can even do something with these technologies. But as we know, there's a big difference between what works for consumer or even, you know, small business and what works well in the enterprise. And so, you know, I want to talk with you a little bit more about some of those distinct things that the enterprise really needs and you know, things like scalability, security, privacy, all those things. I'd like to start with scalability, which is certainly an issue. Even if you're doing a proof of concept or a pilot project or something, scalability is always going to be an important aspect. So can you elaborate on what are some of the strategies for ensuring that a Genai application can scale effectively within a large organization?
- [13:20]
Sumit Agrawal
Yeah, no, it's a great question. And that goes back to the point we were making, that whenever you deploy a Genai solution, don't think of it as a science tool, think of this as an enterprise solution, right? And the reason why you have to think about it as an enterprise solution is because scalability is very important, right? And when we talk about scalability, I think the first thing, very, very first thing is the infrastructure and architecture you pick. So you need to be very wary of what kind of infrastructure you pick, right? Which kind of cloud ecosystem you want to pick. And many organizations have relationship with some of the cloud vendors, whether it's aws, Azure, Google. So make sure that you pick the right infrastructure which gives you the right scale as well as the discount you are looking for in terms of your cost. Then the other big important thing is that how are you doing your model optimization? I mean, whether, how are you using your GPUs, etc. So we value of that. Deploying it on containers, making it serverless, all these are the important strategies you have to make. Generally, if everybody start deploying it on their own and they don't do any kind of reusability, then your cost will definitely go for. You will definitely have a cost over an issue. So having scalability deployment with containers, Dockers, having serverless Azure architecture is an important part of it. Cost management, as I said is important. Like having good strategies of cost management which can help overall on your scalability is important. Governance is important, like who is using it, how many people have access to the Genai solution is important and how you govern it is extremely important. And last and not the least, I think reusability is extremely important. I think before you need to build a reusability framework at every level so that anything which is built by your team is being reused across the board so that not everybody start building their own language model or any, any of the modeling aspect. They kind of first see what is available and then start building.
- [15:37]
Greg Kilstrom (Interviewer)
Yeah, I mean along those lines, like in your experience, I mean I think most, even if they're not a CIO or in an engineering team or data team, most people are familiar with the concept of cost for storage and data and stuff like that. Is Genai adding with GPU usage, all that stuff, is it adding another layer of complexity? Are most people that you're talking with in the enterprise aware of the, you know, the cost structure or the cost structure differences? You know, there's a lot of, for instance like chief marketing officers and, and people listening to this show that again, very familiar with cloud, you know, cloud in, in a sense and stuff like that. But is, is it, is it a lot different?
- [16:24]
Greg Kilstrom
Is it a little different?
- [16:25]
Greg Kilstrom (Interviewer)
You know what, what are your thoughts there?
- [16:28]
Sumit Agrawal
Yeah, so it is definitely going to add an entire layer of cost in your overall budget. Right? Because remember first you were doing, when you were doing a digital transformation, you were moving from your data centers to cloud and when you had a good finop strategy on how to use containers or how to use your EC2 instances to get your better cost. Now when you start deploying your gen solutions now you have to think about how will you deploy these language model or large language model. And some of them tend to be extremely expensive. And like say for example, if you go and deploy Azure OpenAI solution in your private interface, it is going to cost you. And then if you try to do like generally just deploying the vanilla LLMs is not going to help you. Sometimes you need to fine tune it or on top of your vanilla models with your enterprise data and that tends to be even more expensive. So, so yeah, having the right strategy on how to deploy These models how when you are doing fine tuning what kind of right infrastructure to use who should have access to because some of if you're using for example public gen AI solutions then how much query you are making is going to cost you based on that. So who should have access to your genai solutions also makes a huge difference on your cost. So yeah all these are the different factors which you have to look in when you are deploying your gen AI solutions, especially in terms of cost.
- [18:12]
Greg Kilstrom (Interviewer)
And so you know there's I think most of us have been through the some of the changes with consumer data privacy, you know starting essentially with GDPR and I know there's. There's quite a few regulations here in the United States as well as elsewhere in the world. It's. It looks like we're basically going to be going through similar things with AI and it appears like Europe is kind of leading the charge again here but elsewhere in the world is going to quickly catch up as well. How can enterprises ensure that the gen AI applications they're currently working with are keeping these things in mind? Again, anyone that's been through the GDPR and CCPA and all those things with consumer data privacy know that these things are often start out a little, let's call it ill defined but get get more more defined over time. So you know, what's your recommendation there and just you know building something that's going to be able to comply with something that is kind of a work in progress right now.
- [19:16]
Sumit Agrawal
This is a extremely important question and an extremely important area for any enterprises to look into that. Having a right governance standard is extremely important. In fact I'll tell you a story where we have released a feature within Informatica which helps customers to build their own. I mean it's something where you can use a chat prompt to actually build Informatica solutions. And we have seen customers asking a lot of questions around how you train your models, what kind of compliance you have used and all that is extremely important. So I think I'll give few tips which I kind of learned from my customers speaking to my customers. I think the first thing is establish a clear ethical guidelines, right like what kind of guidelines you want to align with. You mentioned gdpr, ccpa, all these are the different regulatory compliance and you need to stay updated with these, with these regulatory appliances they keeps on changing. So make sure that you are keeping up to date onto these compliance. You need to also build a robust governance framework to oversee development and deployment of AI applications. Which means when, whichever, how are your Engineering team or your AI team is how they're building these Genai solutions. You need to build a framework to see how they are building it, what kind of data they are using for training. And then you need to adopt bias mitigation strategies. Right. You need to detect and mitigate biases in AI models and just to. You need to build a framework which can actually understand that hey, this data which is getting trained here, it is biased, it doesn't represent everybody here and it doesn't have the. Or your algorithm which you're using is not in, doesn't have the full fairness. So you need to build that mitigation strategies and also the bias framework, identifying the bias framework and then also engage in stakeholder consultations where you need to have a diverse set of stakeholders in your AI development process to gain the wide range of perspectives. You cannot have just stakeholders with for example only males there. You need to have stakeholders which kind of is more diverse in different, in different angle. You need to provide training and education to everybody who is using AI, especially ethical AI. Yeah, you need to do that. And then also finally also have a monitoring and auditing framework which can monitor how well are you doing in this area and especially on the ethical and regulatory areas and then how you can conduct regular audits to identify and address any issues. So these are some of the strategies many of the organizations are taking to be very fair and to have this kind of AI governance in play.
- [22:27]
Greg Kilstrom (Interviewer)
Yeah, yeah. And you know, as you mentioned, there's a lot of organizations that have already started down this path. There's many more that are starting down this path. As an organization is looking at, you know, let's say they have a pilot project or two or even some, some you know, fully functioning apps out there. What's the, you know, there's always a tendency to kind of reach for something brand new versus maybe build on, on the, the existing. You know what, what would your advice be? Obviously knowing that there's, there's a lot of use cases out there, but how should organizations be thinking about kind of creating net new applications and use cases for Gen AI versus kind of building on existing ones, what should guide them?
- [23:19]
Sumit Agrawal
No, it's an important thing, especially when you are getting started with Genai. Is one of the very big dilemma many companies have is shall I enhance what I already have or shall I look for newer use cases? And I think my answer is very simple. Right. I think it all depends upon in how mature you are as an organization in your Genai practice, how well, how good Is your team how aligned your stakeholders are on this. Right. And especially if you are starting new with Gen, I think my approach is that, I mean always start with enhancing your existing use cases because your existing use cases has already been bought by your. I mean your business has blessed it. Now if you can make it more efficient and more automated, there's nothing like it. And it helps you to manage your risk, it helps you to show your easy win to your organizations, control your cost because many of that framework has already built. And then once you have a solid understanding and operational base with Gen AI, then you can start exploring with the newer use cases to drive innovation and capture additional market opportunities. But if you start everything from brand new, it will be difficult first of all for you to get the buying from your business and then to show the value it is adding up. Right. So I always tell my customers that take something which you already have and see how you can enhance it and then and move on from there. Right. And you will definitely see a big success with that strategy.
- [24:57]
Greg Kilstrom (Interviewer)
Yeah, yeah, that makes sense. Just to talk a little bit about since you have a good purview of not only what's current state but as well as what's ahead. And I know you mentioned some statistics about just how fast organizations are adopting these things. What do you see as far as what's ahead in the months and years to come with generative AI? What would your predictions be for the. For the months ahead?
- [25:26]
Sumit Agrawal
Yeah, so I think in my opinion Gen AI will be widely adopted across industries. I mean today many of the industries are either experimenting or piloting with it. But I definitely see that in the next few years it will be widely adopted across industries. Whether you're in healthcare, fincer, retail. I mean you will see that which will be widely adopted and it becomes a commodity. I mean it's just like today if you go to Internet and search, I mean it is not like a big deal, right? I mean in the few years, I mean many decades back it was such a big deal of using Internet. So in my opinion it will be widely adopted, it will become a commodity. There will be a lot of advancement in creativity and content creation with Gen AI. I mean I've seen, I mean in my opinion many of the, I mean a lot of things will change with GENAI coming into picture on the content creation and creativity side. And I kind of think of Genai to be something like a human assisted tool. Like I would think human and AI collaboration will change with Genai and there'll be more and more Genai will become like a co pilot for human for collaboration. And then as we discussed. Right. With all these things there are two things which also will change is how you do ethical and regulatory frameworks. There will be more and more of such things will come up. Every countries will have something on their own and if you are adopting to it, you need to comply to some of these regulatory and governance policy. And since Genai is always cost sensitive so there will be definitely a cost angle. I see that more than large language model, language model will also come into picture. I mean it's already there. So whether it's small language model or just the language model, those will also be more prominent in the next few years. So overall I believe that there'll be a big adoption of Genai and it will become more and more mainstream in the next few years.
- [27:35]
Greg Kilstrom (Interviewer)
Great. Yeah, we'll have to have you back on the show to follow up on a few of those things as well. But yeah, no exciting stuff. Well, thanks so much for joining today. One last question before we wrap up here. You touched on a lot of great stuff already. So even if it's a point that you've already made so far, what is one step that you'd recommend that an enterprise should take to just focus on the right things with with Gen AI?
- [28:02]
Sumit Agrawal
Yeah. So I think the one thing if one step I would all I'll recommend all of you is develop a clear and strategic roadmap for your Gen AI initiative. It's very easy to get distracted with so many things going on, but in my opinion, start small, take a very business outcome focused use case and see how it can add value to your existing businesses and establish a framework which can measure the success of your use case and also think about investing in ethical governance. I think these are all the different things you can do to get started with Genai and once you see the success then scale beyond your existing use cases. So that would be the one thing I would recommend all the enterprises listening to this podcast to take back on it. And Greg, again, thank you so much for having me on the show and it was pleasure talking to you.
- [29:09]
Greg Kilstrom (Interviewer)
Yeah, it was great, great talking with you again. I'd like to thank Sumit Agrawal from Informatica for joining us today and sharing his expertise on building enterprise grade generative AI applications. You can learn more about Sumit and
- [29:21]
Greg Kilstrom
Informatica by following the links in the show notes. Thanks again for listening to the Agile brand brought to you by Tech Systems. If you enjoyed the show, please take a minute to subscribe and leave us a rating so that others can find the show more easily. You can access more episodes of the show at www.greggilstrom.com. that's G-R-E G K-I H L S T R O M.com While you're there, check out my series of best selling Agile brand guides covering a wide variety of marketing technology topics. Or you can search for Greg Kilstrom on Amazon. The Agile brand is produced by Missing Link, a Latina owned, strategy driven, creatively fueled production co op. From ideation to creation, they craft human connections through intelligent, engaging and informative content. Until next time, stay Agile.
- [30:15]
Strayer University Announcer
The agile brand.
- [30:23]
Audi Q3 Commercial Voice
With the all new Audi Q3, the answer is always yes. Yes to adventure. Yes to escape. Yes to right now. The all new Audi Q3 made for
- [30:35]
Progressive Insurance Announcer
the s life ever notice how life's best stories don't happen in your living room? They happen on the open road, out on the water, or parked under the stars. At Progressive, they get that you want to focus on the experience, not worry about the what ifs. That's why they offer quality insurance designed for your ride, whether That's a boat, RV or motorcycle. Adventure with confidence. Visit progressive.com and see how easy it is to protect your favorite way to getaway. Progressive Casualty Insurance Company and affiliates not available in dc. Prices vary based on how you buy.