← AI to ROI
AI to ROI
The Use and ROI of AI in Finance - with Sowmya Ranganathan, Former Controller OpenAI and CEO, Lumera
00:31:15
Loading summary
Transcript49 lines
- [00:00]
A
Hello, I'm Ray Wright, Founder and CEO of Benchmarket and your host of the Metrics at Majorette Podcast. We talk to a wide variety of the top B2B SaaS and Cloud Thought Leaders, CEOs, executives, investors and people just like you to discuss the metrics and benchmarks they use to make metrics informed and benchmark validated decisions. Now onto today's show. Welcome to today's episode of the Metrics Measureup podcast. Today I am joined by Samya Rankanathan, the founder of Lamara and a former controller at OpenAI. We'll be covering three primary topics for Samya today. First, how OpenAI actually used generative AI in their finance organization. Two, some lessons learned and best practices of using AI in finance and third, how to measure the return on investment of AI in finance. So with that, Samya, would you please take a moment to give a brief overview of your journey to becoming a guest here on the metrics that MeasureUp podcast.
- [01:15]
B
Thanks for having me today Ray. Really happy to be here. My background so I'm a cpa, started my career in Canada, moved to San Francisco on a second, thinking I'd be here for, you know, six months or so and would move back to Canada. That was 12 years ago and we are very much still here in San Francisco. This is home now. I, I left EY to go join Square as an accountant in their controllership team. I didn't actually know how to pick companies. I was a customer of Square and I was obsessed with the product and I had this mindset, I don't care what job you give me, just give me any job and I just want to be at this company. I got very lucky because I happened to join the company a year before they went public and I learned a lot about what building accounting and tech at scale looked like. After Square did a couple of stints at startups. One was a restaurant for robotics for restaurant startup. The second one was Rippling, which is of course an enterprise software company serving hr, payroll and a whole suite of products for different parts of the business. And post Rippling I was on a sabbatical. Lucky for me, my last day at Rippling was the day ChatGPT launched and I was enamored my whole time that I wasn't working. I was, I just couldn't stop playing around with the product and I got a call to interview for the controller role at OpenAI and it was a no brainer. I was, I was just, you know, head over heels over what this new technology meant for accountants for finance teams, for, for the world overall. And so I was very lucky to join OpenAI as controller the month that they launched the paid version of ChatGPT. So I did that for two years and this past March I left to found my own company, bringing AI deployment to finance teams everywhere. So that's the journey.
- [03:14]
A
Well, wow, 12 years out of Canada Square, rippling OpenAI. I must admit, you have had quite the run here in the US of A.
- [03:25]
B
It's. I mean, look, this is, this place has treated me really well. But I'd say the real amazing part of being here and the reason I chose to build my company here is just the professional network you build here is barded. You know, it's, it's really just the best of the best.
- [03:43]
A
Yeah, I was so fortunate. I moved to San Francisco in the mid-90s and my first job up here was the company called Netscape, which was kind of the launch of the Internet era. So same thing, the network I built, but more importantly, the experiences and lessons I learned. Invaluable. But let's move to OpenAI. Okay. They launched, I believe, what, November 30, 2022 is when ChatGPT launched.
- [04:05]
B
Yep.
- [04:07]
A
So you were there during two years of just amazing growth. So the first thing I have to ask you is just, can you explain a little bit about the environment for those two plus years where growth was just off the charts?
- [04:21]
B
It was very humbling. You know, many people, me included, thought we had seen just hyper scalers, hyper growth coming into the company. And it was the kind of job where there's no playbook, you could rinse and repeat. Nothing you thought worked in a different place would work here because this was just operating at a whole different level. It's almost like, you know, maybe you thought you were a star player and someone changed the rules of the game right as you were in the game. That's, that's realistically how it felt. Now we were very lucky. We had an all star team. It was a very small team. The all of finance was around 10 people, I think, when I joined. And you know, we were, I joined in March 2023. So a few months after ChatGPT launched, but the month we launched the paid products. So from an accounting and finance perspective, there's a lot of rebuilding to do. But we had folks around us on the team who, like I said, you couldn't really bring your playbook from anywhere that you'd worked before. But we had the ability to think from first principles and think about, okay, how should this really work? What is the problem? What does the solution look like? And it was this culture of being super empowered to just go do the thing that needs to be done. In many ways it was, you know, it was crazy, but it was also the most energizing type of work that you can imagine for a finance and accounting org.
- [05:52]
A
You know, we have this old kind of analogy. It's a little bit like changing the wheels as the bus is going around the road, but quite frankly, you were changing the wings on a plane or you know, a jet that was going quickly. So let me ask you this question. Accounting, you need to make sure that, that all those debits and credits are right, that the reports are accurate. So what were one or two of the big challenges that you faced in finance over those two years? Was there a particular aspect, process or reporting that most challenged you?
- [06:24]
B
I'd say anything that got impacted directly by the volume of activity in the business. So most directly we're talking revenue accounting and the accounting for infrastructure costs. Because, you know, if you think about revenue, every $20 transact is a row in some revenue transaction table that we now have to grapple with. And then the compute side of it is the details of what GPUs were used, how much were used, what's the cost of this relating to servicing these users, both free and paid. The first one, we of course need to know how much money we're making. And for compute, it's really important to get a good grip on where you're spending your money on Compute1 to know how much you're spending overall. But also, is it for R and D? Is it to serve our free users? Is it to serve our paid users? Because when you start thinking about what is the margin profile of this business, if your compute numbers are wrong, your margin is wrong, and where you start your analysis for anything you want to do in terms of decision making in the business, you're starting from something that doesn't make sense, I think in practical terms. What does that mean? It means that you couldn't do these things in Excel anymore, right? Maybe we were doing revenue accounting in Excel or compute accounting. There wasn't a maybe we were doing compute accounting in Excel and suddenly you can't open these things in Excel anymore because you've exceeded the row limit. You have a CSV file from a vendor for compute data. You open it and it says, Excel is unable to open this file because it exceeds a million rows. And you're staring at like, okay, now what do I do?
- [08:06]
A
Well, And I would. You have both the expense side and as you launch a new model, it's probably even hard to forecast and predict actually what the cost of goods sold and operating expenses are going to be because it's so dynamic. So did you even think about worrying about trying to forecast and predict the expense side of the equation?
- [08:25]
B
We had some very smart people on our strategic finance team doing this kind of forecasting. And, you know, I'd say, given how difficult a business it was to forecast, our forecasts were actually really good. I think a big reason how we got them to be good was actually to kind of not do your forecasting sitting in front of a spreadsheet, but really go talk to the business, talk to the researchers building these models, talk to the engineers deploying them, and really understand the nuts and bolts of what does it take to serve this model and have a really like deep understanding of what is the business trying to do when you try to model it. Because I think if you thought of looking at historical data and tried to pull it forward, like that kind of forecasting mindset just would not have worked at this company hands down.
- [09:18]
A
And on the revenue side, because you were also there when you opened up your APIs and you introduced token pricing. Right.
- [09:26]
B
They already actually had the API business going. So we had a little bit more practice on that side before ChatGPT subscriptions.
- [09:34]
A
But because there were so many new use cases with your enterprise customers, I think it had been very hard to actually do a good job of predicting demand or same thing, you had a lot of really smart people or you had some good AI tools to help predict demand. Which was it?
- [09:50]
B
I think at that time, to be honest, it was definitely very human driven. And even if we had the AI capabilities, I do remember we did an internal team hackathon for AI applications. One of the demos I saw was someone on our strategic finance team used AI to come up with a predictive model. And it basically took in all these drivers for top line growth. And it built a statistical model using R code. And it said, okay, here's the model and here's the things you can tweak. And it was actually cool how quickly they came up with that in a hackathon. But in the early days, I want to say it was definitely more human ingenuity than the AI model.
- [10:37]
A
Well, I love the idea of actually you were doing hackathons in the Office of Finance back then.
- [10:42]
B
Yeah.
- [10:43]
A
And I still remember Maggie Haught actually spoke at SAS Metrics Palooza about a year and a half ago. And she mentioned that OpenAI was leveraging your own AI product for the financial close. Can you tell me a little bit about how you applied AI to the closing process?
- [11:02]
B
Yeah, I'd say, you know, the compute cost was kind of my first aha moment. And, you know, before that, we were all really excited about what chargpt could do, right? So we were using it to write policy docs. Our company was suddenly growing from being this 50, 100% company to much bigger than that, and we needed to rewrite a lot of internal policy documents. So ChatGPT was already being a good writing assistant in many ways for those types of docs, technical memos. It could write a first draft for you really quickly and you could clean it up. But I think the really big aha moment for me was in a moment of desperation, I went to ChatGPT and said, Look, I have this, like, four gigabytes of CSV data. Excel won't let me open it. Tell me how I can do my work with this file. And I, I didn't go to ChatGPT because I wanted to be this like, you know, showcase of OpenAI using OpenAI or anything. It was a moment of engineers are super busy keeping the, you know, site alive and the product working for our customers. There's no extra resources. I can't wait to go hire some talent. Like, we need to close the books right now. So it was in that moment that I started talking to ChatGPT and said, give me ideas. And ChatGPT said, you know, Python is really efficient at processing CSV files. And I was like, well, okay, this is an accounting team, we don't know how to use Python. And it said, well, you just tell me what you need done and I can write the Python code for you. And it felt so silly that it didn't strike us to go ask for it to begin with, because of course, our engineers and everyone else was talking about how good it was at writing code. But then that's what kind of helped me connect the dots. And it started with those humble beginnings. And then by the time I left, we had this script running in the cloud, in our data warehouse, without needing manual intervention. So we took this initial ChatGPT generated code and over time really expanded it to really be embedded in the rest of our process. But that is an example of how we used ChatGPT and AI in general to really supercharge something that we were struggling with to do manually for close.
- [13:12]
A
So that's a great example. So you took it as a Personal help me with my productivity or accomplishing something. But then did you actually convert that into a programmatic approach where you had the Python code, but then you actually got it documented, defined and enhanced so multiple people could use it?
- [13:31]
B
Yeah. So at OpenAI, the strategy for this particular problem was we had a really large data set coming right for usage data for these GPUs, and we figured out how to send it from the vendor to databricks. And we realized if we could run our script in databricks, the script basically was a filtering and tag of the costs. We could then make that a real time dashboard, not just for our close purposes, but for the rest of the business who needs to know how the costs are tracking in real time during the month. So our productionizing or programmatic approach for turning this ad hoc script into something that is really useful is to put this script in databricks and run it in the cloud in real time. So as new data comes, it automatically sorts that data and categorizes it and puts it in a dashboard that's really useful for more teams than just finance.
- [14:21]
A
That's interesting. So it was running natively within databricks, and I hate to ask this question, but a lot of finance people, as we do benchmarking research, are still concerned about the quality of the output. And I hate to use the word hallucinations, right, but what did you do to kind of put some QA around this programmatic process to have more confidence in the outputs that are being generated?
- [14:45]
B
I'd say hallucination is a very valid concern, actually, and especially with my audit and accounting background, I am as conservative of a finance user of these products as the next person. And I'd say the nice thing about doing code and the AI part of it, or the hallucination risk really is limited to the part where it's writing the code for you. And once you have this code, it's a static Python script. There's no additional AI happening there. I've given it very specific business logic, and it's just applying this logic at scale to a large data set. The nice thing is, because it's a static code, you can test it for all your edge cases upfront with historical data. You can do backtesting, you can reconcile, you can run all the different use cases, test cases you want to run. Once you're comfortable, you can put it in prod, and you know that the script is going to do the exact same thing every time. Now, if we were putting, say we were doing an implementation of AI in a more stochastic way where we fed in data and we were asking AI to do something on the fly using these LLM API capabilities, we would definitely put a layer of human review as one check and balance. The other kind of check and balances, we can reconcile it to other data sources that are not stochastic and then surface any differences. I'll give you an example of what that means. We were using our internal API to extract contract data fields from signed contract PDFs. Right. And not all these PDFs were super. You know, it's, it's like readable enough, but you still needed some kind of LLM API or OCR tool to read it. Initially we were typing all this by hand into a spreadsheet and that felt like, okay, this is ridiculous. We don't have the number of people we need to do this, so we need to automate this. But there was a real risk that the AI, you know, if it saw some fuzzy text, what if it just hallucinated the text instead of really knowing what. What was There we did was we then reconciled it against Salesforce data CPQ data. So then those tools don't have any LLM components. So if it reconciles, you can save a decent confidence that, okay, this is fine. When it doesn't reconcile now, you don't know if it's because the LLM messed up or it's because there's an actual data inconsistency problem. So we essentially set up a control where any differences are manual reviewed.
- [17:12]
A
So you had an exception log to get manual reviewed and did that almost become human reinforcement to continue to enhance and train the model because, you know, where there might be some common. Or was it really case by case?
- [17:23]
B
In this case, we didn't have the technical. I mean, we could have gone back to the model team and said, hey, we want to fine tune. We were using just straight, you know, out of the box models. We weren't trying to do any reinforcement in this case. And our exception log was small enough where it wasn't so bad that we felt we needed to. The good news is most times it was inconsistent. It wasn't because of hallucination. We had an actual problem where somebody redlined in Ironclad after something got approved outside Ironclad. So those were much more people and process issues and not so much LLM model issues.
- [18:03]
A
Gotcha. Okay, Samya, we have about 50% of our audience are heads of finance out there. So let's get out of the rearview mirror and let's look through that windshield to the future. You now are running a company that helps finance organizations deploy AI in the Office of Finance. So first of all just tell us a little bit about what you do. Then my next question is going to be about how to get started with AI in the Office of Finance.
- [18:28]
B
So I started this company called Lumera. My initial thinking was let me start teaching finance teams everywhere how to AI AI. And in this case I'd say the most accessible version of AI for finance teams is ChatGPT. So when you log into ChatGPT, you don't necessarily think of it as a finance and accounting tool. You think it's a chatbot and I can ask it questions. So my goal was let's do these workshops where we'll show people five to 10 use cases in real time and it starts getting people thinking about what else do I do at my job that is a good use case for something like this. So the first part of it started like that. In my workshops would teach people how to Write code using ChatGPT similar to what I did in my own job. Right. And I truly think this is one of like the big unlocks for accounting and finance teams to be able to have an engineer in their pocket, so to speak, to build the things they want built. Very quickly my customers for workshops came back and said can you write these scripts? Because we don't have people to do it internally. We love what it can do, we just don't have the bandwidth or the people in house. And then where do we host it and run it? So that's what our product does. We help generate these scripts using AI, of course and we give folks a place to run these automations to connect to the tools they're already using so they can do end to end implementations of this AI code. Gen Automations.
- [19:52]
A
So for each client you will create tailored scripts to them and then run it for them in your environment.
- [20:00]
B
Yes. And this would have been a crazy business idea pre AI because it costs a lot of time and energy to build these customers system scripts because you essentially have to handwrite them. But in an AI world economics is wildly different because you describe the problem and your code comes 90, 95% done and the last 5% is usually tweaking or unblocking whatever error it's generating. It's amazing how it'll get to the 95% mark. And if you try to just keep prompting your way out of it, you're going to just give up because it's so hard to get it to get it 100% right. But an experienced engineer can find the error, make a one line change and it works beautifully. It's been quite interesting to learn also about where the models are working really well and where it's not yet prime time.
- [20:53]
A
What's interesting, I'm going to give a new definition for fte. Not forward deployed engineer, it's the finance deployed engineer.
- [20:59]
B
There you go.
- [21:02]
A
Let me ask you this because so many people, my wife had a finance for a $12 billion organization in the healthcare industry and there's so much concern about data privacy, Right? So how do you help a CFO get comfortable regarding the privacy of their Data using a ChatGPT for some of this work that you're recommending?
- [21:25]
B
I mean, look, I am not privacy console. I don't, I'm not the expert in this space. I do think it's really important to get the company on some enterprise agreement because beyond just the IT and admin controls and permissions, you also have the safety and security of these legal contracts in terms of data privacy and the regulatory framework you need to operate in. ChatGPT Enterprise, for example, is HIPAA certified and you want to make sure that if you have access to these things where you are required to be HIPAA certified, then you want to have the same expectations of your vendors. Once you're in the enterprise environment, your company then has to do this exercise of what tiers of data do we have and what are we comfortable putting in this tool versus not one thing I find is for the more sensitive data, you can write your scripts and whatnot. With ChatGPT and these AI tools with entirely dummy data, all they need is column names really. And you don't have to give it any real data. You get the working code and then you can run it securely in an environment you're comfortable with. For more scaled companies, they tend to have their own version of databricks to run these scripts. For smaller companies, that's usually the area we're able to serve. At Lumera, they don't have a data team, they don't have these tools. And so Lumera is that secure cloud environment where they can execute on these scripts without worrying about, we're not a model company, we're not taking this data to train on or do anything else. And so it gives them that peace of mind.
- [23:02]
A
That was a very diplomatic, balanced and helpful recommendation. So let's talk about that kind of this smaller company, right, because we have a lot of those listeners here. Is there one or two areas that you recommend to that head of finance where she or he can maybe start their AI experience at the Office of Finance beyond personal productivity of one of their analysts, you know, saying, create a summary of this quarter's financial reports, et cetera.
- [23:32]
B
You know, this is a really interesting question because I think a lot of people go to try this tool or try that tool. I think the challenge for adopting AI is AI is so general purpose, it can do a lot of things really well. And I actually think a lot of finance teams I talk to are coming out of this place of being really distracted because they see a demo, it's really shiny and flashy, and they try to adopt it. And then the ROI is a little questionable because maybe it wasn't much of a problem to begin with. And now they've gone through this adoption and yeah, it's still flashy, but, you know, it didn't really change. The late nights people are pulling in their team. So my recommendation is always to start with, look, do an assessment of where the problems are today. What are your process bottlenecks? Where do you feel like you need to keep hiring more people because, you know, the volume of work is growing really fast and you don't have enough hands on the team to help. And I think once you get a good understanding of where the top painful bottlenecks are, you then start thinking about what does the solution look like. And sometimes it's AI, sometimes it's just a big data set and you need something to automate it. And maybe it's, you know, you can use AI for the code gen, but you don't really need an LLM categorizing or doing anything. And I think think it's not much different from how we would implement finance software pre AI, right? You'd look at the problem you have and then you try to find a solution that works for it. I think in the AI era, it's a little distracting and frankly a little harder than before because there's so many tools in the market catering to the space that it makes it really hard to actually shop for the thing you need. So I tell heads of finance as buyers, look for your own what is the thing you need solved, and then try to find the vendor that does it and don't do the opposite.
- [25:23]
A
I love it. It's not AI technology looking for a problem, it's identifying the challenge and opportunity and then decide which technology might be the most applicable. And it might not be AI, it might be some type of data, intelligence, et cetera. So Samya One last question, because we're coming to the end already and I could speak to you for hours. So we had that CFO out there and you just gave him her one recommendation, which is identify the problem first. But to get their journey using AI started, should they partner with a third party? Should they try to hire someone internally? Like, where do they get started? Do they need an AI expert?
- [26:02]
B
Initially, I say this as an AI expert, quote unquote, that people bring into companies. I actually think you don't need one when you're getting started, right? And you can get started by getting Your Team A ChatGPT Teams or Enterprise subscription. If you have Gemini, if you have Clock, I wouldn't sweat too much what exact tool you have. The chatbot is still, I'd say the best place to start because the accessibility, it's honestly very cheap for how much you can get out of it. It's very easy to get started. There's not this long implementation time. You can put it in the hands of your team. And I think the most important thing is to cultivate this mindset of prototyping and experimenting within the team. And these tools are amazing for that. Just like I went and asked it, I have this, this CSV file problem. Tell me what to do. You can go talk to it about the thing you need to do now where you might need a little bit more, you know, maybe like a third party vendor or a consultant or tool or someone that's seen this type of thing before, is when you're moving from that prototype to production. It might work great as a prototype, but you'll realize, well, this doesn't really work for an audit or this doesn't work with our socks requirements. And that's when you need somebody to help strategize. And if you have the talent and has great benefits, but you can also kind of lean on external help for those parts. But in the early stages, when you're still trying to identify what does a great solution here even look like? I think it's really hard to beat one of these chatbots as a better starting point.
- [27:39]
A
Got you. Okay, I'm going to ask you three rapid fire questions to let the audience get to know you a little bit better. So first, is there a CEO or company that you think is a must follow here?
- [27:49]
B
And late 2025, honestly, I would say follow the AI model companies, if you haven't already. It might sound like a very basic recommendation, but it's one of the things where the technology is changing so fast, where if you don't follow it and you kind of come back to the space six months later, you might feel very, very behind. I personally, I'm not super like influencer driven person. Like I learned much more by doing and by thinking about things from first principles. But I do think keeping up to date with what's happening and like the overall model provider and you know, frontier technology space is probably where I would point people to.
- [28:33]
A
Great, great idea. Okay, you've started your own company. What's a tool that you think every company should be using out there today, not your own?
- [28:41]
B
I'm a daily user of ChatGPT. I had to upgrade to the max tier of subscription because I kept hitting usage limit. So if that's not a glowing recommendation, I don't know what is.
- [28:53]
A
Well, you do have some, maybe a little bit of bias there with that perspective, but I think about 74% of finance organizations that do use an LLM are using ChatGPT, so it's by far number one. Okay, last question. How can people learn from you? Follow you hear about more great use cases of how Companies are adopting AI in their finance organization.
- [29:17]
B
My website is lumerahq.com I started a substack earlier to share some of this content so you can find some of my posts around there. It has practical examples that you can try on ChatGPT. You can also find me on LinkedIn. Granted I'm not as good with sharing content regularly as I want to be, but those are the places you can find me.
- [29:40]
A
Do you have a podcast? Soumya?
- [29:41]
B
I started a podcast, yeah. The goal was to interview CFOs and controllers and talk to them about their tech stack and AI adoption journey. So that's starting podcast today.
- [29:54]
A
Okay, well best of luck with that. It's one of my favorite things is getting to talk to people like Samir Ranganathan about their experience using AI in the Office of Finance. So thank you so much for being my guest here on the metrics that Measureup podcast.
- [30:08]
B
Thanks for having me Ray.
- [30:10]
A
And to our listening audience, if you're enjoying the conversations and finding real value from guests like Samya here on the Metrics Measure up podcast, we'd love for you to go ahead and subscribe to the Metrics of Asia podcast. Go ahead and give us that five star rating and reach out to me personally Ayrike on Twitter or LinkedIn and let me know who you'd like to hear next. Bye bye everyone. Thank you for listening to this episode. The Metrics at Measureup Podcast is brought to you by BenchmarkIT, which enables SaaS, companies and executives just like you with benchmarking research, events, media and the largest benchmarking index in the industry to make better metrics informed and benchmark validated decisions leading to more efficient revenue growth and increased enterprise value. To learn more, visit BenchMarket. That's BenchMarket with an IT AI. BenchmarkIT AI.