Loading summary
Transcript105 lines
- [00:00]
Dana Decker
I don't think all software is going to get ripped out. It can't. But I also think that things are moving so quickly that we are a lot more careful about diving into anything and we are thinking about what we can build.
- [00:11]
CJ
If you have 90 days to go and work on something, what is something that they can do?
- [00:15]
Micah Richard
Just because the word AI is in there, people are like, what do I do? The answer is just start. Your company almost certainly has a set of tools that are in place. The problem is that the longer you wait, the harder it's going to be for you to adapt.
- [00:27]
AJ Lubich
There's easy things like Holiday, where usage kind of can dip in a temporary period of time. The harder thing is that we also have exposure to the seasonality of our customers. Like if you have a video game manufacturer and they have a big release, there might be a surge in usage there. Or there's like streaming companies that have exposure to a certain event. So that's something that's a little bit more nuanced that, you know, so far, AI isn't picking up perfectly, but maybe
- [00:49]
CJ
over time is this thing on?
- [00:54]
Dana Decker
Yesterday's price is not today's price.
- [01:06]
Host / Narrator
Welcome back to Round the Numbers. I just did a panel. That's right, your boy sits on stage, asks other people the tough questions. That is my job. A modern day Walter Cronkite. Someone said Rillet was kind enough shout out to Nick Kauf and Stephen Hedlund to invite me to their annual user conference Recon. It rings a bell. So I was on stage with Dana Decker, VP of finance at Opendoor, AJ Lubich, SVP of FPA at Datadog, and Micah Richard, principal of AI and machine learning at PwC. I started my career at PwC. I had about 1,000 copies. It was called Flavia. It came in like this plastic bag and you would make it late at night and it tasted horrible. But the audits were done on time. So I had three practitioners and experts around how AI is being used within the modern CFO's office. I asked them all sorts of questions about how they're tactically applying AI into their decision making and how they're actually figuring out what the ROI is. We get into a discussion on token economics, how to figure out how many tokens is too many tokens and what you're getting from it. And we also talk about pushing AI not only through your team, but your management team and getting them to adopt. So shout out to Rillet for having me on stage. At Rillet Recon, I had a blast and shout out to the panelists. They're the real stars of the show. Let's get into it.
- [02:28]
CJ
I think we've assembled the highest hourly rate in New York City for the day. So thank you all for coming. I'm CJ. I'm a recovering tech CFO. I came up in the FP and a space running FP and A groups at companies from 10 million to 200 million in ARR. And I helped sell a private software company that was much smaller than the folks on this stage. So I'm in awe of the momentum of all of your companies. And funny enough, I actually started my career at PwC back in the day. We're going to get into the tactics of how some of these amazing companies are applying AI day to day. But first we're going to do some intros. So, aj, tell us a little bit about yourself.
- [03:07]
AJ Lubich
I head the FP and a team at Datadog. I've been at the company for about three years in this capacity, but I actually boomerang back. So my roots are on Wall Street. I was at Datadog originally back in 2018, helped bring the company public and then sat in the IR seat for a while before going to another company, UiPath, leading up their FP&A team before coming back to Datadog. In the last three years you helped
- [03:29]
CJ
take two companies public.
- [03:31]
Host / Narrator
Yeah.
- [03:31]
CJ
Maika, what do you work on day to Day? Because it sounds like you have the coolest job ever.
- [03:35]
Micah Richard
So I'm a partner at PwC specifically focused on AI. My background is a little non traditional for a PwC partner. Started my career in industry leading data science teams. I did spend five years with the firm and kind of in the middle there. So I got a crash course in accounting and finance. I left to go to Facebook where I led ads, ranking, machine learning, engineering teams and then I went to a different tech company as an engineering director. So. So most of my conversations with clients really focus on the practical implications of using AI. As you'd expect, a lot of those discussions are with accounting and finance teams.
- [04:06]
Dana Decker
Right now I'm Dana Decker. I work at Opendoor. I've been there for a couple years and before that a late stage company. I worked at Stitch Fix and helped take them public. And prior to that I started my roots in public accounting at Ernst and Young.
- [04:17]
CJ
What's your revenue size? How many employees are your company today?
- [04:20]
AJ Lubich
$4 billion run rate, around 8,000 employees globally.
- [04:24]
CJ
Dana, what about you?
- [04:24]
Dana Decker
Similar size, thousand employees, billion dollar run rate. We are also have earnings next week so we Will keep that all nice and tidy.
- [04:31]
CJ
All right, I'm going to start broad here. Your, your job day to day. We're kind of on this quarterly treadmill. What was it like a year ago? And how would you compare that to what it is this year as you prepare for the next earnings cycle?
- [04:41]
AJ Lubich
When I joined about three years ago, the team was pretty lean. We were like seven or eight people, which for an FP and a team for a public company is pretty lean. So I've been simultaneously kind of building out the team. We've also been on this journey of going from, you know, manual work and Excel spreadsheets.
- [04:56]
Dana Decker
2.
- [04:57]
AJ Lubich
You know, we've both been hiring some folks that are more focused on data, like an actual sort of data analyst backgrounds, which I do think is kind of the wave of where we're. Where we've been heading in finance already is even pre AI and we've been working on some systems implementation. So we've moved within our FPA practice to Pigment as our new planning and forecasting tool, which has already been pretty impactful. So sort of pre the big wave of AI, we were already kind of getting closer to the data, automating a lot of things. And then what I think AI has done in the last year or so was really just sort of compounded those move movements. And then today what I'd say is we've really turned the dial from going from a reporting function which is based off of Excel spreadsheets and a lot of manual work, to really a strategic driver of the organization. Because finance really sits at the epicenter of everything that happens across the business. And we have a bird's eye view into every single department, plus we have access to all of the data. So those things combined, you know, getting some more data analytics background, having the system that enables us to be powerful with that data. And then what I think AI has really done is democratized it. Every single person, whether you understand data analysis or not, has sort of the power at their fingertips to be dangerous with that data. Now today I think we're much less focused on just sort of closing the books and the reporting function. But actually what do we do with it? What is the context of the data? Who should we be getting this in front of? How does it influence the way that we operate as a business? So we've really been able to accelerate that sort of strategic decision making dial.
- [06:23]
CJ
So two things I heard there, more of a focus on data, data integrity, data cleanliness. And also it sounds like the hiring profile of some of the People you brought.
- [06:31]
AJ Lubich
The board has changed because typically we hired folks like myself, which are, you know, finance by trade and bank. Grew up banking background, grew up in spreadsheets and going more to folks that actually, you know, want to sit in the office of the cfo, Right. But actually are powerful with data, has really been a compounder and a multiplier of everything we've done.
- [06:50]
CJ
Dana, reflect on last year versus this year.
- [06:52]
Dana Decker
One of the biggest things that I think has changed is actually probably the speed of the business. And I think with AI and with our ability to code and to deploy and to build, we have actually fundamentally changed the pace in the business. On top of that, we have an entirely new leadership team. We are a default to AI company. So while we are a public company, we are a company that deploys AI from top to bottom. I said this earlier to some people, but I actually think that my boss, who's the president Vibe codes more than I do. And so it starts at the top. And the amount of AI that we have sort of running through the business is astronomical. But where that comes is things like the speed and the agility and the pace that you actually move. As a company.
- [07:33]
CJ
I want to stay on that leadership actually having fingers in the keyboard. I would joke that sometimes you go to a company and you have what I call an iPad leader, someone who literally doesn't open their laptop. They send emails from their phone or like, they. They're like, let me forward this email from my iPad. Can you speak to the value of leadership? Being in the weeds and trying something.
- [07:53]
Host / Narrator
I also imagine you can probably see
- [07:54]
CJ
how much they're spending on tokens.
- [07:56]
Dana Decker
We are at a place where we encourage token usage. It's the only way to learn. We have, you know, our monthly company meetings where we actually have both leaders and members throughout the company sharing what they have Vibe coded and the new applications that they've made. We have our staff meetings where they're showing this. We have a channel where everybody from tops down is showing what they have built. I think all of our AI tools are super helpful, but so much of it is learning by osmosis. When you see your leaders in there building a dashboard, building an application, it encourages everybody else at the company to go do it too.
- [08:30]
CJ
Just to stand you for a second, can you point out maybe a couple practical examples of where you've incorporated AI
- [08:36]
Dana Decker
day to day, you forecast on a quarterly basis. When the month end closes, you do a flash at the month end, you look at metrics, you know, weekly. And one of the things that we transitioned to kind of right. Actually when Cowork came out is because Cowork was able to pull from a whole bunch of different places. A very tactical example is we really wanted a daily view of, of our P and L. We are a large company and so those components lived in a lot of different places daily. A daily view of our earnings. A lot of the data was in Snowflake, a lot of it was in random Google sheets. I sat down one afternoon and I was like, I'm, I'm just going to build it. And so it was me plus Clyde plus a couple hours in the afternoon and was able to actually pull all the data from across the company and across several different tools, put it into the form that I wanted it to be in with different graphs and different visualizations. And it took a couple of iterations and then from there I had to figure out best way to deploy it. But because of that, our executive team now has a view of our P and L every single day. And so you are really able to see where the company is moving and how it is moving. And that, as I said, ultimately changes the pace of your company.
- [09:39]
CJ
Hey, thanks for listening. We'll be right back after a word from our sponsors.
- [09:44]
Host / Narrator
The CFO role has evolved faster than the tools built to support it. Most finance teams are still running infrastructure designed for a job that no longer exists. The reporting, the reconciling, the close that bleeds into the next month or quarter. That's not finance. That's overhead with a really crappy title. And the cost isn't just your time. It's everything your best people aren't doing while they're buried in it. Agentic finance shouldn't multiply your output. It should eliminate the work that was never worth doing in the first place. And that's why I run mostly media on Brex, an intelligent finance platform with AI powered agents that capture expenses, automatically, enforce policy before the spend happens, and close your books in minutes instead of weeks. 35,000 companies like OpenAI, Coinbase, Anthropic and Doordash already run on Brex. It's time to get Brex AF. Learn more at brex.com metrics Today's episode is brought to you by Anroc, the sales tax platform behind companies like Anthropic, Notion and Vanta. Here's a fun way to totally ruin a Open a letter from a state you've never set foot in, telling you that you owe back taxes you didn't know existed. Happened to me because the rules never stop moving. States are now racing to tax AI digital ads streaming really anything new, and they're doing it faster than a spreadsheet can keep up. Anrock handles all of it. One platform that watches your exposure everywhere automates compliance and flags risk before it turns into that nasty Grammy. That's why thousands of finance leaders trust anrock to stay ahead. Talk to an Anaroc sales tax expert for a personalized exposure estimate@anrock.com RTN that is a N R O K.com RTN here's a growth tax that nobody talks about Every new pricing model you ship creates a nightmare for your finance team. Usage based pricing. Now you're tracking usage against commitments product bundles. Now you're untangling what to recognize and when for every line item. Mid cycle upgrades. Good luck. Manually reallocating revenue. The pricing strategies that drive growth are the same ones that break your finance process. Right Rev turns that irony into a competitive advantage. Your product team can ship new pricing without asking finance for permission, and your sales team can close deals without worrying about downstream chaos. But I've seen too many companies where sales are celebrating a huge quarter while finance is still trying to figure out how to recognize half of it. It's actually me. So here's a good place to start. Right Rev built a free tool@calculator.wrightrev.com it scores your RevRec process, shows what's exposing you to risk, and tells you exactly where to focus before it bites you in the rear. Probably explains why your last close took so long. Check it out@calculated.wrightrev.com all right, without the Boston accent. Calculator.wrightrev.com Micah or AJ have you seen
- [12:40]
CJ
any cool use cases with forecasting specifically?
- [12:43]
Micah Richard
What I'd say is that there are a lot of people, a lot of clients I interact with who are actively exploring how to use things like Claude code specifically to accelerate some of their forecast buildings. There are also, of course, a number of vendors on the market that support that exact use case. I think the larger problem is that there are a lot of opportunities, but it's also an area where organizations haven't necessarily expected to see a lot of acceleration, so it's not a natural place for especially larger scale companies to start.
- [13:11]
CJ
And I ask that around the ROI piece because it's kind of like the old adage you don't want to check your brokerage account every day or you may start to make some silly decisions. I sometimes wonder that if I'm a cfo, I'm like I print out the P and L, I'm like, oh Damn, we're down 2% today. Sometimes I think too much information can actually make you start to see things in a, in a blurry manner.
- [13:29]
AJ Lubich
We're a consumption mob.
- [13:31]
CJ
Yes.
- [13:32]
AJ Lubich
And you know, we have over 30,000 customers and we have something like 50 plus billable SKUs. And then you add the time dimension on top of that. Just the volume of data is just tremendous. The way that we forecast our business typically is we look at the consumption patterns every single week and we do some extrapolation at the customer level of what that is going to look like. I'd say we're not at the place where AI is able to be let off the loose of running that model, but it's a good sanity check because you need some human context of the model might pick up on some spike of some customer usage. And we understand given the context of that customer relationship that we're going to credit or maybe they're working on a deal that's going to lower their unit rates or something like that. But it's a good starting point from which then a human can pick up and say, okay, now let me overlay this and get something that's more refined and thoughtful.
- [14:17]
CJ
I think a ton of people are wrestling with how to forecast usage based models and what I'm finding is AI can pick up on a lot of the seasonality quirks. So I'm sure in your business with housing purchasing become springtime, it probably picks up. I'm sure there are holidays where usage goes down because people aren't in the office.
- [14:33]
AJ Lubich
The harder thing is that we also have exposure to the seasonality of our customers. If you have a video game manufacturer and they have a big release, there might be a surge in using there or there's like, you know, streaming companies that have exposure to a certain event or like a game or something like that. So that's something that's a little bit more nuanced that, you know, so far AI isn't picking up perfectly, but maybe over time.
- [14:53]
CJ
Dana, any quirks in your business model that AI is helpful?
- [14:56]
Dana Decker
We have seasonality with both timing of him when we buy and sell homes. But I would actually say it's almost simple for it to see it. You feed it a couple seasonal patterns, you added a few things, it can pick up on it no problem. And so obviously kind of similar to what you said, you want to influence sort of your human judgment on top of it. But the starting point for a lot of our Models and a lot of our forecasting we start there.
- [15:16]
CJ
Steven Hedlund put a big vocab word here for me. Dana, you built what you call a semantic layer in Snowflake with blessed queries. This is a lot to unpack.
- [15:24]
Dana Decker
We actually have in GitHub a repo of sort of the top like 30 dashboards at the company and behind those dashboards all the queries that are used to pipe. And so this was actually a partnership with our data science team because I think the data science team was like, oh my gosh, I don't want every single person at this company thinking that they can find every piece of data in the data warehouse. And like that's fair. And so they created this repository. On top of that we have an org wide skill similar to what some people were talking about earlier, that we can now across the board, across the org, point our skill to the repo and make sure that every query and every set of data that we pull is accurate. It's not always perfect, but like it is much closer and it hits on the right sort of data tables on top of that. What we've also done though is we've given everybody a set of instructions, at least on my team, so that when they are building they include the prompt and the queries that they use so that it can be created and it can be audited. So when I think about the sort of the semantic layer, it's like the starting scaffolding that says, okay, how do you get like a large scale people at your organization pointed in the right place?
- [16:28]
CJ
Micah, does this scare you to hear that or no?
- [16:30]
Micah Richard
It's great to hear organizations actively thinking about this because one of the things that really drives challenges with accuracy and that sort of thing when you're using AI is that inability to provide the right context. So building a semantic layer is actually a driver of increased accuracy in a way that organizations don't think about until they've gone a few use cases in and they realized actually we're not getting necessarily what we want.
- [16:53]
CJ
I'm curious what percentage of data queries in the company can be self serve realistically versus you need to have a business partner pull it for you in order for it to be correct.
- [17:03]
Dana Decker
I always start first and pull it myself. Now if I get a hunch that it's not quite right or that there's some piece that I'm missing, then I'll go talk to somebody who's much more technical than I am. But I would say to start with the majority of Them I can get myself.
- [17:16]
Host / Narrator
Now.
- [17:17]
AJ Lubich
It used to be that you really had to start with the subject matter expert and allow them to go query it to begin. Now I'd say that's really flipped where and at least especially being a public company and there's a lot of like important data that's behind a lot of these things. We're trying to instill the behavior of, you know, run it yourself, run the query, do any analysis you want, but then at least as a check come back to the subject matter expert and say, hey, did I get this right? Do these insights make sense based on the context that you know, that's been one of the more challenging parts because we want people to run quickly and do interesting work. But every once in a while you get a, you know, a misread from something and unless they're coming to a human that actually like studies those things on a regular basis, you might not have that context.
- [17:55]
CJ
I was speaking to the VP of analytics at Superhuman recently and he was saying that the goal is to get more than 60% of the requests to be self serviceable. What that does is it puts a sense of like skin in the game to the people who are asking the question. To not ask stuff that's like silly. Like, well, what happens if the weather pattern changes in like Minnesota? Like how will that, like, it doesn't matter. For what we're working on here, I think the term was blessed. Queries is the idea to make it so you have a set of questions that someone can ask reliably and get the same answer in a deterministic way every time.
- [18:28]
Dana Decker
Having our data science team help enable us, they have effectively all of the queries set so that you can ask sort of a multitude of different questions but go to the same data sets. And so from the data sets, it'll pull it from a bunch of different places in order to get an answer. So it doesn't necessarily all drive the exact same answer because you could be asking different questions, but it is all from the same sources of data.
- [18:50]
CJ
Micah, you've seen a wide range of public companies who started building two years ago and others who are still waiting for their vendors to catch up. What does best in class AI adoption look like right now in finance?
- [19:02]
Micah Richard
Well, it's interesting, you know, I rejoined the firm just over a year ago from a tech company and I was really surprised at kind of the state of the profession overall. You know, it's relatively slow adoption. Kind of the set of tools that exist in the market weren't that Great. In many cases, obviously that's changed a lot. But when you look at organizations that are on that leading edge of adoption. So some of the clients that I work with are currently scaling the use of AI in finance and accounting. In many cases, they actually made a conscious decision to focus on build rather than buy. And in many cases what that meant is that they're going to start with their own kind of internally built platform because, and to be clear, built that means that they're building on top of open source components. And the reason is kind of twofold. First, companies kind of are divided in, or at least for, for a while there were divided into two broad groups. Right. You have the companies that were like, we're going to give everyone access to Copilot or chatgpt. Yeah. And of course people will magically figure out how to save a bunch of hours. Like that kind of hasn't happened. And the other group were companies that really expected that, you know, vendors were going to save the day. It's like, hey, look, we've got the best in class. Vendors are going to rise to the top. We'll be able to cherry pick the ones that we need and that's going to solve our problem. And again, that just didn't evolve as quickly as people had expected. And what that means is that by choosing to consciously to build on top of their own platform, these companies were able to move much, much faster. And again, they're the ones who are currently kind of, as I said, scaling the use of AI in finance.
- [20:23]
CJ
Hey, thanks for listening. We'll be right back after a word from our sponsors.
- [20:28]
Host / Narrator
Hey founders and finance folks. You know how early on everything is fast and scrappy and then suddenly equity gets messy, spreadsheets break, grant docs pile up, lawyers are setting new forms and collecting fees. It all shows up right about when you're already swamped. Well, if managing your cap table feels like one more frustrating thing on a very long to do list, you need to know about Pulley. Pulley makes equity management simple and stress free. You can issue Options, model dilution, complete 409As and more, all in one place with support from real experts when you need it. So if you're raising, hiring or scaling, pully keeps your cap table clean so you can stay focused on building, not wrangling spreadsheets. Learn more. Request a demo@pulley.com mostlymetrics that is P-U-L-L-E-Y.com mostlymetrics I got news for you. The ERP category is finally getting disrupted. And if you haven't heard of Rylit yet, please pay attention. It's the AI native ERP built specifically to replace netsuite, and it's already won over hundreds of finance teams. Their mission is to make the zero day close a reality. And they're actually doing it. We're talking teams closing the books at 1:35pm on the first day of the month. Companies like Windsurf, Merkor and hundreds of others run their entire finance stack on real estate, revenue recognition, close management, multi entity, native Stripe and Salesforce integrations Woo everything a scaling company needs. They've got 5.050 wow 5.0stars on G2, they're backed by A16Z and Sequoia. Heard of them and CPA LED implementations that get you live in 45 days. That is simply unheard of in the ERP space. If your books aren't running as fast as your business, check out Rillette. Book a demo@rillet.com cj that is R-I L-L-E-T.com cj that's me. Everyone in finance is adopting AI right now.
- [22:21]
CJ
There's a problem because there are a dozen disconnected tools.
- [22:24]
Host / Narrator
You got one for revenue, one for ap, one for close, and each one of these needs training and documentation and the expertise to run them lives in your team's heads, not your systems. Maximore takes the opposite approach. One autonomous finance platform for the whole operation order to cash procure to pay the close, cash management reporting running on a single unified finance context that pulls from your ERP billing, banks, even email and Slack. Maximore runs on self learning agents that learn how your team already works. No prompts to write, no workflows to build, nothing to document. Every output is audit ready and when a call needs human judgment it escalates for review because the agents optimize, not just execute and your KPIs improve. I'm talking about DSO working capital cash conversion. In fact, one PE back customer cut their clothes in half. Their audit findings from 7 to 0 back office cost down 70% and in 6 months no ERP rip and replace. You pay only for real outcomes, not seats. See it@Maxmore AI that is M A X I M O R AI.
- [23:30]
CJ
I'm going to ask a potentially stupid question with how fast AI is moving? How much does having had a 2 year head start even matter at this point?
- [23:38]
Micah Richard
If you think about where our employee base is right now, we have a large group of people who fundamentally understand how to use a chat based interface like Copilot or a chat.
- [23:45]
CJ
Everything's a prompt bar at this point.
- [23:46]
Micah Richard
And the, the problem, though, is that there is still meaningful limitations to the capabilities of those tools. And organizations very quickly realize what those are, especially if they're using like a chat to GPD or a copilot and trying to connect it to a bunch of different sources. Like, you need people who are able to conceptualize how to apply AI to more complex problems than can solve with those tools. And what that means is they've been moving people from left to right on that capability curve over the last two years, which means that they have not just a head start in terms of the things that they've built, but also in terms of their people's ability to use it.
- [24:18]
CJ
Well, speaking of that, we talked about two years, but if you zoom in just on like the last six to eight weeks, has the build versus buy mix changed to more people building?
- [24:29]
Micah Richard
There's, of course, a number of schools of thought here. Like people who believe that, you know, cloud code is going to fully replace SaaS software. Like, I, I think, think it's difficult to guess where the market's going to go. And we get this question a lot. Like, if I need to do a build versus buy assessment, like, what are the criteria I should consider? The criteria haven't changed. What has changed is actually the velocity of change of your target, the thing you're actually potentially buying, whether you choose to build or buy. Even if something theoretically much, much better comes out in six to 12 months, as long as you're still getting value from the thing that you've built or bought, there's no real buyer's remorse in that content context. Right. If you're thinking about companies that are on the far right of the adoption curve, they got a head start and it's going to be much, much easier for them to figure out how to navigate this.
- [25:09]
CJ
So your inbox is just flooded with, you know, hungry BDRs. We were saying a lot of it's how much do you believe your vendor's roadmap? AJ when you're approached for take a look at this new tool, how much of your mind is thinking about what are they going to build in the future?
- [25:24]
AJ Lubich
Yeah, it's a, it's a big component of it. And I do think just the, the whole idea that like package software is no longer relevant, I think, you know, because I hear some of my colleagues sometimes say that, like, well, just give me, give me a spreadsheet like your baseline ERP and Claude. And like, I'M good. Which might work for some companies, but for a public company that has repeatable reporting and metrics that need definition and need to be beat up and audited and all sorts of things like, that's just not going to work today. And frankly, I don't think it's. It's going to get there. But I think what then is important is if I'm buying a package software solution which maybe entices me with, with a standard UI that has standard reporting that I can, you know, give to different departments and executive stakeholders, that's a great starting point, but it probably isn't good enough, you know, a year from now or even six months from now. But what is important is what is their roadmap, because every software company everywhere has some sort of roadmap of their own models or training, some sort of LLM off of their contextualized data. And as long as I can get that value out of that solution, plus the sort of standard reporting that I need, that gets me pretty far and where I need to be.
- [26:25]
CJ
Dana, has your attitude towards procurement for your own team changed?
- [26:28]
Dana Decker
Things are moving so quickly that we are a lot more careful about diving into anything than we are thinking about what we can build. I don't think all software is going to get ripped out. It can't. But I also think that there's a lot of applications that instead of having a small like point solution for you now can build. So I think that there's going to be a balance of the bigger tools that are super embedded that you have and that you're going to build on, and then the smaller point solutions that we are probably not going to be investing in as much, but the tools that will matter will, I think, be the complement that says we can build on top of this internally. We can build some stuff internally and we have a powerful SaaS tool that we are using in combination with it.
- [27:11]
CJ
Because I'd be worried about potentially training the wrong tool on my data and having that sunk cost.
- [27:17]
Micah Richard
That training question is an interesting one. It's actually also funny because we've now pivoted from organizations saying, well, we're super worried about OpenAI or anthropic like training their models on our data too. Like, how do we get more value out of our data? Like, it's been a pretty rapid evolution. It's not something that companies have to worry about too much. The cost of building or implementing a new vendor solution is frequently not high enough that you have to worry about, like replacing it in the near Future. Obviously if you were doing a full replacement for your erp, that might be a different conversation, but for many of the uses it's not that hard.
- [27:48]
CJ
Well, just to stay on you for a second Maika, because a lot of people in this room work at public companies or companies that they hope within the next 18 months can be a public company if the IPO gods bless us and they're saying I can't just plug Claude into my financial data and have as much fun as some of the private companies. How do you actually think about data access controls and what goes into that AI versus what doesn't? Because I do think we have crossed the chasm where a year ago if you asked me to connect my Google Drive to OpenAI or to and I'd be like are you crazy? Like I would not do that. But now I think about it just like having Dropbox at this point which is like a mind blowing shift from just six months ago.
- [28:27]
Micah Richard
The data access piece of it is interesting. There are many different models that companies choose when it comes to like controlling access, you know, AI access to their underlying data. I don't think there's really a consensus on the best approach at this point. There are many papers that have been published on the topic. Most organizations are ending up ending up with some level of rbac. You know, it's basically, it's tied to a specific user, et cetera. Now that's not where they want to be. There's also a larger question around kind of the control environment that you need to have to have autonomous, especially autonomous agents like actually performing financial stuff. And in that context, actually a lot of organizations like when they start, start to dig into it, they actually realize that their ability to think about things like data risks and process risks and legal risk is actually pretty mature. Where the struggle is really in the area of model risk risks and use risks. The model risk being things like hallucination and use risk is like people over relying on AI output. When companies are starting to go down their first three to five use cases, they quickly realize that they actually need to invest in things like monitoring tools. We didn't have to monitor the quality of our ERPS processing because it was deterministic. And in a probabilistic world like investing in things like monitoring and then also like what that looks like in your broader control environment is where companies then need to spend a bunch of time. These models are predicting the next most likely token based a sequence of tokens and they're doing it in a probabilistic Manner, which means you don't get the same outputs from the same inputs. And what that means though is a lot of the traditional thinking around user acceptance, testing, post change management, giving you a significant amount of comfort around how an application is going to function. That logic doesn't hold up quite so well. One of the ways you can address it is by having effective quality monitoring of your use of AI. Now you need a system or a technical component to do that and you need to have a process around that as well. But as I said, it's, it's a muscle that a lot of accounting and finance orgs haven't had to build in the past.
- [30:15]
CJ
Well, that was even my question. I don't have that person on my team.
- [30:18]
Micah Richard
To be fair, for a lot of companies, especially tech companies, it's very common to already have an observability platform and many of those do actually directly allow you to perform these tasks. For companies where that's less of a need, you know, sometimes it means you need to go through it to make sure that you have the right, the right system in place. As long as you think about it far enough in advance, it's a pretty easy thing to solve. Companies don't think about it until they've gotten pretty far down the path and then it's harder to go back and like figure out how to retrofit it.
- [30:44]
CJ
AJ can you talk about data warehouse access controls? The R back story?
- [30:49]
AJ Lubich
Yeah, I'd say we're still in a process of discovery and I think like most places this has kind of really become forefront of what we're thinking in the last couple of months. So by nature we're a growth company that's product focused and we like to move quickly. So I think we've taken the approach of being iterative about it. So RBAC is very important.
- [31:06]
CJ
Sorry, what is rbac?
- [31:07]
AJ Lubich
From my own rule based access control.
- [31:10]
Micah Richard
There we go.
- [31:10]
AJ Lubich
But basically the idea is that as a non technical person you're basically limiting each user to what they can see and act. Yes, it's important because everything we're doing sort of links to the underlying data warehouse. So whatever the user's training, LLM or querying from an AI provider, it's going to tie to whatever access that they have for the data warehouse. We've played a little bit with what if we like, like maximize access and let people kind of off the leash. Then we look at what access is and kind of roll things back. Inevitably things break a little bit in that cycle and then you know, you see A slack channels are sort of all exploding of like hey, why can't I see this thing that I'm used to seeing? So I feel like we're in that phase of trying to find the right balance of you find some legitimate use cases that come up in that of folks that they have a legitimate reason why they need to and you learn about okay, we need to modify slightly here or there but it's, it's a balance that frankly we're still, we're still on a journey and haven't, haven't solved so far today.
- [32:04]
CJ
I didn't mean to put you on the spot man. I honestly thought rbac, I thought RBAC was like a dish at Outback Steakhouse. Token usage, what patterns are you seeing?
- [32:12]
Host / Narrator
A.J.
- [32:12]
CJ
just to stand you for a sec,
- [32:14]
AJ Lubich
back to my point of just being a product focused org, I'd say it depends on the organization. We've had a pretty significant top down mandate specifically from our CTO to really get into the vibe coding applications and, and the idea that that's the way that our engineering organization is going to be headed into the future. So we've seen the token usage in our engineering department has one exploded. But if you look at the usage by person, you definitely see this sort of haves and have nots and you see some power users.
- [32:41]
CJ
Can you say more about that? So you can look through and see who on the team is going nuts, right?
- [32:45]
AJ Lubich
Yeah, you can. Now the thing that we don't have perfect visibility on today is what they're doing with it. So right now still in that like exploratory phase of you know, we're encouraging folks to use tokens, I think similarly be sort of creative and go out there and conquer and understand sort of what the right balance is. I think the day will come eventually where we start saying well hey, why are you using so much? Like have you actually shipped product? Is it high quality? Like what features have you delivered on? We're still trying to figure out internally exactly how we define roi, you know, what the stage of how we go after those things, what good product versus not looks like. But we're happy to be on that journey and not to find those things quite yet.
- [33:24]
CJ
Dana, what are you seeing?
- [33:26]
Dana Decker
We're encouraging it. We are tops down, we're asking everybody to do it. We have given everybody access. Obviously our engineering organization uses the most but when we look at sort of the non engineers, we actually look similarly. We can see who uses tokens and how they use their tokens. And there are some super Users. And when you look at those super users, maybe there's not like a direct connection, but there are some super users where you're like, yeah, they actually did spin up that analysis and that dashboard and that decision and whatnot. And so you can start seeing those who are using it a lot and some of the impacts that they're doing. It's not a direct ROI by any means, but the token usage across the board is very different. Heaviest with engineers and then the non, the top to the bottom, you can see who's really active in it.
- [34:11]
CJ
Other than engineering, who would you say? Which department is up there?
- [34:15]
Dana Decker
Our chief growth officer. A lot of our executive team and our leaders at the company are the ones using the most because I think that they're ones sort of leading at the front. And we are being asked as leaders of the company, and I'm sure all of you will be asked to, as leaders of your company to go ahead and use and use the tokens.
- [34:32]
Host / Narrator
I remember OpenAI, they had their user
- [34:35]
CJ
summit and it had the top 50 users of all the tokens. And it reminded me of that scene from the Big Short where the mortgage guys are in there and he's like, I can't believe they're confessing. He's like, they're not confessing, they're bragging about it. So it's funny that we've gotten to this point that it's a status symbol of how many tokens you can actually consume. Mike, are we just going to invent new metrics for roi?
- [34:53]
Micah Richard
It's interesting because from my perspective, the metrics really haven't changed, right? Our ways of measuring hard and soft roi, the specific metrics you'd use, you would use for measuring that ROI are not that different than for any anything we've done in the past. Really the problem, there are different types of adoption metrics certainly, but like true roi, not really. But I think this is a common thread with pretty much all AI related topics where it's like, how do I actually consider, think about build versus buy considerations, et cetera. It's like just because the word AI there is in there, people are like, what do I do? And the reality is that there's a lot of very transferable knowledge from kind of everything we've been doing in the past. There will be some small tweaks and in some cases you will. You'll need to think about things slightly differently. But yeah, at the end of the day, when people are calculating ROI, it's based on the traditional factors.
- [35:40]
CJ
If you have 90 days to go and work on something to improve next year. What is something that they can do?
- [35:46]
Micah Richard
Fortunately or unfortunately, right now, for many people, the answer is just starting. Start, like, your company almost certainly has a set of tools that are in place. If your teams aren't actively using them, it's the easiest place to get started because the problem is that the longer you wait, the harder it's going to be for you to adapt. You can get expert help, which is what some organizations have chosen to do. It's like we don't, especially when you start thinking about more scaled uses. So, like, individual use is one thing, whether that's going to be a quad code or copilot or chatgpt or similar. But scaled use, where you're actually potentially building lightweight applications or larger scaled applications, a lot of companies struggle to get those for a first couple of those off the ground. So there's always a question about does it make sense to bring someone in who has that expertise and whether that's a new person or whether that's kind of, you know, an external, like a consulting firm or similar. It's valuable to at least think about it. But you know, in terms of personal, personal use, grab whatever tools your company has, give them a try. And also, to be honest, think about whether they're working for you. Because many of those tools, they're not going to directly solve finance problems unless they've been set up correctly to do that.
- [36:51]
Dana Decker
I think that there actually has to be a mentality shift at your companies that just says you're going to go do it. Obviously we say start, but where does that start comes from? Like, where does that activation energy comes from? And I actually think you need a mentality shift starting with your, with your leaders. Whatever your executive team or your leaders, they need to go, because I think the teams need to know that, like, that you've pulled some of the red tape that they have access to go do it. So I do think over the next 90 days, shift your mentality, start shifting your leadership, start pulling away the red tape.
- [37:20]
AJ Lubich
If you don't have the data set up in a clean way that's actually readable and your metrics defined, do that. Like, don't even, don't even get started on the AI piece yet. Like define in your data schema, work with a data team, define your metrics so that you can layer AI on top and be productive from there if the data scheme is already set up and you're empowered to do so. I think my tactical steps, which, which we've started doing on the team is a lot of the gating factors really is just time and energy of getting enabled on these tools. We've kind of done two things. One is to start just asking the question, especially from like a senior person to a more junior person. Every time a project or a task comes up, just ask the question like did you try this with Claude? Did you try this with OpenAI? And it just changes the mindset of they're so stuck in a cycle of just doing things in an Excel spreadsheet or their standard way of doing things just sort of shifts the mindset. And the other one is just getting a little bit of like, like hackathons or idea sharing sort of venues because a lot of folks are like I want to, but I just don't know where to start. We just pilot everybody in the room together and say like this is a no dumbs question, like safe environment. Let's all just like work on some projects, ask some questions, help each other. And that's the multiplier enablement event that I've seen actually really work for, for our team.
- [38:29]
Micah Richard
For a lot of organizations, getting the data right ends up being a major barrier in people's minds because they're like, well, we've. I have to fully solve data governance and I have. It's basically the unified feedback field theory of data. At that point, right when you start peeling the onion, what they quickly realize is that even if they don't have everything, there's at least going to be pockets of well controlled, accessible data. And if you start there and you also layer on top of that some level of enthusiasm, like one person in a specific area, like within your function who's excited to do this, where they have access to the data, they can actually move really, really quickly and help kind of your teens as a whole move faster.
- [39:06]
CJ
We're gonna end on that. Thank you for having me.
- [39:07]
Host / Narrator
Run the Numbers is a mostly media production yelling an intro by Fat Joe. Artwork by Meg Delesandro show is executive produced by Ben Hillman. Nothing said on this podcast is intended to be business or investment advice. It's the sole opinion of me. A guy who feeds his dog way too much ice cream and has a history of net operating losses. Lol. If you like this podcast, hit subscribe
- [39:29]
CJ
and give us five stars.
- [39:30]
Host / Narrator
It will take like two seconds and our algorithm overlords love it. Drink water, call your mom and have a great day.
- [39:36]
AJ Lubich
Peace.