← The Pragmatic Engineer
Loading summary
Transcript133 lines
- [00:00]
Farhan Tharoor
We've been using AI tools for a long time in engineering. I believe we were the first company outside of GitHub to use GitHub Copilot. And the reason, really, yeah, the reason I know that is because when Thomas Domke became the CEO of GitHub the same day I emailed him saying I would like GitHub Copilot. This is 2021, a year before ChatGPT. He messaged me back saying it's not available for commercial use. And I said, that's not what I asked. I don't care what you have for commercial use. I would like this deployed for all Shopify engineers as soon as humanly possible. I think it took them about a month and we were not charged for two years because there was not a SKU to charge us. And we said in exchange we'll give you lots and lots of feedback. And we were using Copilot for a long time and then we worked with them as we started looking at the roadmap. We then deployed Cursor internally as well. Actually, we're all very new to cursor about a year into Cursor, so we are kind of trying all these things and as we try them, we want to see what's working, what's not working, and then we let more engineers use it. The most interesting thing about Cursor is that the growth in cursor at Shopify is happening a lot outside of engineering and outside of R and D finance sales support. Those are the teams using Cursor.
- [00:59]
Podcast Host
What happens when a company goes all in on AI? Shopify did exactly this a few years ago and Shopify's head of engineering, Farhan Tavar, told me exactly how it's going so far. In this episode we discuss how Shopify works closely with AI labs and why Farhan paired for an hour with an engineer at Anthropic working on the cloud code team, why Shopify is planning to hire 1,000 interns in a year, and how they incorporated AI into their interview process, why Shopify has no cost limit on how much an engineer or team can spend on AI tokens and many more. If you're interested to know what an AI First Tech company operates like, this episode is for you. This podcast was recorded as a live podcast at LDX3 in London. If you enjoy the podcast, please do subscribe on your favorite podcast player and on YouTube.
- [01:42]
Interviewer
So welcome everyone to this very special live podcast with Farhan Tharoor. And we're going to talk about AI at Shopify. But before we Start about AI. I just wanted to talk a little bit about you, Farhan. So I did a little bit of research. I talked with a couple of engineers at Shopify and they told me, you know, your role is the head of engineering. And engineers told me that they see you in everything. You have reworked Shopify's intern hiring program. You've defined what Shopify does. In fact, you even were in the weeds to deploying an internal hackfest to more than to getting the WI fi right for a hackfest. So I'd like to know what does a head of engineering do at Shopify and specifically what do you do? Or maybe what do you not do?
- [02:26]
Farhan Tharoor
Yeah. So I think what's interesting about Shopify is that we use this line. We're not a swim lane company, which means that we don't try to put people into these roles where you are like you're in product, so only think about product or you're in engineering only think about like how the code is written or architecture. We're very much just curious problem solvers. And so if something's broken, we expect curious people to go and look at the problem even doesn't matter what their role is and try to solve it. And so last year at our Shopify summit, which is our employee event where, you know, know 7,000 people come to one space, the Wi fi did not work super well. And so I spent all my time in that summit for three days trying to fix the WI fi. I became known as the chief WI fi officer. And so this year, which was, it was two weeks ago, I deployed 300 Ubiquiti APS800 switches in order to have access points. Access points? Yeah, to have a much, much smoother WI fi experience, which is how that all got started in terms of. And they had the team built lots and lots of memes because they thought the WI fi was not going to work. But it did work and so I did publish all those memes.
- [03:28]
Interviewer
I need to ask you because a lot of leaders would say, look, your time is super valuable. You're kind of ahead of an organization. How many people are 3,000, 3,000 people? You're kind of up there and your time is now. If you measure it in dollars, it'll be expensive. It's a lot better to just get a specialist to do these things. Why do you still do these things? And do you disagree with that advice of focus on the high leverage things?
- [03:52]
Farhan Tharoor
Yeah, so we don't love the. There's that notion of like hire smart people and get out of their way. Instead, what we say is like hire smart people and pair with them on problems. So the difference for us is we really like to bring in obviously lots of smart people, hands on people, and instead of just saying hey, you take this area and then go away and just solve it for me and come back, we like to pair with them and say you're smart, I'm smart. Can we work on this problem together?
- [04:13]
Podcast Host
This episode is brought to you by Work OS if you're building a SaaS app, at some point your customers will start asking for enterprise features like SAM authentication, skim provisioning and fine grained authorization. That's where Work OS comes in, making it fast and painless to add enterprise features to your app. Their APIs are easy to understand and you can ship quickly and get back to building other features. WorkOS also provides a free user management solution called AuthKit for up to 1 million monthly active users. It's a drop in replacement for Auth0 and comes standard with useful features like domain verification, role based access control, bot protection and mfa. It's powered by Radix components, which means zero compromises in design. You get limitless customizations as well as modular templates designed for quick integrations. Today, hundreds of fast growing startups are powered by WorkOS, including ones you probably know like Cursor, Vercel and Perplexity. Check it out@workos.com to learn more. That is workos.com if you want to build a great product, you have to ship quickly. But how do you know what works? More importantly, how do you avoid shipping things that don't work? The answer Statsig statsig is a unified platform for flags, analytics, experiments and more, combining five products into a single platform with a unified set of data. Here's how it works. First, Statsic helps you ship a feature via feature flag or config. Then it measures how it's working, from alerts and errors to replays of people using that feature to to measurement of top line impact. Then you get your analytics, user account metrics and dashboards to track your progress over time all linked to the stuff you ship. Even better, Static is incredibly affordable with the super generous Free Tier, a startup program with $50,000 of free credits and custom plans to help you consolidate your existing spend on flags, analytics or A B testing tools. To get started, go to statsic.compragmatic that is S-T-A-T-S-I G.compragmatic Happy building.
- [06:11]
Interviewer
So this leads me really nice into the next Thing I heard about you, which is at Shopify, you have gone kind of AI first, and we're going to talk a little bit about that. But one thing that you do is you have access to some of the top AI labs. And what you told me just before, that you pair with them. You just told me that the other day you were pairing with Anthropic engineers. Can you tell me about this whole pairing and how you work with these AI labs?
- [06:35]
Farhan Tharoor
Yeah, because again, we're like super curious problem solvers. And. And so when there's something interesting happening in the industry, we want to be close to it. So obviously, ChatGPT comes out and then Anthropics builds their model. Gemini has their model, Cohere has their model. We want to be as close to the leading edge as possible. And so we try to be close to those people and what they're working on. And one of the things we do is when something comes out. So in this example, it was cloud code. We deployed it inside of Shopify. We saw people use it.
- [06:59]
Interviewer
This was in May, right? Three or four weeks ago.
- [07:01]
Farhan Tharoor
Exactly. Well, so cloud code came out before that, like, let's say six months ago. And then we started spending time with Anthropic. I wanted to see how Anthropic was using cloud code internally, similar to, like, OpenAI is now releasing codecs. I want to figure out how they're using it internally. And so what I did was I reached out to Anthropic and said, hey, I'd like to pair with one of your applied AI engineers. We spent an hour building a bunch of things, and I learned how he sees Claude code being used internally in Anthropic. Then I can take that back and say, well, here's how we're using it at Shopify and we can together figure out where we think this thing can go.
- [07:32]
Interviewer
Is this just a matter of, like, you just asked them?
- [07:35]
Farhan Tharoor
Yeah, I mean, because we are Shopify, we're very lucky. People want to see how we work. We tend to work in an unorthodox way. And they also are interested in how we use it, but they also use it. And so it was more of a me, again, a pairing. It was a true pairing. It wasn't like me as a customer trying to push them and say, I really want it was, we have a shared Slack channel. I pinged in there saying, hey, who'd want to pair with me for an hour on something I'm building inside Shopify? So somebody put their hand up on Applied AI Engineer and Said, we'll show you how we work and we show you how we work and we can together figure out where we think this product could go. I do that with a lot of the providers.
- [08:07]
Interviewer
And as you're pairing, is it just you pairing and then you later tell people? Or like, do you.
- [08:11]
Farhan Tharoor
Yeah, I might record the session and then share it with the internally to say, hey, I just did a pairing session with Anthropic. They might use it for a case study that they're building. But the idea is, again, because I'm just curious, when somebody says something, I'm like, oh, how do you. How does that work? How does that. How can we use this? And so I think it's a way for us to just learn.
- [08:28]
Interviewer
And one recent thing that I just wanted to go into before we get into some more of the AI things, it's a really fun fact that most people don't know. You just came out of a Code Red for seven months. I think this is the first time we're building. What was this Code Red? What was it about? And what did people outside and inside of Shopify see of it?
- [08:47]
Farhan Tharoor
Yeah, so we were internally seeing a lot of things that weren't bubbling to the surface, but that can actually affect the operation of a system. So you would typically call it tech debt. But we had a lot of signals showing that the tech debt was growing, not just the general ones, like taking longer to update a piece of software or build a version two of a product, or people having a hard time maintaining some of the software stack. We saw things even, not externally, but internally, that were shooting out signals. Lots of exceptions growing, even in some cases, because we build things at the very low level of the stack. So, for example, we patch we core contributors to Ruby. We patch MySQL I believe we have the second largest MySQL fleet in the world outside of Meta. And we were seeing things like seg faults and things like that, that I was, like, not willing to feel like we should move on and just build features as normal. And so we actually got together, spent. We didn't think it was gonna be seven months, we thought it was gonna maybe be three months. And we took something between like 30 to 50% of engineering. We didn't have an actual number. We just said, these things have to not grow anymore. Like exceptions, unique exception counts have to shrink. There should be zero seg faults. We should understand everything that's happening in the system. And so we spent almost seven months doing that over the last, you know, from November till Now how did you.
- [09:59]
Interviewer
Make sure that you fixed the right thing? Did you like talk to the team saying, hey, bring up the things. And also how did you decide when you will stop? Because like whenever I hear these code reds, it's always, you know, a lot of companies do code yellows, code reds, but sometimes they can drag on forever.
- [10:12]
Farhan Tharoor
Yeah. So we had multiple metrics that we used. One was literally what I said, Exception counts, unique exception counts and seg faults, which we literally would say, like seg faults are at zero now and exceptions should be going down. But we also looked at other metrics that we have. Like, you know, we have four, we want to have four lines of reliability across our different surfaces, storefronts, merchant admin, point of sale. Like we looked at all these things and we also used the 28 day rolling average of those things and once those were all green and we saw that again, seg faults was zero and unique exceptions stopped and was shrinking. We saw, then felt like we were in a place where we can start.
- [10:46]
Interviewer
Building features again and now going on to AI. So you're a huge early adopter of AI coding tools. Can you tell me on what tools you started to use and what you're using right now and what you're liking, what engineers are liking specifically? So focus more on software engineering, but maybe even outside of software.
- [11:05]
Farhan Tharoor
Yeah, so we've been using AI tools for a long time in engineering. So we were the, I believe, believe we were the first company outside of GitHub to use GitHub Copilot. And the reason, really? Yeah, the reason I know that is because when Thomas Domke became the CEO of GitHub, the same day I emailed him saying I would like GitHub Copilot. This is 2021, a year before ChatGPT, he messaged me back saying it's not available for commercial use. And I said, that's not what I asked. I don't care what you have for commercial use. I would like this deployed for all Shopify engineers as soon as humanly possible. I think it took him about a month and we were not charged, I think for two years because there was not a SKU to charge us. And we said in exchange, we'll give you lots and lots of feedback. And so we were using Copilot for a long time and then we worked with them as we started looking at the roadmap. We then deployed Cursor internally as well. Actually we're very new to Cursor. About a year. Really.
- [11:52]
Interviewer
Yeah, they were a tiny startup Back then, yeah.
- [11:55]
Farhan Tharoor
So yeah, I think that's super late. You're saying it's super early. I feel like it's very late to Cursor. We knew about Cursor for a long time. We like to have one tool at Java Fi for the things that we're doing. And so we don't like to have multiple tools. So like we have like figma, we don't try to have lots of other visualization Tools, we have MySQL we don't have other databases. So we try to focus on one tool. And so our bet was VS code. But because of AI's proliferation and we don't know what's going to happen, we then started making a change in our stance around trying more tools than just one. Maybe we consolidate, maybe we don't. But right now we have Cursor and VS code as our two AI tools. And then cloud code came as the agentic workflow and we started using cloud code. We tried Devin last year. So we are kind of trying all these things and as we try them, we want to see what's working, what's not working, and then we let more engineers use it. The most interesting thing about Cursor is that the growth in cursor at Shopify is happening a lot outside of engineering and outside of R and D finance. Sales support. Those are the teams using Cursor.
- [12:54]
Interviewer
What did he use it for?
- [12:56]
Farhan Tharoor
So what's happening is there? Because cursor is actually so ubiquitous on building things, they're using it to build MCP servers outside of regular engineering domain. So for example, Salesforce, Google Calendar, Gmail, Slack, they're using, they're building MCP servers to access those services and then building like homepages for themselves. Like you're a salesperson, connect to Salesforce, connect to Google Calendar, connect to my email and tell me what opportunity I should be working on.
- [13:26]
Interviewer
Just going back. With an MCP server, when you have a service, you can put an MCP server and then you can access it with, for example, with Cursor or with an agent.
- [13:33]
Farhan Tharoor
Exactly.
- [13:34]
Interviewer
Do engineers help build that MCP server?
- [13:36]
Farhan Tharoor
Sometimes yes, but a lot of times no. Now that these services are coming out with their own MCP agents so that you can literally just download GitHub's MCP server. Right. You don't have to actually build it yourself, but a lot of times they're actually just building it themselves and, and they're watching like videos and they're actually just deploying it on their own without any engineer intervention. At all. Now it's end of one, right? They're building software for each individual person, not like infrastructure for all people.
- [13:58]
Interviewer
Does this remind you of anything? You've been around in the tech industry for a long time, but the fact that the non technical people are now kind of building. Do you see any parallels from the past of did this happen before or is this new?
- [14:10]
Farhan Tharoor
This coding part is new. Although you remember like WYSIWYG editors and people starting to self serve, I think it reeks of that. But they are. I mean they're not reading the code right. They're really much Vibe coding and if it doesn't work they just delete and start over. But even if you read Anthropic's documentation on cloud code, like they say one third of the time you can kind of get something working with one shot. And so we're starting to see that. And people are getting used to the fact that if it doesn't work they just delete and start over and try to get it working.
- [14:38]
Interviewer
I find this really interesting on how this is happening. Is it changing dynamics with engineering? Because one thing I notice when working at places like Uber, the everyone was a bit jealous of engineering. Like they wanted engineering resources, they couldn't get it. They really want one of these things. Do you see any of this change?
- [14:54]
Farhan Tharoor
I'm going to say yes and no. So there's a weird thing happening whereby. So let's say you're a pm, let's say you're a technical pm. You now probably have enough knowledge to Vibe code some new feature you've been waiting for Engineering to build for you yourself. The question is, can you submit a priority and should the PR be accepted by Engineering? I'm going to ask you first and then I'll tell you what I think.
- [15:18]
Interviewer
Well, I think you can always submit a PR and Engineering will tell you why. It can either be accepted if it's good or what is missing. May that be conceptual, may it be coding standards and all those things. But at least this feedback for them, they're seeing what this person wants to do. I think it should be a great thing.
- [15:38]
Farhan Tharoor
Yes, I'll say yes with one caveat. And the caveat is the problem with Vibe coding today is you might generate 10,000 lines of code for a very simple application feature that now the burden on engineering to read the 10,000 lines is there. No, what we say is yes, you can submit a pr, but you have to understand the code you are writing yourself before exactly before you submit it. Because I Could do the same thing. And I can say, hey, I'm going to write a blog post, generate 20 pages and then send it to you and be like, hey, edit this and post it on your blog. And now you have the burden of having to. Now if I said I read it all and it is my voice and I agree with it, then you might be more likely to read it.
- [16:20]
Interviewer
The interesting thing that we're probably going to see this problem with open source projects because it's so easy to use AI tools and they're going to get a lot of these things. I think policies like this are interesting.
- [16:28]
Farhan Tharoor
Exactly.
- [16:29]
Interviewer
Now I wanted to get your take on one thing which is going around in social media. There's a lot of people predicting it's the end of SaaS because people can create their own SaaS. Now what you told me is some of your non technical people are creating like small solutions for themselves. For themselves. How do you see this potentially changing SaaS vendors? Or are you, you know, and you're also someone who actually, you know, buys a lot of SaaS vendors, you understand them. Do you think this changes anything or not really? Is it just more personal software?
- [16:58]
Farhan Tharoor
Yeah, I'd like to think we buy less SaaS than most companies only because we try to consolidate down. That being said, there is this notion we're in this middle zone right now. The middle zone is basically this idea that you can vibe code something maybe for yourself. It's unclear whether you can vibe code as a platform or vibe code something that gets into the infrastructure layer of what you're building because you do really have to understand again, like you mentioned it earlier, you might be putting yourself in a very precarious situation. If you're building on top of something and building on top of something, you don't understand what you're building. Right. Maybe it's a prototype, that's fine or a proof of concept, but if you're building infrastructure for the Internet, you likely want to understand how that's being built. So that is not. We're not there yet. Is it coming? Yes, it's for sure coming where someone can vibe code something and the, you know, anthropic or OpenAI or Gemini or who know, who cares who model build something that's actually architecturally elegant and the right architecture for what you need. That's we're not there today. So where we are is this notion of still human in the loop and still like start over and like this is not what I meant. And like prompt, prompt engineering. English as the Programming language. And so I do think you can get yourself very far. But we're not. I'm not worried yet about sas. I think we. I don't think we should be worried, by the way, because let's take another view. How much software do we think should be? Should there be in the world? Probably 10,000 times as much as there is now. A hundred thousand. Like there's a lot of software that should be in the world and we are, we are satisfying 0.0001% of the demand. So now that everybody can generate software, we should welcome them right into the software world. We believe everyone should be writing software, just like my sales team is and everybody becomes more productive. I'm not yet worried about the software industry. I still believe this is Jevons Paradox. The more we get, the more we want.
- [18:40]
Interviewer
Yeah, a really good analogy that has stuck with me is Simon Wilson was saying, how has the filming industry changed with this? Everyone has. This used to cost like 5 or $10,000 a few years ago, this camera. Everyone has one of their pockets have. And you know, there's still a professional movies industry, but now there's a bunch of, you know, there's YouTube there, there's TikTok, there's all these things. So it has become bigger, but the barrier for the pros has not really changed. You know, Dune and some of these amazing movies are so hard to make.
- [19:09]
Farhan Tharoor
Jevons Paradox. The more you have, the more you want. We all want to be filmmakers now. We can be filmmakers and then there's still going to be people at the high end of the low end. The. I would say the controversial, maybe controversial spicy take is that these phones actually benefited the experts more. Meaning the camera like base, you know, like the being able to pull out your iPhone and take a. Take a video actually benefited those who were really, really good at it already. And the same thing is maybe potentially true of vibe coding. It's like the AI agents are going to help the best engineers more than the mediocre engineer.
- [19:44]
Interviewer
So there was a memo that went around in the media about a month ago, Toby sent a memo. It was meant for internal consumption. It was about reflect of AI usage and I guess it got leaked and until we posted the whole thing. And this is about saying that, hey, everyone is expected to use AI at Shopify. Can you tell me why you felt that memo was sent out? What was their response internally and has it changed anything or was it just kind of stating what was there already?
- [20:11]
Farhan Tharoor
Yeah, it was a fundamental statement that this is Something new. And it was meant to get everyone to move from passive to active. Right? So yes, there's AI in the world and ChatGPT and all these things existed. But we started building infrastructure internally to make it easier to use. And we ex. We basically said the following. You don't have to use AI in your, in your workflow, but we're going to expect that you have this plethora of tools available to you such that the expectation is you're going to be like, your impact is going to be evaluated as if you had the tool. So for, for example, you know, imagine not having like Excel or Google Sheets and you had to like, you know, like do a complicated analysis. You don't have to use those tools. You could use a pen and paper. But we're going to expect that you have these tools available and we expect the level of analysis to be treated as if you had these tools. And so by us doing it, what changed internally was actually probably more of a change outside of R and D. Like I mentioned, R and D was already leaning in. Now of course there were some folks who were waiting for, I don't know.
- [21:05]
Interviewer
R and D is engineering.
- [21:06]
Farhan Tharoor
R and D is engineering product data and design. They were more likely to use these tools because they tend to be more forward looking and wanting to use the latest and greatest. However, some people were waiting that it changed their shape because we said, hey look, we're going to expect that you use these. But outside of R and D, we saw it even more again. Sales, finance, like customer success help centers, like all those roles started using AI as well. And we again gave them the infrastructure to use that. So we have like an LLM proxy. It has all of the APIs and all the models available. You don't have to be worried that your personal information is being leaked to a model. You can kind of use those tools.
- [21:41]
Interviewer
Let's talk about these things. Let's talk about the tools that you either create or adopted early. I heard, can you talk a bit more about this LLM proxy? And also I heard something about mcp.
- [21:50]
Farhan Tharoor
Yeah, so we are. When I talk to people. So let me start with LLM proxy. The problem at the beginning was people would go to ChatGPT or Gemini or Claude and they would like, you don't.
- [22:00]
Interviewer
Want to put customer data there.
- [22:01]
Farhan Tharoor
You want to put customer data there or even employee data. You want to use it to write your employee reviews. You don't want to put like, you know, Simon, you know, is having problems with this and like put their first Name and last name. And here's the project they worked on and the code names leak. So we wanted to quickly have an internal LLM proxy. Plus we built on top of LibreChat, which is the open source negative chat product, and we're core contributors to that as well. And with the LLM proxy, you're sure to be using the Enterprise APIs. We can check to see which you get a token, you ask for a token, we can see which teams are using how much, like what, what cost is being incurred by that team or by person. We have a leaderboard where we actively celebrate the people who use the most tokens.
- [22:39]
Interviewer
Oh really?
- [22:39]
Farhan Tharoor
Because. Yeah, because we want to make sure that they are. If they're doing great work, of course, if they script like that's not what I mean. I mean they're doing, they're doing great work with AI. I want to see how did they spend $1000 a month in credits from via cursor or maybe that's something, they're building something great and they have an agent workforce underneath them. So we have, we have this proxy and again, it's used to build anything that you want to build and you can ask for a token and use it. And then we have, we, we are very early on mcp. We're on the MCP steering committee. We are fans of MCP ing all the things. And so when I talk to people about how to get access to some piece of data inside the company, we quickly will spin up an MCP endpoint for them so that they can just use it. So for example, our internal wiki is called the Vault. It has information about all of our projects that are happening. It has information on every internal talk that has been done. It takes the transcripts of all those. So you could say something like, when did the point of sale at Shopify launch?
- [23:33]
Podcast Host
And.
- [23:33]
Farhan Tharoor
And it'll come back and it'll say, oh, in 23rd, 2012, there was a hackathon where the point of sale was started. And then in the 20 based on wiki docs, based on the 2013 Q1 board letter it was mentioned and it would tell you all of the history of what happened in Shopify. And so we do that by just having all you do is put up an MCP. And now the LibreChat has.
- [23:50]
Interviewer
It's like an internal perplexity or something like that.
- [23:52]
Farhan Tharoor
Exactly. And so we. Basically any internal document is crawled by this thing and now you can have access to it by mcp. And so any new system comes online, we Put an MCP in front of it and now everybody has access to it.
- [24:03]
Podcast Host
This episode is brought to you by Sonar, the creators of sonarqube, the industry standard for integrated code quality and code security that is trusted by over 7 million developers at 400,000 organizations around the world. Human written or AI generated, Sonarcoube reviews all code and provides actionable code intelligence that developers can use to improve quality and security. Early in the development process, Sonar recently introduced SonarQube Advanced Security Enhancing Sonar's core security capabilities with software composition analysis and advanced static application security testing. With advanced security, development teams can identify vulnerabilities and third party dependencies, ensure license compliance and generate software bills of materials. It's no wonder that sonarqube has been rated the best static code analysis tool for five years running. So join millions of developers from organizations like Microsoft, Nvidia, Mercedes Benz, Johnson and Johnson and ebay and supercharge your developers to build better, faster with sonar. Visit sonarsource.compragmaticsecurity to learn more.
- [25:04]
Interviewer
And how many MCP servers do you have given?
- [25:07]
Farhan Tharoor
Two dozen.
- [25:08]
Interviewer
Two dozen? Yeah. And they're growing quickly.
- [25:10]
Farhan Tharoor
They're growing because people are building a new thing again, one for Salesforce. There's one's for, you know, Figma. There's an MCP in front of anything. But I tell anybody who's looking at building AI systems internally, the way to make this easy is start putting MCP in front of everything. And now your data is accessible. And we Happen to use LibraChat, you might use something else, but you don't have to be an even in cursor. Right. You just use the chat interface.
- [25:30]
Interviewer
Yeah, you can do chat, you can do the LLMs. And I like how you spun up basically a platform team to build the infra to do monitoring, to worry about things like a security, decide on models, all these things. Does this team have a name? Is it some AI platform or it's just a platform team? Infra team just took it over.
- [25:48]
Farhan Tharoor
Yeah, data platform. I don't know if I don't even. I'm trying to figure out because they're inside the data platform team. I don't know if they have a name outside like I call them LLM Proxy, but.
- [25:55]
Interviewer
And was this more like you or someone upset, like hey, we need to do this or are they just like hey, this is a good idea, let's kind of bottom up.
- [26:01]
Farhan Tharoor
Or it was more top down? Because we felt like this was going to grow internally. We wanted to make sure that we could we didn't want people signing up for their own personal tokens. And it's also a good way for us to see, hey, this team is growing. Is the usage growing? What's happening there? This person's usage is.
- [26:16]
Interviewer
Yeah, because number one pushback I get from people or even engineering directors working at companies is the data security issue. And obviously when you host it yourself, it's good, but then you need the infrastructure. So it sounds like. At what point did you decide. What was this month ago? Was this more than a year ago to like, probably invest?
- [26:33]
Farhan Tharoor
Probably. I mean, I would say every week it's getting better. Right? But I would say probably a year ago we started thinking about it this way. We had the LLM proxy when. When ChatGPT came out. We actively have a warning. If you go to ChatGPT says, hey, by the way, like, you can go to the internal version. That was a long time. For a long time.
- [26:49]
Interviewer
Now, you mentioned cost, and I am hearing stories of CTOs, engineering leaders saying, okay, we were paying for GitHub Copilot. I don't know, it's $10 or $20 per month per engineer, but we have all these seats or maybe we have for everyone. And they're telling me cursor feels a bit too expensive because they're now assuring 30 or 35 and we're not going to go there. And generally there's a big pushback on costs. People are saying, look, AI should make engineers more productive. But it's now it's kind of ridiculous to spend like $1,000 on an engineer. You think about cost differently. Tell me about it.
- [27:19]
Farhan Tharoor
Yeah, I did a sweet storm on this because I think that people are looking at it differently. So think about it this way. If I could give you a tool that could make your engineering team more productive by even 10%, would you pay for it? The answer is yes. Would you pay $1,000 a month for it? Is the question. And my Hypothesis here is $1,000 a month is too cheap. Like, it's too. If I can get 10% more, it is way too cheap. Like, which should be $5,000 a month. It's insane that people are looking at this, like, $100 a month, whatever. Now, I did meet somebody who said their engineers were spending $10,000 a month per engineer. And I said, I want to talk to them because I want to see either they're doing something very, very smart or very, very stupid, but I want to meet them. And so I literally said, please introduce me, because I'm Unclear how you can use that much like either they've got fields of agents building amazing things, or something is going very, very wrong. But I would like to understand because there's something interesting happening and you should not be penny pinching on AI tools because actually the productivity gains are there. We don't know what they are because we actually don't know how to gauge developer productivity. But it is clear that there is something happening there. And you should be very open with your pocketbook there. And so again, we celebrate those who spend more and of course we talk to them and what did they do? So they were getting the do.
- [28:33]
Interviewer
I understand that even though you cannot measure it right now, or maybe not as accurately, you believe that there's productivity gains and you believe that people need to use it to get these. So you're kind of treating as an investment right now for a year, for two years, whatever that may be, and then probably forever.
- [28:48]
Farhan Tharoor
Probably forever. Like, I mean, again, if you, like, you wouldn't turn off spell check and grammar check and G Suite and like, you wouldn't turn off these tools, right? Yeah, I mean, there's probably a debate on when you turn off slack because it's like, that could be a productivity failure. But like, you are seeing the gains. And I don't think that we should be penny pinching now. Over time, we might realize there's an equilibrium and like, oh, this is the amount of tokens people should be used. Actually, if anybody's using cursor here, the number one thing I'll tell you is please move away from the default model because the default model is nowhere close to like actually using a sophisticated model. And by default, cursor puts you there. And so we tell people, please move off the small models, move into something more bigger. We'll even say things like, please use O1 Pro for this project, or please use O3O3 Pro, because those.
- [29:32]
Interviewer
They're more expensive.
- [29:33]
Farhan Tharoor
They're more expensive. I shall go even further. Mikhail, the CTO will say something like the following. If you, you don't Pay personally for O1 Pro or Gemini Ultra or Claude, whatever it's called, advanced the $200 a month one, you are crazy because you can afford it. And actually, you are missing out on actually all of the progress happening in LMS because you are just stuck to the $10 a month or $20 a month chat CBT. Like, that's how far he would go.
- [30:00]
Interviewer
One interesting thing that I've heard when I talk with engineers, they're saying leadership is leading by example, Toby is, is hacking. And if you go on his Twitter or social media, you're going to see all the, all the stuff that he's doing. Even a year ago. I want to ask you, like, what are you doing on the side or experimenting or playing with and what tools are you excited about?
- [30:20]
Farhan Tharoor
Yeah, so when I was building, when I was pairing with that engineer with on Anthropic, I was building something for myself, right? Yeah, I was building a couple of things.
- [30:27]
Interviewer
So one great way to use company time, I guess.
- [30:29]
Farhan Tharoor
Exactly. Well, I was like, let me see what they can build. Because we didn't want to go super deep into the Shopify API. But we do build a lot of things that, you know, one was actually a commerce flow. I was trying to build a commerce flow via like operator where you can actually record the browser session. And I wanted to see how the session was storing like credit card information or secure tokens as we were getting through a commerce flow. And I was trying to figure out, I was trying to build almost like subscription product without using our subscription product from Shopify.
- [30:58]
Interviewer
Yeah.
- [30:58]
Farhan Tharoor
And so it was an interesting experiment where we were trying to see how do we store those credentials, Especially when you want to store like credit card. Is it stored in the cloud already? Is it stored by the vendor? Are we storing it in the browser session? That's how I kind of stay as close to it as possible. I use cursor pretty often to build my own workflows. In this case it was Claude code. We actually even use Gum. I don't know if you've heard of Gumloop. Gumloop is an automation platform that is very interesting at its own layers. Here's one example we tried to build. I did something super dumb. I was like, let's figure out when the next SanDisk Extreme Pro 8 terabyte drive comes out. We tried to build in cloud code, took us an hour. We did not get there because it actually tried to do something at the wrong layer. It was trying to read the JSON and it was try to get the image from the SanDisk side and figure out if the 8 terabyte drive is out. It was actually quite complicated. We built it in two minutes in Gumloop. Not because they're different.
- [31:48]
Interviewer
What did Gumloop do?
- [31:49]
Farhan Tharoor
So Gumloop is basically, it was browser based. We said, go to this website, look at the, like, go through the search box, like using English commands. And they came back right away and said, there's no eight terabyte drive available. And I said, okay, cool. Run it every week and send me an email. It was just they're different tools for different jobs. And an hour with cloud. Again, nothing wrong with anthropic cloud code. I use it all the time. It was not meant to do this kind of like web scraping. Because what it is, what does it do? It tries to build a web scraper versus Gumloop. It already knows how to scrape websites. So it's actually just a different layer of the stack. And what I would encourage people to do is figure out where what you're trying to do. And. And we have a lot of automation happening in Gunloop now because people have to do something. I want to scrape a LinkedIn profile. I want to find out if the what platform this company is using behind the scenes. I want to scrape a website like all of these things are. It's built for the right thing at the right time.
- [32:35]
Interviewer
You have a lot bigger tool set.
- [32:36]
Farhan Tharoor
Exactly. So you should figure out where you want to work. Right. If you're writing code, plot code. If you're doing web skyscraper, use Gumloop.
- [32:44]
Interviewer
So far it was an AI AI. And when this letter came out externally, there has been a lot of some speculation saying, hmm, is this a sneaky way for Shopify to freeze headcount? Because it says you shouldn't hire unless you can check if you can do it with AI. But then you told me something interesting. I'm not sure how big Shopify is, but you told me that you're applying to hire a thousand interns. Can you put that in context with the size of Shopify? And tell me, why are you hiring interns where a lot of companies are saying, let's not hire entry level people because we have AI?
- [33:16]
Farhan Tharoor
Yeah.
- [33:17]
Interviewer
What are you seeing that others are not seeing?
- [33:18]
Farhan Tharoor
Yeah, so a couple of things. One is there is this real notion of like generations as they come up in different that graduate from different programs out of university or even coming from high school. And you really want to be close to the next generation of people. One, they're like, you know, they know things that you don't know. And so we restarted our intern program in full thrust this year. So last year we had like 25 interns a term. I convinced Tobi to hire a thousand interns this term. Mostly because I felt that they would. Yeah, I know, they would be more AI reflexive than everybody else. That was our hypothesis. So my tweet actually said, I want you to come to work with an LLM and a brain, not one or the other.
- [33:57]
Interviewer
Yeah.
- [33:57]
Farhan Tharoor
The idea was that bringing these like centaurs, AI centaurs, they know how to work with an LLM to solve their whatever workflow it is, whether it's a finance person or an engineering Internet. And we really focused on them coming in and changing our internal culture. And so 1,000 engineers over the year meant like 350 a term. And what we saw there was one, they're excited, they're hardworking. We also have them as a cohort, meaning they come to the office. We're a remote company, but we have the interns come in because we felt like the younger people need to have like younger people around them to work together versus just being remote in their like condo or whatever. We're changing the culture that way as well. And we did see this. They come, they have their AI reflexive kind of like we saw the same thing in mobile. You were in mobile as well, and so was I and my last decade of work. And I hired lots of interns because they grew up with mobile phones. And so this next generation is growing up with the Internet and they're growing up with phones and they're growing up with LLMs. So we wanted them to come in and change our culture around that axis.
- [34:50]
Interviewer
So do you feel like people are learning from we all, we always, we always were.
- [34:54]
Farhan Tharoor
Interns are the secret weapon. You always learn more from the interns. Like, people think about internships as like community service. Oh, that's so great, Shopify, you're helping the next generation. We don't do it for that. We do it to learn from the interns. That is always the reason. So if you don't have an intern program, an early talent program, I encourage you. You should be bringing those people in because you will learn from them more than you will get out of them. Like you will, you will take a net positive benefit from them. And of course, for those who like tweet at me and say, please don't hire interns, you have to pay them. I'm like, we pay them. I don't know why people think that you don't. We pay them like very well, actually. And we do hire. I think I've hired almost 100 from the, in the last, in the, this year from the intern pool to come into our program. And we're very much thinking about the only way to get into Shopify as an entry level engineer is through the intern program.
- [35:36]
Interviewer
I'm fully with you with, with on the paying. I never understood why companies are cheaping out on this because sometimes interns will reject some of the offers because it doesn't cover their living costs, which is.
- [35:46]
Farhan Tharoor
Why you have to pay them.
- [35:47]
Interviewer
Actually, one of the best intern I. It was another company's loss and in Amsterdam they offered €500 per month, no housing. And that intern would have loved to work there. What a experience. But they applied to Uber because they said it's just too low, like, and then we gave them a new grad salary. They could afford housing, they could afford.
- [36:03]
Farhan Tharoor
To travel, of course.
- [36:04]
Interviewer
And it was. That intern could have done amazing things at that company. And now this person is still senior at Uber, you know, been there for five years. So you know their loss.
- [36:12]
Farhan Tharoor
So think about it. 350 engineers interns to about 3,000 engineers. About 10% at any one point. Yeah.
- [36:19]
Interviewer
So if I joined Shopify as either a software engineer or an engineering manager, can you tell me how stuff gets done first as an engineer? Like, what tools do people use? I'm assuming it's just no longer Vs Code. Do code review. There's going to be AI tools here and there.
- [36:37]
Farhan Tharoor
Yeah, well, first on, we have a very specific way of writing software to the point of we built our own project management system. Like, we don't use jira, we don't use linear, we don't use. Yeah. And the reason is because we believe that there's that line like, first you make the tool and then the tool makes you. And so we believe that. And so we make the tool so that it can make us versus we. It's not nothing against those amazing tools. I'm a big fan. Like, linear is beautiful. And like, you know, I've known Jira for a long time. I'm a pivotal FRACO guy myself, being a pivotal. Ex. Pivotal person. But we do believe that we have a specific way of writing, of building software. And if we use one of those tools, we would be adopting someone else's way. And so instead we have a tool called gsd, which stands for Get Shit Done. And it is a tool that allows us. It's a program management tool, like a product management tool, which allows us to think about what are we building, who's on the team, what's the latest weekly update. It has metrics. It actually pulls in PR reviews. So you can see. It pulls in PRs and you can see there's activity happening on this project. Here are the core contributors. Here's how long it's been going on for this tool is how you kind of get work done and you're forced to write an update every week. And now actually we have an AI. Here's an interesting one for you. We have an AI tool which will pull in the latest prs and the latest conversations from the Slack channel and it'll write a update for you. And then you can look at it and say, this looks good. And you can also tell it, what did I not think about? And say, please emphasize this. And it'll help you write the project management update every week.
- [38:03]
Interviewer
So this is interesting. I want to push you a little bit on this because I think there's two schools of thoughts here. One says your weekly update. Like, I appreciate that, that you're doing it, the weekly update. The point should be for you to look back, correct on what you did. Obviously everyone hates doing this, by the way. And by AI doing it, would you not lose that kind of reflection?
- [38:22]
Farhan Tharoor
Well, this is why we pair you with the AI. So we do auto publish it if you don't do anything. But if you auto publish it and don't do anything and we look at the stats, there is a little bit of loss from you because you are now losing out on the context that you were supposed to gain as the champion of the project by looking deeply. However, we expect everyone, just like a PR written by AI to have read the code and read the update.
- [38:47]
Interviewer
Yeah. So again, you're responsible.
- [38:48]
Farhan Tharoor
You're responsible. And so we're early days in this. We're like literally three weeks in on this project and we may revert it and say, hey, by the way, it turns out everybody stopped looking at what was happening week to week. But we also, every six weeks at a company level with Toby, go through every project in the company and we see if we think that it's on track, not on track, should be kept working on has the right resources, is it on too long, is it aimed in the right direction? You better know what's in there because literally we're going to review it at the leadership level.
- [39:14]
Interviewer
Yeah, I think you have. No, I like how. Because I feel like with AI, I kind of want it to automate the stuff that I really don't want to do. And ironically, it's really good at automating stuff that we like to do sometimes, which is coding, but like writing an update or writing documentation. I don't think as engineers we loved doing that. And so as an engineering again, we're.
- [39:35]
Farhan Tharoor
Trying to reduce toil. If we find it reduces actual context, we will revert.
- [39:41]
Interviewer
I love the experimentation and going back and forth as an engineering manager, what additional things might people use? Do People build their own tools to stay on top of things. Do people go deeper because of.
- [39:53]
Farhan Tharoor
Yeah. As an engineering manager and a director, we try to surface again, metrics so that you can see how your team is doing. So for example, we have tools that allow you to see things like focus time of your team, how often are they in meetings, AI adoption, are they using like AI tools or not? Like, we're trying to surface this information. How many of your people are on a GSD project versus not, because maybe a project ended and they're like freed up to work on something else. So we try to surface that. And engineering managers too. We try to get them to be. A lot of the best ones come from IC land. Like they started as ICs, so we try to get them to do. If they're hired outside. I start as an IC at Jobify before they move into management.
- [40:29]
Interviewer
Oh, here. Here's a fun one that I think we should talk about. You mentioned something super interesting to me. When you're hiring engineering directors and above, in the past, it was the usual interview. You know, culture, fit, strategy, all that stuff. You added a coding interview for every single engineering director and above hire. Can you tell me about this?
- [40:46]
Farhan Tharoor
Yeah. So it's interesting. Like, it's maybe shocking to folks. And I know, like, Rodney sitting there, he used to work for me. I did the pairing interview with him. And so what happens is it is shocking for VPs especially to be like, whoa, there's a coding interview? I'm like, yeah, because we believe that the best leaders here were ones who were not running away from coding. They just felt like they got better leverage from running a team. They still are deeply in love with technology and they still. And Rodney's case still codes on the weekend. Right. And so we. It worked out super well. But our whole idea is that you're not running away, you're running towards technology. And this is just a better way for you to express it. So I pair with the candidates and they also see that even though I'm not writing code every day, I'm still deep in the weeds of technology. I still love technology and I still want to talk about technical topics. And so we pair. And some people believe that that's not the best for them. And they can. You know, there's lots of great companies out there where that's not the requirement. But at Shopify, we believe people should be as close to the details as possible. And we are such a tech nerd company. Like, again, our salespeople are vibe coding now. We Want our engineering leaders to be coding as well. And so that doesn't mean coding day to day, but you should understand code and how code works. And a lot of it comes back. The muscle memory of coding will come back in these pairing interviews.
- [41:51]
Interviewer
Yeah. And I guess especially now we have these coding tools when you, you know, you can generate something as long as you can look through it like you're in.
- [41:57]
Farhan Tharoor
The best part about copilots is that a candidate comes, the copilot generates tons of code. And now I'm like, great. Is that good code, Bad code? I love.
- [42:06]
Interviewer
Hold on. So you're using AI in your interview process?
- [42:10]
Farhan Tharoor
Yes.
- [42:10]
Interviewer
Oh, you're not running away from it?
- [42:12]
Farhan Tharoor
No.
- [42:12]
Interviewer
You know, one of the interesting stories that we just learned, Cursor has just banned it on the entry process. They're the AI tool company.
- [42:18]
Farhan Tharoor
Right.
- [42:19]
Interviewer
So you're embracing it.
- [42:20]
Farhan Tharoor
We're embracing it.
- [42:20]
Interviewer
Okay, how's it working? Tell me.
- [42:21]
Farhan Tharoor
I love it. Because what happens now is the AI will sometimes generate pure garbage. So.
- [42:25]
Interviewer
So you're screen sharing, and you say.
- [42:26]
Farhan Tharoor
Literally use that screen sharing. And they're using.
- [42:28]
Interviewer
I say, co pilot or whatever.
- [42:29]
Farhan Tharoor
Let them use whatever they want. Here's what I'll say. If they don't use a copilot, they usually get creamed by someone who does. So they will have no choice but use a copilot. Sometimes I will shadow an interview, and you do the questions myself. I've never seen them with a copilot. And send it to the interviewer and say, please mark my assignment as well against a candidate I have not locked yet. If they have not. If they don't have a copilot, they will lose. But when they do have a copilot, I love seeing the generated code because I want to ask them, what do you think? Is this good code? Is this not good code? Are there problems? And I've seen engineers, for example, when there's something very easy to fix, they won't fix it. They will try to prompt to fix it. And I see. Are you really an engineer? I get the nuance of just prompt and prompt and prompt, but sometimes it's right there, and they will not prompt. I'm like, change the one character and they won't change it. And I'm like, okay, so they're interesting. I don't want you to be 100% AI code. I want you to be like 90 or 95. I want you to be able to go in and look at the code and say, oh, yeah, there's a line that's wrong.
- [43:25]
Interviewer
So I want to ask you, what does a standout senior or staff engineer today look at Shopify and what has changed? Let's say three years ago when we didn't have any of these tools. I'm trying to prod here, like, how you think AI or AI tool usage might have changed of what we expect an amazing engineer and you think of the person who you think is like, this fantastic person.
- [43:46]
Farhan Tharoor
Well, I would say the thing that's changing is I'm seeing some of my engineers now actually use these AI tools to, like, do the infrastructure that they've always wanted to do but never felt like they had the time. So, like, debt reduction, refactoring, making things easier to read. Like, those are the things I'm seeing the best engineers at Shopify use AI tooling to actually do. And they just never felt like they had the time or the resources. Like, the engineer who always felt like, I wish I had, like six months to do a code red now can use an AI refactoring tool like a Claude code or a OpenAI Codex to like, unleash these agents in parallel to then and then review all the PRs, but then unleash it and have it and feel like they have a team themselves. So that's what I. That's where I'm seeing. And they're not afraid of. Of trying things that may or may not work. They might unleash something for 24 hours and like a Devon like tool and comes back and like, utter garbage. They deleted all, but they said it was worth trying.
- [44:35]
Interviewer
Yeah. And as a closing question, and a lot of people are thinking, we'd love to be like Shopify. Like, I'm either an engineer, I'm either a staff engineer, director of engineering, etc. I'd love to transform my culture to be a bit more AI. First, what would your advice be? How can people start? Like, not everyone has the luxury of having the CEO saying, okay, I love these things. Do you have tips that you give to your peers? I know you're in CTO groups, for example.
- [45:00]
Farhan Tharoor
Yeah. So I think that the number one thing is role modeling is the best example. Like, I haven't seen anything work better than role modeling. And the way that happens is, like, you have to do it. So if you are coding and you are showing people your workflow and you are showing you are in those same channels asking like, hey, I was trying to get this working and it wasn't working, or I was able to get this workflow going via this prompt, we Have a prompt library internally where you can literally grab prompts. Yeah. And like, say, like, you know, hey, this prompt worked for somebody. Let me try it and modify it for my use case. That's the number one way in which I see people trying to adopt AI is because they've seen other people do it. And so we try to share, like, AI use cases we have, again, prompt libraries. We had a hackathon like in our as a part of Shopify summit a few weeks ago, and we leaned heavily into AI. And it wasn't just junior people learning AI. It was senior people trying to embrace Claude code. And one guy said, I haven't coded in six, six weeks. I've only used cloud code. Then I pushed him and he said, okay, I have to make changes like here and there. I said, cool, you're 95% AI, but that's the point. You're not supposed to be 100. And role modeling is the number one thing. I've seen that. And sharing examples. So role modeling and then sharing, especially.
- [46:04]
Interviewer
Because no one has it figured out. So we. No one has learned.
- [46:06]
Farhan Tharoor
Exactly. No one. No one's. No one's there yet.
- [46:08]
Interviewer
Well, this is wonderful. I love how much you're experimenting. I love how you are not afraid to take some turn to learn more, and I love how much you're sharing internally. I think we got a little window of this, but I think it's just something I hope more people, more companies will do. So this was wonderful. Thank you so much.
- [46:26]
Farhan Tharoor
Thanks for having me.
- [46:28]
Podcast Host
I found it refreshing to see just how practical Shopify is about LLMs. It feels to me that they understand that you need to invest and experiment to get results. One surprising thing for me was understanding how Shopify expects to become more effective using AI tools by hiring interns who come in with an open mind towards AI tools and can come up with clever use cases for it. For more details on how Shopify operates, check out Deep Dives in the Pragmatic Engineer linked in the show notes below. If you've enjoyed this podcast, please do subscribe on your favorite podcast platform and on YouTube. This helps more people discover the podcast and a special thank you if you leave a rating. Thanks and see you in the next one.