← Lenny's Reads
Lenny's Reads
AI tools are overdelivering: results from our large-scale AI productivity survey
00:13:36
Loading summary
Transcript1 lines
- [00:02]
A
Welcome to Lenny's Reads, where I bring you audio versions of my newsletter about building product, driving growth and accelerating your career. I'm excited to share my fourth collaboration with the great Noam Siegel, the AI Insights Manager at Figma and former UXR leader at Zapier, Airbnb, Meta, Twitter, Intercom and Wealthfront. Let's get into it there's no shortage of debate about the impact of artificial intelligence on work. Is it delivering real productivity gains and where's the return on investment? But since data have been scarce, we took it upon ourselves to find out what's actually happening on the ground. We ran an independent and in depth survey on how AI is affecting productivity for tech workers. There were 1,750 respondents. The full appendix of who took this survey is included in the written version of this post, which is linked in the show Notes we surveyed product managers, engineers, designers, founders and others about how they're using AI at work. The TLDR is Artificial Intelligence is Over delivering here's the top seven insights from the survey. 1. 55% of respondents say AI has exceeded their expectations, and almost 70% say it's improved the quality of their work. 2. More than half of respondents said AI is saving them at least half a day per week on their most important tasks. We've never seen a tool deliver a productivity boost like this before. Three Founders are getting the most out of AI. Half report that AI saves them over six hours per week, dramatically higher than any other role. Close to half also feel that the quality of their work is much better thanks to artificial intelligence. Four designers are seeing the fewest benefits. Only 45% report a positive return on investment, compared with 78% of founders. And 31% report that AI has fallen below expectations, triple the rate among founders. 5. Engineers have accepted artificial intelligence as a coding partner and now wanted to handle the more boring but necessary work of building products, documentation, code review and writing tests. 6. N8N is currently dominating the agent landscape, though actual adoption of agentic platforms in 2025 has been slow and 7 a whopping 92.4% of respondents report at least one significant downside to using AI tools. There's definitely room for improvement. One thing's clear, AI is far from the novelty it was a year or two ago. It has cemented a place as work and productivity infrastructure and tools are improving at a breathtaking pace. If AI is already giving most people back at least half a day per week in late 2025, what does 2026 or 2027 look like we're watching the early innings of a compounding productivity revolution. Now, before moving on to the specific findings, one important note this audio version will focus on the insights. Visuals of the comprehensive results are included in the written version of this post. Okay, let's kick off the specific findings. Here's what artificial intelligence is doing for people function by functioning PMs are seeing the most value from AI tools to write PRDs 21.5%, create mockups to prototypes 19.8% and improve their communication across emails and presentations 18.5% Prototyping at number two signals one of many role boundary shifts happening now. With tools like Lovable, V0 and others, PMs are increasingly going from idea to prototype without waiting on design. But look Farther down the top 10 list and a pattern emerges. AI is helping product managers produce, but it lags in helping them think. The top jobs are all production tasks, docs, prototypes, and comms. Meanwhile, strategic and discovery work sits near the bottom. User research at 4.7%, roadmap ideas at 1.1% PMs have cracked how to use artificial intelligence for the last mile of getting ideas out of their head, but they still have a big opportunity to embrace it. For the upstream work of figuring out what to build, Designers are finding AI most helpful with user research synthesis 22.3%, content and copy 17.4% and design concepts ideation 16.5%. Visual design ranks number eight at just 3.3%. AI is helping designers with everything around design research, synthesis, copy ideation, but pushing pixels remains stubbornly human. Meanwhile, compare prototyping PMs have it at number two 19.8%, while designers have it at number four, 13.2%. AI is unlocking skills for PMs outside of their core work, at least in the case of prototyping. Whereas designers aren't seeing the marginal improvement benefits from AI doing their core work. Unlike others, founders are using AI to think, not just to produce. The top three jobs are all strategic. Founders lean heavily toward productivity and decision support 32.9%, product ideation 19.6% and vision slash strategy 19.1%. That's a stark contrast to PMs, whose top jobs are documents and prototypes, and designers research, synthesis and copy. Founders are treating AI as a thought partner and sounding board, not just a tool for specific deliverables. Productivity Decision support at 32.9% is unlike anything else in the survey. No other role has a single use case this dominant. This tracks with Tal's excellent post on building AI copilots as long term thinking partners and Amir's recent post on building your second brain using ChatGPT, both of which are linked in the show, notes There are two surprise misses for financial modeling at 1.8% despite founders living in spreadsheets during fundraising. Same with recruiting at 1.3%. Even though hiring consumes enormous founder time, these feel like opportunities Waiting for Better Tools this pattern may explain why founders report the highest satisfaction throughout the survey. They've figured out how to use AI for higher leverage Strategic work, not just production tasks. Engineers are the outlier. AI is doing just one big job for them, writing code 51%. This is different from PMs and designers, where AI is helping them with supporting work. Farther down the list are jobs like documentation 7.7%, testing 6.2% and code review 4.3%. These are the boring but necessary tasks engineers typically dislike, as you'll see in the upcoming opportunities data that's about to change. Engineers have accepted AI as a coding partner. Now they want it to handle the tedious work that comes after the code has been written. Here's one more pattern worth noting. Engineers report the most mixed results on quality later in the survey 51% better but 21% worse. The highest worse of any role. Next, let's look at tool preferences, function by function. ChatGPT is the number one most popular AI tool for most roles. 57.7% of PMs, 49.6% of designers, and 72.1% of founders use ChatGPT over any other AI tool, with Claude coming in second for those three roles. But engineers have a very different behavior. Engineers are the only role where ChatGPT isn't number one. GitHub Copilot was first to market, has Microsoft and GitHub's distribution muscle, and is baked into the world's most popular code repository. Yet it sits behind three tools that launched after it. Engineers are choosing newer, better alternatives over the incumbent. For engineers, the top three are in a dead heat. Cursor 33.2%, ChatGPT 30.8% and Claude Code are all within 4 percentage points. This market hasn't consolidated and switching costs are low. Another notable insight is that Claude code 29% outpaces Claude's chat interface 20.7%. Purpose built tools are winning, but Claude is also helpful with several core coding related tasks like code migration and more that put it at fourth. Gemini sits at a distant 10.6%, but this space shifts fast. A few strong model releases or product updates could reshape these rankings quickly. What's true today may look very different in six months. ChatGPT is a far and away winner for PMs. Perplexity is also surprisingly highly ranked, probably due to its strong research capabilities. However, farther down the list, lovable 8.7% and cursor are cracking the top seven for PMs. This reinforces the pattern we saw earlier. PMs are increasingly building things themselves, encroaching on what's traditionally design and engineering work. The PM toolkit is expanding beyond documents and decks. Copilot 8.4% edges out cursor 7.7% among PMs, but the reverse is true for engineers. This may reflect Microsoft ecosystem lock in at larger companies, or simply that PM's discovered copilot first and haven't yet explored alternatives. Now let's look at AI's impact. For most, it's driving significant time and quality gains. 63% of PMs and 83% of founders report that artificial intelligence saves over four hours per week. Even the most skeptical group Designers still shows 47.5% reporting over four hours saved. Only 1 to 5% of respondents across roles say AI is no faster than manual work. On quality, though the story is more nuanced. PMs and founders are bullish. Over 70% report quality improvements, but engineers are more mixed. 51% of engineers tell us that AI makes the quality of their work better, but 21% say it's worse. Designers fall in between at 60% better, 13% worse. The quality ratings among engineers may reflect the higher bar for correctness in code. A somewhat better first draft of a PRD is useful, but a somewhat better but buggy function is not. Also, bad code is easier to spot than a bad prd. The gap between where people are using AI today and where they want to use it next reveals a lot about where the opportunities are for founders and startups to jump in and deliver new tools and functions. For PMs, the biggest opportunity story is research. User research shows the largest demand gap of any task. Only 4.7% say it's their primary AI use case today, but nearly a third want it to be. The pattern is clear. PMs have figured out how to use AI for output tasks like writing PRDs and drafting communications, but they're hungry to apply it upstream to the messy work of understanding what to build. Prototyping is the breakout category for PMs. Creating mockups and prototypes jumps from 19.8% as currently using to 44.4% for what they want to use next, making it the single most wanted future use case for designers. Prototyping and interaction design show similar momentum. This tracks with the rise of tools like Lovable, V0replit and Figma make. People have seen what's possible and want more. Engineers want AI to handle the work after writing the code. Writing code is fairly saturated as a use case, 51% current, so it has a demand gap of only 5.6 percentage points. But documentation, code review and writing tests all have a demand gap of 23 to 25 percentage points. This shows massive opportunities for growth in engineering AI tooling Founders are doubling down on AI as a thinking partner. Product ideation shows massive demand, jumping from 19.6% as currently using to 48.6% for what they want to use next, a 29 percentage point gap. Growth strategy and go to market Planning and market analysis follow close behind with a demand gap of roughly 24 percentage points. Founders already use AI heavily for personal productivity 32.9% currently, but they want to move upstream. They're looking for a strategic collaborator to test ideas, explore markets and think through. Go to market. It's AI as a co founder, not just an assistant. Based on these reported gaps, the next wave of AI adoption will require not just better models, but better workflows for human AI collaboration on fuzzy problems. Writing a PRD has a clear output. Competitive research does not. Writing code can be tested. Product ideation cannot. Next, let's examine which AI tools have product market fit. We asked which AI tools would you be very disappointed to lose access to? The classic Sean Ellis product market fit question, 83.6% named at least one tool. This is a remarkable signal of how embedded AI has become in daily workflows. But the relationship between the number of people who regularly use a tool and would miss that tool if it went away and tells a story of the products that have truly found product market fit. ChatGPT dominates, but perhaps only for now. Half of respondents would be very disappointed to lose ChatGPT, but that's notably lower than the 60 to 75% of respondents across most roles who say they regularly use the tool. This in part explains why OpenAI recently declared a code red as it watches Gemini and Claude begin to erode market share. Switching costs in AI are still very low. ChatGPT Claude and Gemini top the list for PMs. They're such multipurpose tools well suited to the PM job. It's most interesting to see Cursor right behind Gemini. We wouldn't expect an engineering tool like Cursor to be so popular among PMs, followed by lovable, which currently seems to be winning in the prototyping market. Designers 23.3% and founders 20.6% Index highest on Claude the Claude ecosystem Claude and Claude code combined reaches 27.5% Overall, this feels like a big win for anthropic Specialized engineering tools have found loyal users and a clear product market fit among engineers. For engineers, the product market Fit leaderboard looks completely different from everyone else. ChatGPT 25.3%, Cursor 20.7%, Claude Code 17.1% and Claude 13.4% three of the top four products they'd miss are coding specific tools. Engineers have found and want to hold on to specialized tools that fit their needs rather than relying on general purpose chat interfaces. Cursor's 20.7% product market fit among engineers versus 7% to 9% for other roles shows how deeply it has embedded into coding workflows. In fact, a handful of role specific tools are winning their niches. This is the end of your free preview. To hear the full episode, become a paid subscriber@lennysnewsletter.com subscribe if you're already a premium member, you can add the private feed to your podcast app by going to add Lenny's Reads. Com. Thanks for listening and see you on the next show.