Building AI for Creators | Luma & Phota Labs

Summary

The a16z Show: "Building AI for Creators | Luma & Phota Labs"

Podcast Date: June 30, 2026
Guests:

• Matt Tancik (Head of Applied Research, Luma; co-creator of NeRF)
• Zach Xiao (Co-Founder & CTO, Phota Labs)
• Hosted by: Yoko Lee (a16z), with additional input from the podcast producer/host.

---

Episode Overview

This episode delves into the evolving relationship between AI and creative work. With AI's rapid progress—especially in generative imagery, video, and agentic systems—founders and researchers from Luma and Phota Labs explore how AI both empowers and challenges creators, personalizing tools while shifting what creativity means in the age of machine collaboration. Central to the discussion is the role of the artist versus the tool, the shifting boundaries of human versus AI-driven creativity, and the future of nuanced, agent-driven workflows.

---

Key Discussion Points & Insights

1. The Role of the Artist vs. the Tool

• AI as an enabler, not a replacement for creativity.
• Tools alone lack story or direction — human vision is still the spark.
• Matt Tancik: "Creativity is building a story. The tools alone aren't a story. Someone has to direct them." (00:00)
• Zach Xiao: "It's not about mastering those tools, it's about directing an agent who can use those tools to achieve your creativity." (00:05, 12:56)

2. Evolution of Creative Tools

• Historical context: Photography transformed painting, Photoshop transformed photography, now AI is transforming everything.
• The iterative cycle: Tools change, but the relationship between technology and artists remains symbiotic.
• Traditional creative software like Photoshop or Blender are complex; AI agents can adapt to users' needs and preferences, from simple to highly complex workflows.

3. Creativity, Iteration, and Personalization

• Iteration is key: Creative work is inherently iterative—for both humans and in AI-powered pipelines.
• Engineers/researchers often focus on pushing technology, but users care about practical, sometimes simple, workflows.
• Zach: "Users, they want you to solve their specific problem... but the technology needs to be ahead of users a little bit so you can reimagine their workflow." (07:55)
• Matt: Artists rarely have a complete end-goal in mind; tools must support exploration and iteration. (09:38)

4. AI in Photography: Past and Present

• Photography's creativity used to be concentrated in the "decisive moment" of capture due to technological limits.
• Now, AI extends creativity to the editing/post-capture phase—making photo creation more accessible and malleable.
• Zach: "A lot of photographers are having more fun post capturing than before." (05:42)

5. Research to Product: Bridging the Gap

• Tension between research (benchmarks, optimization) and building practical, delightful products.
• Research often pursues what’s novel/possible, while users seek solutions for routine, sometimes mundane, needs (e.g., removing backgrounds, fixing lighting).
• The best products find a balance: solving users’ needs while also pushing the frontier.

6. Unexpected User Behavior and Fat-Tail Use Cases

• Users often find creative, unforeseen workflows—sometimes 'hacking' tools for unintended applications (e.g., using a photo identity tool for product photography or video). (10:03, 23:10)
• Certain professional use cases (e.g., enforcing brand guidelines) emerge only after release, requiring tool adaptation. (22:22)

7. Controllability and Interfaces

• Text prompts alone are insufficient for nuanced creative control.
• Next-generation interfaces: Mix text, visual tools (canvas, scribbling, region pointing), and agentic layers to let users shape outcomes to their desires.
• "There has to be something more than just text... we've been doing a lot of work in the video-to-video space because that's a way to incorporate precise controls." (Matt, 26:14)
• Agents should adapt their abstraction level to the user, offering low-friction creativity for some and granular control for others.

8. Taste, Evaluation, and Personalization

• "Can you measure taste?" Subjectivity rules—so evaluation before and after shipping is needed, involving users closely.
• "Having user feedback in an ambient, non-intrusive way is important for personalization." (Zach, 19:11)
• Models must learn and adapt to both individual and population-level preferences and styles.

9. The Blank Canvas Problem and the Importance of Process

• AI tools are most productive when they support rapid iteration, not just single-shot outputs.
• The process remains gratifying for creators—even as AI does some of the "work."
• Many still pursue challenging, process-rich mediums (e.g., film photography) for enjoyment, not just results. (35:37)

10. Access, Demographics, and Latent Demand

• AI creative tools lower the barrier for users who lack the equipment, funds, or formal training for traditional photography and art.
• Examples: Generating headshots with AI for those who could not attend in-person shoots; recording meaningful moments with no expensive gear required. (37:59)

11. Old Tools vs. New Tools

• Legacy tools (Photoshop, etc.) may serve as infrastructure for a new generation of agents or be replaced by purpose-built solutions—the landscape is constantly shifting.
• "At the end of the day, all these things are just tools... For some, it'll always be Photoshop. For a newer generation, it won't be." (Matt, 40:23)
• Rapid iteration outpaces any fixed notion of "the tool of record." (Zach, 40:52)

12. Custom Models, Open Source, and Product Vision

• Having custom AI models allows tight alignment with the application's needs, supporting rapid improvements and specialization.
• Personalization is crucial—users should "own" their model, allowing foundation models to be swapped in or out as desired.
• Open sourcing offers benefits, but bespoke models enable better user experiences for specific applications. (41:38)

13. Who Will Excel in the Age of AI?

• The best artists will distinguish themselves not by tool mastery alone, but through vision, direction, and the ability to leverage tools in unexpected ways.
• AI raises the floor for everyone, but "the gap between the very best and the average artist is going to get even bigger."
• Even with the same AI tools, results reflect the creativity and intent of the user, not merely the tool’s capabilities. (43:07–44:49)
• "It's up to the humans to create something out of it." (Matt, 45:16)

---

Notable Quotes & Memorable Moments

• On Human-Centered Creativity:
  "The tools alone aren't a story. Someone has to direct them." – Matt Tancik (00:00)
• On Directing Tools/Agents:
  "It's not about mastering those tools, it's about directing an agent who can use those tools to achieve your creativity." – Zach Xiao (00:05, 12:56)
• On Personalization & Taste:
  "The agent needs to have really good memory. If I tell you once, shouldn't need to tell you again." – Zach (15:27)
• On Unexpected User Innovations:
  "People find crazy ways to get these things to work... for every reason it shouldn't work, but people find crazy ways." – Matt (11:07)
• On Evaluation:
  "Can you measure taste? ...depends on how coarse you get in that measurement." – Matt (18:55) "Beautification has long been a debate—the spectrum of it. Everyone has their own taste." – Zach (20:49)
• On Raising the Bar:
  "With AI tools, they're definitely raising the lower bar...but the gap between the very best artist and average artist is going to get even bigger." – Zach (43:48)
• On Creative Slop:
  "They assume AI to be slop. It's up to the humans to create something out of it." – Yoko/Matt (44:50–45:16)

---

Important Segments & Timestamps

• 00:00–03:13: Defining the relationship between human creativity and AI tools.
• 05:13–06:08: AI's impact on photography—shifting creativity to post-processing.
• 06:45–08:59: Transitioning from research to building practical products.
• 09:38–11:26: Learning from real-world user workflows and unexpected feature adoption.
• 12:02–13:47: Comparing complexity between classic tools and agentic systems.
• 14:41–15:57: Discussing interfaces and personalization for agents/humans.
• 18:39–20:49: Taste, subjectivity, and the challenge of evaluating creative outputs.
• 24:52–28:02: Trends in controllability and cooperation between user, workflow, and model (user-agent collaboration).
• 31:37–33:59: Representation choices in AI models, the value of artistic "intuition", and the blank canvas problem.
• 35:37–37:59: Photography: Enjoyment of the process vs. focus on results, and democratizing access for new users.
• 40:02–41:25: The evolving role of traditional tools—will they become infrastructure or be replaced?
• 43:07–44:49: What will distinguish the best artists in an AI-driven future.

---

Tone and Takeaways

The conversation is candid, reflective, and practical—rooted in deep industry and research perspective but focused on the lived experience of creators. Both guests emphasize humility, adaptability, and the continual blurring of the line between technical and artistic talent. The episode concludes that AI won’t replace human creativity—it will amplify, reshape, and differentiate it.

Key themes:

• AI democratizes creative power—lowering barriers while expanding what’s possible.
• The most transformative creators will be those who remain curious, adaptive, and iterative—directing AI as a partner, not a replacement.
• There is no single "correct" interface or representation—personalization, flexibility, and human feedback remain paramount.

---

End of Summary

Transcript

[Matt Tancik]

I think the creativity is building a story. The tools alone aren't a story. Someone has to direct them.

[Zach Xiao]

It's not about mastering those tools, it's about directing an agent who can use those tools to achieve your creativity. Generated AI have become so good. You can be sort of authentic to that moment while getting a little bit creative of stuff. So I just think a lot of photographers are having more fun pose capturing than before.

[Matt Tancik]

How do you make something unique with the tool tools you have access to now? There has to be something more than just text. If you go to say a studio and you say make me a 10 second video about a dog jumping in the grass, they're never going to take that deal. They're going to want more specific. And I think AI tools are no different.

[Yoko Lee]

Something I heard from someone today earlier was they assume AI to be slop. It's up to the humans to create something out of it. What are your thoughts?

[Zach Xiao]

I feel like that's such a hard question.

[Podcast Host]

Creative tools have always changed the way people make art. Photography changed painting, digital software changed photography. And now AI is changing how images, videos and entire creative workflows come together. The technology is improving at an incredible pace. But the most important ingredient in creative work may not be the model, it may still be the person behind it. In this episode, Yoko Lee speaks with Matt Tancik of Luma and Zach Xia of Photo Labs about AI generated imagery, creative workflows, personalization and why they believe the future belongs not to the tools them, but to the people directing them.

[Yoko Lee]

Today we have Zach Xiao who is the co founder and CTO of Photo Labs. He works on personalized AI and AI photography. We also have Man Tanzik who is the head of applied research at luma. He works on agentic systems, fundamental research and he was also the co creator of Nerf. Thanks for coming on the pod. So maybe let me start with this question, Matt. Many years ago, maybe three years ago, you had this online talk about Nerf and how you started this online talk was you posed a really interesting question which is what is the role of the artist and what is the role of technology? Now fast forward three years, which is a lifetime in AI. What is your answer then and what's your answer now? Has it changed?

[Matt Tancik]

Yeah. So if I recall correctly, my answer at that point was that really the point of this AI technology is to act as a tool for artists to better execute on the creative vision that they have. And at the time Nerf was providing an ability for people to create 3D assets. In a way that didn't require tons and tons of manual effort. Because when you're creating 3D scenes, creating those assets isn't really where the creativity part is. It's how do you put them all together? How do you build a story around it? And that's where those tools were helpful.

[Yoko Lee]

Right.

[Matt Tancik]

I think now there's just so many more tools that just allow you to do that, not just in 3D, but also in video, in images. And I think this. Yeah. Thesis that these AI technologies can be used as tools is even more so than it was in the past. Yeah.

[Yoko Lee]

Interesting. And Zach, I know you work at Adobe and then you did a lot of research that later, you know, went into Lightroom, which is also a creator tool us creators have been using. What's your view of this?

[Zach Xiao]

Yeah, I think artists really are where the creative minds are. They have a lot of creativity and technology is sort of a way to help them express that kind of creativity. I think maybe in the past few years, people like artists were also spending a lot of time trying to mastering the workflows, the tools. I don't think that's necessarily where their creativities are. So now their tools have become easier to use, more expressive than before. It's just so much easier for them to express their creativity. But I still think technology cannot replace the creativity minds Tibet.

[Yoko Lee]

Interesting. What would you define as creativity here? Let's what the models are creating, not creativity. And is what user kind of composing together. That's what we call creativity.

[Matt Tancik]

I think the creativity is building a story. And there's so many different ways to build a story and you use tools to help construct that. And the tools alone aren't a story.

[Yoko Lee]

Right.

[Matt Tancik]

Someone has to direct them.

[Yoko Lee]

Yeah.

[Zach Xiao]

I really think that this directive mind is just so important. Like tools, everybody can use their tools, but the stuff that gets created from these tools are very different. And the difference of the things that are created, that's where the creativities are. People have different thoughts, different stories, or even just a simple thing they create, which doesn't necessarily have a very strong story behind it, but they use a tool differently than others to create that thing. I think that's where creativity is.

[Yoko Lee]

Yeah. So Matt and I, we had this discussion earlier. So Matt is also a photographer and painter in his spare time. So he takes a lot of pictures. It makes me wonder what is like AI creativity when it comes to photography nowadays. Right. There's one school of thought where whatever you capture in the real world is the end result. And I don't want to go change the pixels. There's the other view, which is like camera is just one way to produce the reality. Other ways it might be painting like before the cameras came around. It might be Nerf. If you want something 3D and then go scan all the objects, it might be a model. So what is your view here and how has the view changed?

[Zach Xiao]

Yeah, I can maybe say a little bit about photography. I feel like photography has always been a very creative process. I'd say a lot of the creativity on photography was more on the capturing side. Before this AI boom that people were just trying to find the important moments, find the good landscape views and find the right angle. Composition is the right tool, wait for a very long time to take that photo. That's where the creativity that photographers are

[Matt Tancik]

really proud of that they wait and

[Yoko Lee]

wait for that moment to.

[Zach Xiao]

Yeah, exactly, exactly. But I think the reason the most creative was a capturing time is because the decisive moment, once that happened afterwards, there's not so much you can do to it. I feel like that part has changed a lot in the past three years. Genetically di have become so good. It can be sort of authentic to that moment while getting a little bit creative of stuff. So I just think a lot of photographers are having more fun post capturing than before.

[Yoko Lee]

Interesting. Yeah, super interesting. So I also know both of you start off your career doing PhD and doing a lot of very in depth research. And then since you went into an industry, there's a lot of user facing work and building the actual product, shipping the product, learning from the users. From our perspective, it seems like a research community. They care more about the benchmarking, optimization part of it. And then there's art of building the product. How has it been like for both of you when it comes to transitioning from research to a product builder? What did you have to learn and where do you think the gap is? Maybe we can start with that.

[Matt Tancik]

As a researcher, your goal is always to push the technology. And so wherever there's the hole in the technology, you push it. That doesn't necessarily align with what say a customer or end user would actually be interested in. And so quite often you see these things diverge. For example, in image generation, a artist probably doesn't care about generating paragraphs long of really dense text in some esoteric font. Maybe some do, but most don't. But as a research question, that's extremely interesting. So you do have to decide do you want to keep pushing the research in certain directions and when do you take a step back and push the creative side because in many cases what an artist or creative would want is actually very simplistic and the tools already exist out there and it's just a matter of piecing together in the right way.

[Yoko Lee]

Yeah. Like removing background itself is a boring task for most researchers, I guess.

[Matt Tancik]

Yeah, yeah.

[Yoko Lee]

But it's extremely useful for any creators

[Matt Tancik]

out there or even just fixing up lighting of an image. It's something that also doesn't have a very concrete solution. So it's hard for say a researcher to optimize a metric against, but when it comes to a creative task, it's extremely useful.

[Yoko Lee]

Yeah. And what about photography? I guess, like, curious, what's your learning there?

[Zach Xiao]

Yeah, yeah. What I've learned, I'm still learning is that this is such a find balance. Like researchers, they get excited about new technology so they just as Matt said, they just want to push it forward. Users, they want you to solve their specific problem they just ran into. Now there's some balance that we need to do there. We can't just solve the day to day problem the user are creating. We also want to be, the technology needs to be ahead of users a little bit so that maybe you can reimagine their workflow. Right. But at the same time you want to make it balanced. You're still solving their problems. You're not just producing a product that no one is going to use. So that's such a fine art. With photography, there's something like that. I think at the beginning of this AI era, we get a lot of pushback photographers. I just want to capture, I don't want to edit. Now they're slowly getting into, oh, it's fine to remove some distracting object in that image. It doesn't matter to me. And that's still my creativity part. So people are transitioning as well. Technology needs to stay a little bit ahead of that spectrum.

[Yoko Lee]

That's such a great point. It is a chicken and egg problem. You can't implement a hundred percent of what users ask. A seasoned Photoshop user may ask you to reinvent a lasso tool.

[Zach Xiao]

Yeah, exactly.

[Yoko Lee]

And the question is, do you want to do that? Is that the mental model you want your model to have? Like maybe the next abstraction is there's no lasso tool. Like we're not implementing the world in the old Photoshop world. Yes.

[Zach Xiao]

So.

[Yoko Lee]

So I guess like when you work closely with the creators out there and as you're learning from them, what is the surprising thing about their workflow? Anything that you push your product out There, but wasn't designed for, but had a lot of adoption that you were pleasantly surprised by.

[Matt Tancik]

Yeah, I think one of the first things that comes to mind is that you tend to think that like an artist or creative has a end goal in mind, like they know exactly in their head what they want, but in reality that's not true. Part of art is that iteration and so making sure the tools are able to handle that level of iteration ends up being extremely important. And this also makes it very hard to like benchmark these things because their optin isn't that ground shrinking in their head.

[Zach Xiao]

Yeah, for us it's a little bit specific to photo. So like we do a lot of identity preservation, identity consistency. We set out to do it. We're more targeting at image editing real photos that people want to preserve identity in the image. What surprised me is that people are actually very willing to generate a new image, put them in different environments, generating AI headshot, generating AI video of themselves using FODAS technology. So that kind of surprised me a little bit. And people are getting very creative with that.

[Yoko Lee]

So yeah, the video use case is actually a surprising one. Do you want to explain it further?

[Zach Xiao]

Yes, yes, yes. So we start right now. Our technology mostly just work on images, but people are getting creative. People are like, oh, the video models are so good, so bad at identity preservation. What I can do is I'm going to use folder to generate frames of those videos and then I'm going to use a video model to take those frames into a video so that identity is good. Something that we didn't think of at the beginning, but they figured out. So creators. Yes.

[Yoko Lee]

Yeah, interesting.

[Matt Tancik]

I think one thing to chime in on that, I've seen it multiple times now where people find very interesting ways to use tools. Not just our tools, but our tools in addition to other tools that you never expect it to work and for every reason it shouldn't work. But people find crazy ways to get these things to work.

[Yoko Lee]

Do you have an example?

[Matt Tancik]

There's so many. I would have to pick a little bit more.

[Yoko Lee]

Yeah, I'll come back to prompt you. So I guess when it comes to the tool abstraction, what kind of tool? Like I almost see the end user as type of agents who need to use tools. It's just you provide the tools and sometimes they will use the agent, like their AI agent to kind of leverage the tool cost as well. So there's a tools for humans and the tools for agents. How do you see those differ in your world?

[Matt Tancik]

Yeah, I think it's interesting to look at how creative tools have existed traditionally. You look at your Photoshops, your illustrators, you look at your blenders. Whatever modality you're interested in, if you hop into those programs, they're extremely complex. Right. You can literally have a whole career learning the ins and outs of these programs. And so that's great for people who've had that time and that experience to invest in it. Where agents come along and the ability to, like, have agents work with tools is then the ability to tune the tool to the user. Yeah, some users, all they care about is, you know, relighting an image, removing a background. Some want extreme levels of control. So then the question is, how do you interact with an agent or whatever AI tool that you're working with that can choose that level of abstraction that makes sense for you?

[Zach Xiao]

Yeah, I feel like, as we said at the beginning, creativity is about, like, directing something to achieve your creativity, your creative mind. I think this directing thing is so important. I think that's where sort of agent comes into place. Like, it's not about mastering those tools, it's about directing an agent who can use those tools to achieve your creativity. That's just so important. Another thing that I feel like is some lesson that we learn in photography is that, you know, a lot of people edit photos. A lot of people just don't want to edit photos at all. They have creativity in mind. They're just. They don't want to do the work or they're too busy to do that, but they still have a creative mind. So if you have some kind of agent that understands your creativity and can express your creativity without, like, very low effort from you to help you achieve that, I feel like that's even better. Like this kind of automatic or passive

[Matt Tancik]

agents that run in the background.

[Yoko Lee]

I guess just on this point, my observation so far has been, at first, we have simplistic tools where users can go in and prompt and they get an image and that's it, one shot. And later we start to get crazy notes on, like, concrete workflow, and it's like the. You can't even, like, understand it as a human. And then we start to have Canvas and more. You know, it's not, you know, like a graph that you can easily map out. It's more like a set of things you can parallelize once in a while. So how do you think about the next interface for both, like, agents and humans? Like, would the agents be on the campus using Adobe Work Photoshop and humans just don't need to Learn how to use it anymore and just ask the agent how to do it. Or would it be a different interface altogether?

[Matt Tancik]

I think it's false to assume that every user is going to want the same thing. Exactly. You look at directors of movies and some directors go super deep into the weeds of making sure that every glass is in the right place. Whereas other directors, they don't care about any of that. All they care about is like, oh, is the actor doing the line in the right way? And I think all that is true in these sorts of creative tools too. And so you have to create an agent that can work with these different directors, the one that has that ability and that control to go and manipulate these things at very fine grain levels. Or the ones that maybe are more on the like, I don't know, the literary side or whatever it might be, and just wants to talk and like interact and have fun with the tool and see what comes out of it.

[Zach Xiao]

Yeah, personalization is absolutely key here. Different people have different levels of acceptance of like complexity of tools. And they also have different kind of preference and taste. So yeah, it has to be personalized to them. And I think Matt is speaking about like the agent needs to understand the user really well, be flexible about that. I also think the agent needs to have really good memory. So like, if I tell you once, shouldn't need to tell you again in the future. So like, I think that's also just really important.

[Yoko Lee]

Yeah, I guess just to push on that point a little bit. Obviously if you travel down the abstraction layers far down enough, you can build anything and everything with C and compiler. But it would be weird to assume that agents are going to rebuild your creative tools for yourself every time. So there has to be somewhere in the layer you take, like draw a line in the sand and decide that this is the layer we're gonna operate on. How do you think about what that layer is for each of you?

[Matt Tancik]

I don't know if I have a concrete answer because I think it's constantly changing. There's a lot of arguments for different sort of underlying representations. And some make sense in some cases, others make sense than others. Everything from, you know, is it a 3D world that you're trying to model? Is it images that you're trying to model? Are you trying to break up the images into layers? And I'm not sure we really hit on as an industry what the right representation is. Yeah, yeah.

[Zach Xiao]

And I think there's an abstraction of what's a model with a tool. Right. Like a tool can Be a little bit more than a model. A model can do a lot of things. You need a lot of tools, traditional tools to do. So I think that we're going to see how that sort of roll out. I personally think model is replacing a lot of the old tools for not so precise control. But I feel like also at the same time people are putting more precise control back to the model. So it's a back and forth process.

[Yoko Lee]

Both of you have been doing a lot of model and app co design and then serving your own in house trained model on the app you also built. The question is how do you think about what good looks like when it comes to the generation? Because there are so many different ways to judge that. And is it a software is the model or is it something else?

[Matt Tancik]

Yeah, being a researcher is always a focus of how do we convert good into a metric that you can measure? Because then the agents and the research teams can optimize against it. However, there's no way that that'll ever tell the full story. And so you do have to involve humans in the loop during development. It's how do you bring in creatives and people whose tastes that you're trying to mimic can come in and help advise in that way. But that also isn't, can't be the full story because one person creating one video is going to have a different vision than someone else. And so this is where the personalization really becomes key. And the question is, how do you learn those, what the user cares about, maybe as they're using the app, maybe as they've provided additional context as to the type of stuff they're building or based on the history of the stuff they've built in the past.

[Yoko Lee]

Can you measure taste? I guess when you have the tastemakers come using the tool, they'll have different workplace. Of course you can, you know, optimize for every one of them. But is there some way to measure what taste is for a particular tool?

[Matt Tancik]

I think you can measure taste given some constraint, like given a population, you can maybe measure the taste of that population, but then you can also measure the taste of an individual user. And so depending on how coarse you get in that measurement, it just becomes a bit more muddled.

[Yoko Lee]

I see.

[Zach Xiao]

Yeah, I think for us, evaluation, there's evaluation before you ship something and there is evaluation after you ship something. So when evaluation before you ship something, a lot of times for us with specific work on personalization. So really care about having each individual tester or user to say are they happy with the result it's very hard for us to come up with like a universal thing. Especially for very subjective things like taste preference and even for very personal thing like identity. We always ask the person to evaluate their own identity or the person to evaluate very close ones in their life without any of them. And then there's evaluation after you ship the product. I think especially for very subjective things. That's so important for personalization because the personalization technology is going to be applied to different users differently. Have we made the user happy? Are the user satisfied with the result they're getting? And can you collect that kind of feedback in an ambient way? That's just like non intrusive way is just so important.

[Yoko Lee]

This is such an interesting point. I was actually talking to someone earlier about their photo result. His view was happiness and truth seeking are two different things. When he saw the result and he was like, this is exactly me. It's like the best model output, the likeness is 100% violet too fat in the. So he's not happy with the result. So there is some kind of like, you know, filtering like something that makes the user happy. May not be the benchmark thing that, you know, like you might be straight. How are you guys solving that?

[Zach Xiao]

Well, like at the end of the day we think, you know, user is the one who is controlling things. You know, if they want them to look in a certain way in the, in the photo, I think that's, that's their choice and we need to empower them to do that. And again this is a very subjective thing. Beautification has been a long, long time debate of the like the spectrum of it. Everyone has their own taste. Yeah, personalization is key here.

[Yoko Lee]

You know, I imagine personalization is more of a skilled problem compared to having empower everyone, everything to do things on the canvas. How do you think about evaluation or income too? There are so many unlimited workflows one can do on the canvas.

[Matt Tancik]

Yeah, I think somewhat similar. You have to look at how people ultimately use the tool. We can come up with as many guesses on how to evaluate before we release it. But at the end of the day you have to see once it's out there are people using it, where are people using it, where are people getting frustrated, where are they feeling like the level of control is sufficient and where is it not sufficient and trying to gather as many of the signals as possible across the different domains that people use the tool.

[Yoko Lee]

Interesting. So this, this totally makes sense. I think of a very, I guess like a broad tool as there's A distribution of use cases when you launch the tool, there's obvious ones that you probably would have tested before launching and there may or may not be a fat tail on the distribution. The question is, are there specific use cases you want to move from the fat tail to the head? Have you ever run into use cases like that?

[Matt Tancik]

Yeah, I think one interesting one was seeing how brands would bring in same brand guidelines. It's not something we really considered very much. But then once they brought them in, you realized this was an extremely useful source of information for really getting to what a user is interested in. And so there's some questions there of not just how do you enable that pipeline for these, say brands or other companies to include that information?

[Yoko Lee]

Yeah.

[Matt Tancik]

But how do you also get that type of information from others who aren't used to working with those sorts of pipelines?

[Yoko Lee]

Yeah, interesting.

[Zach Xiao]

Yeah. For us there was some users who are kind of our current API only supports human and pets for identity. There's some users who are trying to hack our system so they can use it for personalization for product photography.

[Yoko Lee]

Oh, interesting.

[Zach Xiao]

And they're like, we want this. And so now that's on our roadmap. You know, we want, I mean it makes sense. But so that's kind of an example where like we didn't see that kind of use case. We first launch it and there's market pool and then we just want to increase the priority on those. Yeah.

[Yoko Lee]

How do you see the difference between, I guess human likeness and product likeness? I mean our brain is involved to recognize human faces. So I imagine there's lower margin for error. But what about product? What's the learning there?

[Zach Xiao]

Yeah, I mean, so first of all there are a lot of similarities between these two. First is like everybody say, oh, the general foundation model is already very good at identity until you try to use it to generate image of yourself. Same thing with product photography. Oh, it's pretty good at generating this specific product until you are the product owner and you're trying to generate a photo of that particular product just stops working. So that's a non obvious one. The difference is like identity is a little bit more, I would say, subjective and dynamic. It's really, really hard to put into words what is identity like product. On the other hand, you need to get the text right, you get the shape of the product right, get all of those right. So it's a little bit different there. And there are unique challenges in both. For example, text rendering is so important in product photography, not so much in identity preservation.

[Yoko Lee]

I imagine the question becomes, do you want the model to do it all, or do you want the subsequent workflow to fix some of the things that your new model couldn't do? Just curious about both of your opinions on what kind of task, when you run into it, you feel like it's something the model should solve and what kind of tasks you're okay with having a workflow solving it.

[Matt Tancik]

We've seen a trend in language models where they've transitioned to thinking modes. Right. If you use ChatGPT or one of these models nowadays without thinking, it just doesn't feel smart. And I view that as kind of this in between of. It's like a workflow, but it's still using the core model. And I think creative tools can go in that direction too, where you generate stuff, but you can evaluate it, you can build off of it. Maybe the users involved in that, maybe an agent's involved in that, but it helps you get to the final result. Got it? Yeah.

[Zach Xiao]

I think there's a separation of model versus technology like you in terms of personalization, it doesn't matter. It's like a general model that solves everyone's problem as long as the technology is the same. We can apply the same technology to different users. But again, you want to make the tool that or the model so that you can unlock the creativity in people so they can figure out the best workflows to solve a particular problem that you haven't even thought about before.

[Yoko Lee]

Yeah. How do you see controllability? I mean, phase one of AI is we generate things and we just keep reprompting the model for the thing we want. It's almost like a lottery. Phase two is that we put more control either on the model level or on the systems level to make that work. What does it mean for AI created the creativity tool to have better controllability?

[Matt Tancik]

Yeah, I think it's an open question and there's different ways you can add control, but the short answer is you definitely need to add control. I think that was the number one complaint from anyone who's actually trying to use these tools for any sort of professional or to replace any part of their existing creative workflow, is that you don't want to operate with text. There has to be something more than just text. Or if it is text, you need that level of control, which can be very difficult in text. And so we've experimented a lot. Most recently, we've been doing a lot of work in the video to video space because we think that's like A very good way to start incorporating pretty precise controls, especially in the time dimension, in the spatial dimension. I think there's a lot of interesting things with being able to scribble, being able to point to certain regions and so adding more and more of these controls on top of the model so that the user can do. While also opening up these controls for an agent so that you can instruct an agent to do these sorts of edits.

[Zach Xiao]

Yeah, I think it's also about helping the user to say what they want. I think a lot of the time the model doesn't do exactly what you want. It's because of like the user's prompt or whatever is a little bit ambiguous. And it's also so other than the. You need to tell the model what to do. I feel like this is the under exploited research direction is if the model can tell you what it needs to get your inputs to, asking for your inputs before it does something. I feel like that's also really important because like when you, when you are working with a creative professional as an amateur, a lot of the times they ask you questions and you can, you can tell them your opinions, your preferences and they sort of achieve their preferences and opinions. I feel like that mostly today is almost a one way street. It's like the user tell the model what to do.

[Matt Tancik]

I think that's a really good point. If you go to say a studio and you say make me a 10 second video about a dog jumping in the grass, they're never going to take that deal. They really going to want more specific. Yeah, you have to figure out how to apply that in the AI.

[Yoko Lee]

Yeah, interesting. Do you think model app code design has benefits when it comes to introducing more controllability? What's your view there?

[Matt Tancik]

I think so. I think being able to have the users interact in your app and work through their workflow in your app you learn which things a user actually cares about. You learn which steps do they have to redo multiple times and why. Maybe they chose the step that they ended up choosing. And that can help a lot with model design.

[Yoko Lee]

Right?

[Matt Tancik]

Yeah.

[Zach Xiao]

I think the other direction is also really important is like your product can also help educate the user. What's the best way to sort of use their model. I feel like we're seeing that a lot of that was like coding and stuff. Right. Like you know, people are used to like to do everything like VS code or whatever and now like cloud code just in the terminal you can, you can do that code and it's the best way is to sort of use that model because you cut through a lot of other things. I so feel like app and model co design has that advantage too.

[Yoko Lee]

Yeah. When we think about AI creative tools, the first thing that comes to mind, at least for me, is it's very visual. Just being very visual animals. And now we're used to working with agents. It feels like almost like agents are aliens who don't perceive the world the same way that we did or we do. So I'm curious, is there a world where creative tools become primarily headless? What would that look like if you want to leverage agent tech moves?

[Matt Tancik]

Yeah, I think again going back to like if you were to work with a studio, you are talking to an agent, right? You're talking to someone who is working with you, but they're not talking back to you in text, they're showing you examples, they're walking through, hey, do you like these ideas? Do you like these ideas? And I think that's the direction you'll see these AI creative tools go. Yeah, yeah.

[Zach Xiao]

Like today when I try to brainstorm something and you know, you try to write a code to generate HTML page showing me the graphics of it, I want just a sketch out of it. That's probably the best way to iterate. Sort of an image is more than a thousand words. So that's just so important. On the other hand, I feel like people haven't figured out a way what's the best way to feed the visual information to a model. I still feel like models today is a little bit blind. I mean it's getting better, like having high level semantic understanding. But visual is just such a, I would say redundant information. There's like pixels that are very redundant, so unlike languages. So how do you sort of give the model that kind of information? So the model is not blind anymore. It's so important. I don't think we have figured out that yet.

[Yoko Lee]

That is so interesting. It does remind me of like the importance of having the right representation in the creative tool. I mean Matt, you worked on Nerf and before that people were on meshes and now there's a spectrum on how efficient a representation is on Nerf. And Gaussian slide is great in details, but it's really hard to work with because it's not like a beautiful mathematic function. How do you think about representation or different kinds of representations for the feature iteration of the design tools you want to make?

[Matt Tancik]

I think the representation question ultimately comes down to the control question too. It's what do you one to be able to modify it in the future. If you're designing a poster and you know you're going to be changing the text on it, you probably want a representation that's not pixel level or changing that text. If you're just modifying pictures, then pixels make sense. If you're navigating 3D scenes, then for say, movies, you may want a 3D representation. So I don't think there's necessarily one answer. It's whatever makes sense with a given level of control.

[Zach Xiao]

I think that's so important. I think right now a lot of people are more focusing on what's the best reputation for the model. But as Matt said, I think the consumer of those representation is not just the model, it's also the human. So, like, can you represent in a way that human can understand and then be able to control that representation, do stuff to it?

[Matt Tancik]

Yeah.

[Yoko Lee]

One thing I am curious about for both of you is how do you develop the intuition for building for the creators as former researchers? But Matt, I know you are also an artist in your own right. So like, how does that affect what kind of research or what kind of product you should?

[Matt Tancik]

I think it sort of gives you some insight in the control aspect. Like, one of the first things I remember being drilled in during sort of art classes when I was younger is the blank canvas problem. There's a tendency of wanting to, well, getting stuck by that, but then once starting very meticulously, like doing one section at a time. But in reality it's better to just sort of go at it, just try things out, mess with it until you get to the thing that you're ultimately happy with. And I think AI tools are no different. The iteration cycle is quite fast, but you really want to iterate. It's not just put in a single prompt, get a final answer out. You want to iterate on top of that. So that definitely impacts how to think about building the tools. And then I think another aspect is figuring out what language to describe what works or doesn't work in the tool. Because this can be very important for evaluation. Where, I mean, throughout art, over the past however many years, people have built out these ways to describe composition, lighting, et cetera. So having that knowledge and incorporating it into the design of the models, I think can be very helpful.

[Zach Xiao]

Yeah, talking to users is really important. We talk to our users, we feel like, oh, we definitely recognize the blank canvas problem. Like when you ask the user, how do you want to improve this photo? They don't know when you make some edits to that photo now they have opinions on whether they like your edits or not.

[Yoko Lee]

Is that what they don't like?

[Zach Xiao]

They know what they don't like. Exactly. So talking to users is really, really important. And I also think having a very good understanding of the technology so that you can make this fine balance of, you know, staying ahead of a little bit ahead of the users. I think it's also really important to build that accounting intuition, not just from the user perspective, but also from the technology perspective.

[Yoko Lee]

How do you see the process versus end results? Just because in R, if your human is creating art, a lot of it goes into the process of creating it, and then final result is when you decide the process come to a point of your satisfaction. How do you compare that with how agents create art?

[Matt Tancik]

I think in many ways it can be quite similar. And I think even when agents create art, there's still the human involved in that process because that process is the part that's on for artists. You look at breakdowns, VFX breakdowns, and you see how many different steps and layers went into the final pixels. And it's fun. And I think people who do that kind of work find it very exciting to kind of see how everything breaks down. I think the way things break down in the future with AI tools will look very different, but it'll still exist. And so it'll be this kind of back and forth between agents doing some parts of the work, but then humans also ultimately tying it all together.

[Zach Xiao]

Yeah. And this may not directly answer your question, but there's always this debate of like, okay, we're working on AI photography. Is photography going to die? Regular photography is going to die in the future? I don't think so. There are a lot of people who are doing photography not just because they enjoy the result, they enjoy the process so much.

[Matt Tancik]

Right.

[Zach Xiao]

Like, even digital cameras have become so good nowadays. A lot of people are still taking films. Film is such an interesting process. Right. Like it has this unique thing of you have to capture that moment, otherwise it's gone and you don't see the result right away. You have developed the film, so a lot of it is interesting. So I think people are still enjoying those processes, but at the same time there are people who just want the results. And, you know, how do you empower them with the AI creative tools to make that easy for them to unlock their creativity? I think that's also where I'm excited about the AI tooling for creative.

[Yoko Lee]

There's definitely a lot of latent demand. So the people who did not have the money to buy a very fancy camera, can go out there and take pictures. And they also never took a film developing class ever. Like they found their medium. Like here's how we see the history too. Like the traditional, you know, creative tools, they were all there but new entrants, you know, they come on the market, they targeted users who were not users of the traditional tool. They will want to learn, but there's an easier way for them to get there and get the results. How have you observed where that latent demand come from? Like in photography? Who are the people who are, you know, I don't have money to buy a camera, but I yet I want some sort of results from Photef.

[Zach Xiao]

Yeah, yeah. I think photography is such a. It's not just art, it's also a document of our record of your important moments in life. So that's very different. And a lot of people, they want a good recording of their moments, their lives. They may not necessarily have the skills to take that photo, they may not have the equipment to take that photo. And for them they can only in the past they try to buy a fancier iPhone, maybe the latest generation iPhone so they hopefully they get a better camera out of it, but they don't have the skills to do editing. Those are the latent demand that we try to address. You know, we really feel like whether it's AI model or AI agents, creative tools that we can develop to make that easy for them or even do their job for them so that they can get these good photos of their important moments in their life. That's what we want to do.

[Yoko Lee]

Yeah. So one thing we had in a firm which is like we had photo shoots for people's headshots and those like your premium and you know, invite your photography and then the whole studio and lighting and makeup and recently I found that there's more and more folks who are gorgeous. Like I couldn't make it to the photo shoot, but I'm going to generate it on fota, which, you know, it's like a very interesting segment of market where you need a physical presence to take a photo. Either that's a product or human or something else. Have you seen that too like on the new photo users?

[Zach Xiao]

Yes, yes, I think and also like photo shoot is if you think about it, it's a setup environment anyway. Like you are setting up the lighting, you go there and you sort of take the photo. And most of the time you don't have all control over things just because of physical limit. When you're doing it in a generative ways. There's so much thing, so many things that you can do. You can change the lighting the way you want, you can change the angle, you can even change your makeup if you want. All of this, I think it's just so important and people are finding those use cases for sure.

[Yoko Lee]

Yeah. I would love to hear from actual photographer's point of view too, because Matt, you take a lot of photos, so I guess how do you think about this old versus new world of AI photography?

[Matt Tancik]

Yeah, it's fun. I almost you. There's definitely like the overlap where you can do them together, but then there's also this very interesting where you're like treat them as very different medium. When you start doing things that are purely in the AI world, you can start thinking about what would an image look like maybe halfway across the world or in specific studio lightings, situations that you would never actually be able to do, but you think it's like a cool idea or image. I find myself sometimes going and seeing a cool image out there and trying to think, how would I actually replicate this in AI? What are the right terms to use? And you kind of go through a rabbit hole to get to that point. So it unlocks like a new area that I haven't been able to explore before.

[Yoko Lee]

Yeah. How do you think about the role of traditional tools in the new world of AI Creative tools? Are the traditional tools becoming infrastructure where, you know, they're always going to be here and now we just train agents to know how to use them for, you know, bits and pieces or are they getting rebuilt?

[Matt Tancik]

I think at the end of the day, all these things are just tools. And so it really just comes down to whomever has some vision and they want to execute on that vision what tool is right for them. For some people it'll always be Photoshop. For probably a newer generation, it won't be. And so it'll just be kind of dependent on the person. And as you mentioned, maybe it'll be agents that'll be interacting with these tools. So I think it's hard to say. I think these things will always exist to some extent, but it'll find different uses by different users.

[Yoko Lee]

Yeah, yeah.

[Zach Xiao]

I really like this framing. It's not like we're having this new problem of like, oh, very new tools, very old tools. I feel like the tools have already always been iterating in the past. Like there's always new tools come out, whether it's AI based or not AI based, there's always new Tools come out just. We're probably at a stage where the tools are iterating so fast. So I do think some tools, you may not need them in the future anymore, they're going to be replaced, but a lot of tools will still exist. And even the AI models that we have today, they are going to be replaced as well. Like new models come out. So just an iterative process.

[Yoko Lee]

Yeah. What is the benefit of having your own model as part of what you put in the toolings that you build? And is there a benefit in open sourcing the model?

[Matt Tancik]

So I think one of the biggest benefits of having your own model is the ability to build off of it. At the end of the day, your product has a certain set of users and you want to do the best thing you can for those users. The best way to do that is by customizing it, maybe making adjustments to the model that don't make sense for say, a general open source model, but makes sense for your product. And that's one thing that is not always easy to do on top of existing models and may require some specialization that only you can provide in your own models.

[Zach Xiao]

Right? Yeah. For us, a little bit different. We don't directly do foundation model. We mostly work on personalization. The reason is because there's so many cool companies out there like Luma, we really think personalization is something the user should own. Like that's the user's model. It builds the user's model for them, but at the end of the day they're using that model, they have full control of that. So for that particular thing, we want to build a technology to enable them to do that. But they own the model and we want them to be able to combine that with any foundation model they like to use. Right. It could be a specialized model that DOMA specifically for a particular use case, for example. Right. And to really disentangle this, personalization versus foundation is sort of the goal that

[Yoko Lee]

we have with advancement AI tools. What do you think will separate the best artists from everyone else in the next couple years?

[Matt Tancik]

I can say from my experience already it's very obvious when, when people use our tools, or not even our tools, any of the AI tools, you can tell pretty quickly someone who has, who's just throwing in a prompt and getting a result out versus someone who has thought more about like the whole holistic story. I always get amazed at what very creative artists can do with these tools. Um, many times you see things that just don't seem like they would have been possible given the tools capabilities. But artists have figured out how to make it possible. And so I think that's really the thing that separates it. It's how do you make something unique with the tools you have access to now?

[Zach Xiao]

Yeah, I, I think, I mean, it's hard to say there's good artists and bad artists, but like, they're probably good artists and average artists. I feel like with AI tools, they're definitely raising the lower bar just so that even someone like me can create something pretty, pretty hid with the AI tools. But I also think that the gap between the very best artist and average artist is going to be even bigger. Exactly. Because what Matt said, there's so much easier for people to express their creativity when they have a very good understanding of what they want to get, how they want to get there. And they can use these powerful tools to do that. They can achieve something they won't be able to achieve before. So the gap is going to just get bigger. And of course they also need to have very good understanding mastering all these new AI tools as well.

[Matt Tancik]

I think maybe one final point. You've seen AI generations that are bad, right? I think we've all seen them that are bad. I can guarantee the same tools that made those bad generations, they've also made generations that you would think are amazing.

[Zach Xiao]

Exactly.

[Yoko Lee]

Yeah, yeah. Something I heard from someone today earlier was they assume AI to be slop. Like anything AI models have would be slop. It's up to the humans to create something out of it.

[Zach Xiao]

Right.

[Yoko Lee]

Just because if you just like say I want a picture of a coffee as like a cup of coffee. The first generation is just mediocre. You can't have it in any stock image website. It's nothing unique or it's nothing personal.

[Matt Tancik]

Right.

[Yoko Lee]

So it really is up to the user to kind of make it like theirs to kind of put their mark and style on there. What are your thoughts on personal styles? Like, in terms of personalization, like, what constitutes the style, what constitute personalization for the users?

[Zach Xiao]

I feel like that's such a hard question. Like, it's almost like, I almost feel like there's no good way to define it other than you can learn that from data. So. And it's not just like, okay, the user's existing data, it's also like when the user use it, what's their reaction to it? How do they like it or not? Because even if you ask the user what's their own style, I mean, they can put some keywords there, but it's not complete. So I feel like, you know, if the way to solve the problem is you really to kind of define their style and then model their style, that probably not the approach I would take. Even from a research perspective. I feel like you look at the data, you find the distribution that the data shows you, and then you probably have figured out that style. And even at that time it's still implicit because it's part of the model.

[Yoko Lee]

Yeah, I see. Interesting.

[Matt Tancik]

I think that's also something that changes over time, both as a person changes. Like I know the stuff that I thought was interesting five years ago versus now is very different. But then also, you know, I'm working on one project one week and another project another week and I inevitably need a different style. And so maybe that's how you define like the styles for specific tasks and maybe personalization sort of the holistic, like what you as a user tend to like. But yeah, I think it's, it's a bit mixed.

[Yoko Lee]

Awesome. And with that. Well, thank you so much for coming on the pod. This is such a fun discussion. Yeah, I really appreciate, appreciate it.

[Zach Xiao]

Yeah. Thank you for having us.

[Podcast Host]

Thanks for listening to this episode of the A16Z podcast. If you like this episode, be sure to, like, comment, subscribe, leave us a rating or review and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts and Spotify. Follow us on X16Z and subscribe to our substack@A16Z substack.com thanks again for the listening and I'll see you in the next episode. As a reminder, the content here is for informational purposes only. It should not be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com disclosures.