Loading summary
Transcript44 lines
- [00:00]
Rid
We talk a lot about using AI at startups, but what are the more established companies doing to scale AI prototyping internally?
- [00:08]
Kyler Hall
With AI, it's a lot more about like, okay, how do we document this so that it's available in the AI and the LLMs memory at all times, as opposed to typically with the design system? The way I see that we would do these things is through programs and people and cultural reinforcement, design reviews, that sort of stuff. Now it's like, okay, can we just tell it exactly what we care about?
- [00:31]
Rid
How do you use your design system to get the most out of tools like replit or figma Make?
- [00:36]
Lewis Healy
Now we're kind of going into a fluid model where anyone with any tool can essentially ship to a customer, and we need to figure out how to support that, like, design system. Remit has just blown up into, like, anyone in the organization can essentially ship. And that's a really challenging problem to solve.
- [00:57]
Rid
Welcome to Dive Club. My name is Rid and this is where designers never stop learning. This week's episode is with Lewis Healy and Kyler hall, and they're doing a deep dive into all of the ways that they scaled AI prototyping at Atlassian. And I want to start this conversation by taking a look at how they're using templates in FIGMA make, because I've.
- [01:19]
Lewis Healy
Never seen this approach before with AI prototyping. At the beginning, it very much was about just like allowing people to spin up or ideate ideas with code. So very much like, how do we augment product managers, product designers, content designers, to essentially create a coded prototype of their idea? Initially, when we were kind of piloting AI prototyping earlier this year, Design System Team wasn't necessarily the key integration for AI prototyping. It was very much like, oh, let's just focus on low fidelity and just spin something up. And very quickly, once we gave people access to these AI prototyping tools, it very much was like, well, I need this to look like an Atlassian experience. How do we do that? And that's when the design system team came in and we kind of created what you're seeing here, these templates. So this template essentially is that starting point, that kind of baseline for people to spin up a experience, so whether they want to create their own kind of like sub templates. So we have a lot of apps in our collections at Atlassian, so we have jira, Confluence Loom. They may want to create their own specific template for product designers to use and kind of experiment ideate on. And they don't want to have to create a roadmap every single time. But to do that, they want to essentially get some very similar content, which is like the top nav and the side nav. They kind of need to be the same or very high fidelity. So we've realized that if we create this kind of base template, then you get these things like the top nav and the side nav kind of consistent, but then also allows them to then make their own version. We've landed on this, like, kind of abstracted template where it's not actually a specific product, it's just a bunch of elements that the AI would usually get quite wrong. So our top nav and our side nav was very, very hard to consistently generate. What we would find is when we kind of initially creating these instructions to generate these prototypes, it would hallucinate the icons, the navigation elements. Like, our navigation system has a lot of imports and it would just always get one or two wrong. You know, if people were uploading a screenshot and trying to kind of replicate an experience in production, the top nav and the side nav would also be incredibly wrong because we're in a kind of state of improving the visual refresh of it. This is why this template was born where we like, okay, let's just code the top nav, let's just code the side nav. And then when people are uploading a screenshot of the side navigation elements that they want, we found the agent Figma make and Replit is actually very, very good at just changing the code that already exists. When before, when we were focusing on our instructions, it was taking nothing and trying to build everything. So we found just that kind of that initial starting point reduce probably the error rate from maybe like half of the prototypes were having a lot of navigation issues to probably nearly zero of them. Maybe just a few icon hallucinations here and there. But if you upload a screenshot of any side navigation in Atlassian, it's going to get a lot of the combination of navigation elements pretty correct and that really supercharged prototyping for people. Because what we found is people were spending like maybe two or three hours just trying to get that top nav and side navigation pixel perfect. Because that's what makes you feel like you're in Atlassian experience. And when you're testing with customers, you want the top nav and side nav or the Chrome to feel like you're in an Atlassian product. Otherwise people are going to get too distracted. But the content, it's not that important. What we found is if you kind of do any content or any main content in this area, you can kind of get away with it. But when the top navigation and side navigation is incorrect, people start to get a little bit confused. The wayfinding's a little bit off. So we really solved that by having that hybrid approach of like a pre coded template with design system instructions. So then when you're building on top of it and when you're creating more product specific templates like a roadmap, et cetera, it's not going to get the basic elements and incorrect.
- [05:34]
Rid
Real quick message and then we can.
- [05:35]
Interviewer
Jump back into it.
- [05:36]
Rid
So I got a new computer recently and what do you think was the very first app that I installed? If you've been listening to this show for a bit, then you probably guessed that the answer is Reiki. But at this point it is an extension of my fingertips and a fundamental way that I use my computer. And I'm not alone. I mean, I see this sentiment from people I looked up to on Twitter all the time. So if you're still on the fence, just do it. Download Raycast and thank me later. Just head to Dive Club Raycast to get started.
- [06:07]
Interviewer
All right, here's the thing.
- [06:08]
Rid
You don't need another dashboard. What you need to do is to talk to customers. So I to want, want to introduce you to Genway AI. You can think of it kind of like Vibe researching. To validate your ideas quickly, just draft your questions, select an ICP and then their AI agent runs interviews on your behalf by pulling from a panel of global participants. You could literally set it up in the morning and get actionable insights by lunchtime. It's validation at your fingertips and you can try it out free for 14 days. Just head to Dive Club Genway to get started. That's G E N W A Y.
- [06:49]
Interviewer
Okay, now onto the episode. It's genius because I've spent so much time just making little tweaks to the shell of my UI when actually the only thing that I want to prototype is a piece of the content or maybe it's a different layout on the screen. And I don't, I don't care about the sidebar. I just want it to feel a little bit real. I mean, I've got so frustrated with it that at times I've even taken screenshots of prod and my first prompt is add this image as the background to the page. And then I just put components on top of it because I'm tired of adjusting the Shell. And so this starting point makes a lot of sense. I'm even just going to restate my understanding. So for myself and everybody, it's like really clear how this is being used. So we're looking at Figma make. This is presumably a file in a project somewhere that anybody can just run and duplicate. And it exists in code. It's not any one real product. It's honestly kind of just a set of like core scaffolding and subcomponents and sidebar interactions that the AI is then using as a foundation to build whatever that person wants. And then you've even baked some education for how to use the template into like the core page layout itself. And so anybody can open this up and they just start typing and talking with the agent about what they want to make. Is that kind of correct understanding?
- [08:09]
Lewis Healy
Absolutely, yeah. So this template, you can either use it where you're creating your own sub templates, so you know, that kind of JIRA specific roadmap that then people then want to then duplicate. So it's a bit of a network effect or you can just, if you're creating tile in your experience, you'll just ideating something new, you just duplicate this template and then you can just, you know, go for your life. So we've included some, you know, key instructions to people, you know, if you want to change the product name to Jira. What we've actually tried to do is we found that hallucinations of this icon and this logo were very, very common. There's a huge spectrum of confidence and understanding of how to prompt at Atlassian. And what we found some people were saying like change logo to jira. And what it would do is it would go based on its pre trained knowledge and it would just pull in a really old JIRA logo and then swap that out. So what we've actually found is by having a configuration object of just some hard constants that then includes product icon and some elements and then hooking that up to the top navigation component randomly, we found that actually it hallucinated a lot less. And then on top of that we found it hallucinated even less when we gave people like a copyable command that rather than saying change logo to jira, they can just click that and copy it and then they know how to switch things into that logo. So we've kind of had this really hybrid approach of like pre coded elements, you know, design system instructions, but then also the user experience of how to reduce very common and like annoying hallucinations that would just, you know, create that extra one or two prompts. But it would probably really lower the bar of people's confidence that they can't seem to get the simplest stuff working. And that's because, you know, the AI doesn't know about the intricacies of these new logos we've created. It's just going based off what it has been pre trained on, which is like, oh, this is the Jira logo. So we found that really, really effective.
- [10:13]
Kyler Hall
Before this, we just tried to give people a set of prompts like here's 100 lines. Copy and paste this into Figma Bank. It might work, it might, it might set it up. And a lot of that's like we have a bunch of theming. We have like feature flags and feature gates that have to be turned on that aren't even documented well outside of internally. So it's very hard to ask an LLM to do that. The only way to really do it is basically have an engineer tell you how to do it. So we want prototypes to kind of show off where, where we're headed. We, we might use them for demos and design reviews, that sort of stuff. So we of course want it to ideally be cutting edge and possibly even even, you know, future baked. So a lot of that just requires us to kind of bake in a lot of the code, things that a designer should never even know exist. We started with those prompts and a lot more of them and then over time it's just like, well, what if we just bake the entire default that an engineer should know by, by heart into these templates and people could just iterate very quickly.
- [11:10]
Lewis Healy
We've also taken that a little bit further as well, where you may not want an entire template. Right. Like we have, you know, Rovo, which is our kind of AI and Atlassian, and that's kind of interfaced through a chat box. Now do you have to duplicate a template to just add Rovo in? Probably not, right? Like if you already have an existing prototype, it gets a little bit cumbersome. So what we found is we've created this thing called recipes where they are kind of what Carlisle mentioned before, which is like a code blob with some instructions basically of like, here's how you recreate the chat box and then this is where you put it. And it's not going to be pixel perfect to what is in production, but it's going to be good enough to get that look and feel without having to make every single user of prototyping upload a screenshot and Try and get that pixel perfect. So we found this kind of recipe approach as well, really helps with those smaller elements that may be a little bit more technical for people to do. Another example is like dark mode by default. Some of our user Personas are that developers probably aren't using light mode by default. And if we're prototyping, maybe we want dark mode that's involving a lot of theming coding. It's a little bit tricky to actually change the default mode to dark mode. So we created a recipe where you can just paste a bunch of instructions in and it will just switch out the mode for you into dark mode. So we're trying to think of the user experience of a non technical user using code which can be a challenge in itself.
- [12:47]
Interviewer
Okay, so this is the end state. It's beautiful. You're introducing me to a lot of concepts that I haven't considered before. Even the way that you're thinking about recipes is genius. I think my goal after seeing this now is to kind of understand like, how can someone else reverse engineer this system that you've built? Because I'm pretty sold. So can we even go back back in time a bit and talk about like, what did it take to even arrive at that end state? And maybe we can dig into some of the lessons that you've learned along the way.
- [13:18]
Lewis Healy
Essentially how we started to integrate the design system is, you know, Carla and I thought, oh great, sweet. We're just going to take our Atlassian documentation, we're going to put in a bunch of examples or a text file, upload that text file and say, now build with this. And we got some pretty rubbish results because first of all, the file was massive. So it was just truncating the information and only creating a subset of the components. But it also didn't really know how to use those components and how to use it in an AI prototyping world. Because what we were doing is we were applying just the mental model of a human using documentation and trying to apply it to a machine or an AI. So that was a very, very quick learning of. Yeah, okay, that's not going to work. Let's actually just figure out how to actually talk to this machine. And I think some of the cool things that we did is we got some really good results by actually trying to talk to the AI in a way that it understands. What I mean by that is that we have a guidelines MD file which includes all of our design system documentation. Now eventually this will be an mcp. Once we can plug in MCP but for now these are instruction files and this is essentially how we get design system generations to a pretty high fidelity. But there's an interesting thing that we found really valuable, which I've not seen too many people do, which is we try and instruct it to kind of think in tailwind still, which is obviously gonna, it's, it's pre trained data. I assume most models have been trained on shad, CN and Tailwind because of the amount of open source code out there, which is, you know, why is react as well. We try and go, okay, generate that. When you think in tailwind classes, once you see this thing, oh, it's actually this design system component. So what we mean by that is we actually have, in every instruction, in every one of our components, we have a translating from tailwind section where we say, if you see these class names, actually it should be this react code. And we found this actually really beneficial in terms of reducing some of those hallucinations where you know, we have a component and Atlassian called lozenge. Now I don't see, I don't think I've seen lozenge in many other design systems. It's a very name. And then whenever there's new, new people joining, joining Atlassian, they're like, what's a lozenge? And then we have to explain it. So AI is probably not going to know what a lozenge is. It's probably going to think it's something else. This provides an opportunity for us to kind of say, oh, you see it this way, feel free to think that way. But then now translate it into something that we want to generate. So you can see here like span, if it's using these tailwind classes, it's actually this important badge and we found that really effective in reducing quite a lot of hallucinations. And at the top we then have really specific instructions saying use Atlassian design system first, use design token first and then use Tailwind for missing things and trying to swap things over.
- [16:25]
Interviewer
Who's responsible for this document? I mean this is about as robust as a guidelines document I've ever seen myself and Carla.
- [16:31]
Lewis Healy
The first version of this was fully vibe coded. So Carla was away for a month I think in Japan. And I love working with Carla and I was just in this, this depth of like having to get this design system integration working because we were in the middle of a pilot and we had 300 people wanting to just like use design system and I had no way of understanding how to do that. So the first set of These instructions was absolutely Vibe coded and I'm sure Carlo can attest to it. There was a lot of discrepancies. There was a lot of just contradictions and everything, but it got really good results. Then after that, we then decided, okay, how do we actually centrally manage this and how do we actually generate this with some level of consistency? Which I'm sure Carlo can talk about how we got to that now.
- [17:19]
Kyler Hall
So on Atlassiano design, we have an Elements full file and that's about 5,000 lines of just gente content. And we'll say 90% of that was vibe coded about nine months ago because it was like, okay, let's actually, you know, I think, I think the, the journey for a lot of us was like, oh, hey, we can, we can use cursor. Like it's allowed to use cursor. We also got VS code and like different things click for different people at different times. I think I clicked with cursor and I was like, okay, what does it look like to teach it how to work with the design system? And a lot of that was just like, okay, let me just create instructions to tell it how to use our token system. Because it does not get it from its industry training model at all. So we just have to basically make a table of here's every single token. So this is like the first version that we, I would say we started with I think back in like March. Well, this has probably come a long ways from the first version, but this is the, the current iteration of the first version, which we were like, okay, what is, what does it look like? Just to put all of this agenta content? Like, there's a bunch of prompts floating around which is like, hey, how do you create a button? How do you style something? How do you use our tokens? How do you use our theming? That sort of stuff that was floating around for probably a month and it was like, okay, can we just make this public? So we made an elements. Txt file. The full version is kind of all of our content truncated together. And yeah, kind of a lot of it was just, let's, let's create some mouse to explain like what these things mean. And then over in Figma, make and replit, we've actually, in addition to showing light and dark modes, will actually show like a tailwind class as well. Just for context. I don't have that up. But yeah, this is really where it started. But this is, yeah, this is like 5,000 lines. Whereas we've kind of settled on something more 2 to 3,000 lines within prototyping. That's a little bit more succinct because a lot of these are very verbose in relatively good ways. But it takes a lot of tokens and a lot of context actually generate with this. So we've tried to avoid using it in full, if that makes sense. But yeah, this is basically where it started generating effectively just vibe coding. Okay, go, you know, build a button section, go build a lozenge section, go build the token section. Here's the file. And a lot of that was just hallucinated originally until I could go in or me or another engineer or Lewis could go in and be like, hey, these are all the wrong media queries. Let's say, let's go and fill in with the right media queries and it's like, oh, okay, look at this file instead. Don't just use industry trained things and then we get all the right things. Lewis probably used some things like this to basically be like, can we have you know, a Figma make or replit version of this? And then, oh, tailwind doesn't work. We're hallucinating more on icons than we were before. So let's over index on certain things. And then I think there were also some cases where it's like we don't need all of these props or we don't even need the hide component. It probably knows how to do that. So this is like everything. Whereas I think replit and Figma make, we probably show only 20 or 30 components as opposed to like the hundred plus we have. So not only is it specifically our content, but it's also a very drilled down version of our content for a.
- [20:37]
Lewis Healy
Timeline for you RID as well. Like this LLMs T was the first created file then that's what I tried to use when I then vibe coded when Carla was away for a month, then had to vibe code my own thing. And then once Carla came back, we then figured out how to make a scalable version.
- [20:52]
Interviewer
Okay, so we saw the guideline file. We've talked about some of the lessons learned, even about giving people coded lines of text that they can just like paste into prompts, whether other areas or clear points where you just iterated. You did something different. You learned something about how this process should work, where you made a change or this just influenced the way that you thought about what it would take to enable AI prototyping at scale.
- [21:18]
Lewis Healy
We were getting pretty good results, like maybe 60, 70% from like what we call a one shot. So I was like benchmarking the effectiveness of these AI prototyping tools. And I was kind of screenshotting a simple card, and then I'd screenshot a really complex screen, and I would see how well it performed from a single shot or, like, a single prompt saying, like, build this screenshot and then see what the results were. And we were getting.
- [21:41]
Interviewer
Is this just based off of. Just on the guideline file and that's it?
- [21:45]
Lewis Healy
Just off the instructions? Yeah. Okay. Just off the instructions. It was getting, like, 50, 60% accuracy. Like, with the hybrid template approach, it was, you know, getting a lot higher. It was hallucinating otherwise, but I was still getting annoyed by certain parts of it. And that was, like, things like our text and our icons, and I was just obsessed with trying to just get it higher and higher. And I had a little bit of inspiration from, like, printer sheets. So, like, you know, when you, like, print and you need to configure your printer, it prints, like, this special sheet of all these, like, colors and shapes and patterns, and then it's used to then configure the heads of the printer or something. So I was like, what if I just did that for AI prototyping? What if I just put a bunch of things on a prototype and just said, describe it to me. How would you describe this? How would you actually prompt this and then use that to then reinforce the instructions to basically talk to it how it expects to be talked to? And that really helped with kind of, like, understanding why the icons were being hallucinated. And then also things like text, like why it couldn't figure out certain aspects of text. And I found that it was kind of hard to determine the font size of something when I asked it to kind of be like, oh, here's a bunch of text. Tell me the font size is. Tell me the font weight. It would often hallucinate that from just a screenshot. I found that, yeah, like, with these sticker sheets, that it thought differently than what I thought. Like, so then it was very much around, like, okay, how do we actually meet in the middle and improve those outputs? And we actually got way better kind of primitive layer, essentially, like our typography components, like heading and text that started to be rendering a lot more accurately. Because when it was seeing a screenshot, I was talking to it in the way that it actually read that I'm kind of like, talking to the computer vision element of it when before I was very much just like, oh, this is Atlassian. Deal with it. So I find that if you really try and go closer to the metal, then you're probably going to get better results and understanding why.
- [23:52]
Interviewer
Is there an example that we can use just to get specific about it? Because I'm nodding along because conceptually it makes total sense. But I'm also like, okay, but like, where and what is that change that would allow you to kind of close that gap?
- [24:04]
Lewis Healy
You can see the prompt here, which is like, I want to calibrate your computer vision model. Like, can you add the image in and then create bounding boxes? So this is very much like me just trying to get it to tell me about these elements and what it picks up. Because what I'm suspecting is a lot of things around certain components that have a weak border or things with more white space. It was really struggling to pick up the bounding box of them and then recreate them. So this really told me a lot about the limitations of the computer vision. And what I found as well is the more complex the screenshot, the less it picked up and actually truncated and missed a bunch of elements. So, you know, like things lower down on the. On the side navigation of a screenshot as an example. So let's say this is our side navigation. Things around this prompt box or anything were just being completely skipped because it probably has reached the context limit of, like, how much it can ingest. And that really told me a story around. Oh, okay. Like, first of all, I have to now instruct thousands of people prototyping that. Okay, when you're uploading a really complex screenshot, it's going to miss stuff. So break things down into, like, sections of, like, the top left side now. So it really helped me improve the, you know, the training I gave people. But also it showed me, like, you know, as you can see here, the bounding boxes that it's created around it and how it interprets certain elements. So I got it to, like, name things as well. So you can see with this one, we have a component called icon tile. I was really interested to see how it picked up these smaller tiles with these kind of subtler imagery, which we have a lot in our products. My suspicion was it's not picking it up, and that's why it was hallucinating it. And it was true. So you can see with these, this size, it didn't pick up the color at all. It just picked up the icon. So I was like, okay, I need to provide more specific instructions to improve the outputs of this or help users go, okay, if you have something that's being hallucinated with an icon tile, which is this component you're probably going to have to use maybe thickness mcp or you're going to have to upload a smaller screenshot or like a zoomed in screenshot of it and update it piecemeal. Because if you're uploading a big screenshot, it's just going to miss that. So this didn't necessarily always influence our instructions of our, you know, our design system instructions that would then generate the code. It very much also influenced how I actually talk about how to get the best results from that kind of one shot with the thousands of people using prototyping.
- [26:39]
Rid
Hey, really quickly let me tell you about the all new Dive Talent Network. I've hand assembled over 100 of the most talented designers and builders that I know, so I can recommend them to my favorite companies. So if you're listening to this and you're open to new opportunities, the Talent Network is anonymous and, and super low pressure. It's just an easy way to see what's out there without having to post on social media. So if you're interested in joining or maybe you're looking for your next hire, head to Dive Club Talent.
- [27:08]
Interviewer
I mean, gosh, you're dealing with a heck of an adoption problem, I would imagine, and a lot of education required to get people comfortable and out of their comfort zones and whatever tools they were familiar with before. So what have been some of the tactics that you've had the most success with and maybe any key learnings along the way in terms of how do we set people who are not as technical up for success with AI prototyping?
- [27:35]
Lewis Healy
Our first problem is getting people to even see any training you've done. So we've got hundreds of designers, hundreds of PMs, hundreds of thousands of engineers. We've probably got around like 11,000 employees. So there's thousands of people that are going to be using AI prototyping at one time. It's almost impossible to even reach that many people. Even if I DM'd 100 people a day, right? It's every time it's going to take amount of time. So that is the kind of starting point in your brain. It's like not everyone is going to see this thing. And I've done a lot of training, like adoption training, design system training before, and I found that if you build it, they do not come. You really have to make sure it's as simple as possible and also in the different formats that people learn in. So the first approach we did was like, okay, Looms, let's focus on looms, let's focus on video guidance, but then also written guidance. We found that really effective in just providing a baseline of 101 prototyping essentials, right? Like here's just the basic stuff that you need to know to succeed and a lot of that covered is like one of the videos I recorded. Was it just a UI tour? Just a 3 minute UI tour of this button does this, this button does this, this button does this. Because a lot of what you'll find in a lot of people aren't that exploratory, right? They've maybe got a top down push to, you know, you need to use AI prototyping AI maybe they got a bit of pressure, they may not have time to click every button and understand what it means. And I found it was really effective just to tell them this does this, this does this, this does this, that's important, that's not important, don't worry about that. That created a really nice baseline for people and then you start like building up the confidence and the knowledge in more advanced topics. But the 101 was really around like here's the UI, here's this new thing you have to learn and here's just some tips on how to get the best results. So like that thing I mentioned before around like, you know, if you want to recreate a screenshot, which is like one of the most common use cases, right? Like I don't have this thing, it's too hard for me to constantly recreate it in figma. I just want to upload a screenshot and then start from there so I can iterate or add a text box or whatever. Then it was providing guidance on okay, you know, it's going to hallucinate or it's going to get things wrong unless you do things in this kind of way. And it was just, you know, really providing, providing that guidance in terms of distributing that guidance. We've had a few ways we've tried to be creative as possible. We had like a dedicated AI Builders week which you know, we've talked about publicly, which was, you know, our president Anu just basically said everyone, thousands of people, tools down for an entire week and you all just get to learn AI prototyping. And it really created a lot of aha moments for people where they were then given that space. And we, we had training, we had, you know, guest interviews, we had masterclasses. I did like a 500 person replit masterclass across 2D OS which is incredibly stressful but incredibly valuable because we went through Step by step, you know, this is, you know, let's build something together. And it gave those, those moments for people that are like, ah, okay, I can't do this in figma, this is too hard. If I'm creating a filtering experience, I can't build these interactions in. But in Replit, in figma make, I can do it in five minutes. I can just instruct it, like, build this filtering sequence and it will literally work and feel real. So we kind of made sure we had a lot of space for those moments. Now we're in a phase where it's all about we've got a 101, we've got a 201, we've got a lot of guidance. How do we get that to the people that need to use it? You know, maybe inactive users or people that haven't seen the course material before. What we actually did is we created a Slack bot. So I created it in Replit. Well, first I created it in cursor in like 15 minutes and then I created one in Replit where it's got kind of a WYSIWYG dashboard and we have an AI enablement bottom. And what I found is if you add a bunch of people to a Slack channel, they're probably going to ignore it. At our scale. You know, you've got slack fatigue. If you DM people, they will respond to you pretty quickly and they will almost like feel like they have to respond to you. So I was like, how can I replicate that at scale, right? Like if I put people on a channel, I do an app channel or an app here in whatever communications tool, most people are going to ignore it. So what I created was a Slack bot that creates a group dm. So our design ops leads can then be the key recipient. They can add in maybe 400 people that they want to reach out. Maybe they're inactive users or they haven't used it recently. And it will group DM from the Slack bot to that person with a kind of a canned message of like, oh, hey, you know, hey, first name. We've noticed that, you know, you've been inactive user. Can you fill in this survey and let us know? We got a lot of engagement from people around like, oh, I didn't realize this was a thing. Or like, oh, I'm so sorry, I've been busy. Xyz and it gave us a lot of really good feedback on why people aren't using the thing or what the gaps are without having to like DM hundreds of people constantly. So that was A real unlock for us on how to enable at scale.
- [32:48]
Interviewer
Are there other challenges or things that you're thinking about in terms of just maintaining this system and how everything works, especially as your product surface area continues to expand?
- [32:58]
Kyler Hall
One of the other things we've had a lot of success with was that I've seen from the engineering side is a lot of design leaders are going out and sharing prototypes. So a lot of this is like coming in our design reviews top down in weekly loom sort of thing. Like, oh, here's a prototype I built in this, in this new tool, new technology, or here's an idea for how we could do this with Rovo. So seeing that from, you know, your skip lead or your, you know, your leader in your design space, I think that's kind of promoted a lot of virality where it's like, oh my, you know, my boss is doing this or my boss's boss is doing this. Maybe I should do this as well. And then, and then it's like the, the good demos in our sort of design reviews appear to be the prototypes. So it's like, oh, hey, if you, if you come in with just a static design, you know, it might be a design everything, but if you come in with a prototype, like, people will actually get a little bit excited and a lot more commentary on the looms. That's what I'm seeing from afar. So I think that's really helped stir this thing as well. Is like, it's not just flashy, oh, we could use it. Here's some learnings. But also people hit the ground running and just started using it on a daily basis. Both high level leadership as well as like, you know, low level ICs. Yeah, that's been cool.
- [34:15]
Interviewer
I, I've noticed that too. Like, there's a buzz when you share something that is fully functional. It's just, it's cooler. There's like a novelty factor still that draws people in.
- [34:24]
Kyler Hall
Yeah. In terms of like maintainability, how do we maintain this thing today? So I'll, I'll show you a little bit behind the scenes on what these templates actually have. So this is Replay has a lot of boilerplate. So we've kind of put our stuff into this bullet plate. So these documentation files are ours. Like this is the guidelines file that Lewis showed. We have slightly different ones for FIGMA make and for replit. So we kind of split out the examples MD and the guidelines MD because they get indexed a little bit separately with relet from their guidance, we kind of let them guide us on. Hey, how should we serve you this 2,000 line file? I get mostly into the template here. But yeah, a lot of it is just let us define the exact base of the template. So if I go into our actual front end monorepo like this is good code. That's the baseline is you're starting with good code, you're starting with relatively production code. This might even be better than sort of a brand new PoC of an application that someone might spin up. So we'll say this is front end blessed stuff for the most part. There is of course little bits in here that may not be super blessed as we're trying to get it working in this environment because we don't have like GraphQL and the whole teamwork graph and stuff behind it. So there's, there's some, some mocks and other things. Like we, you know, the way we, the way we do theming and routing a little bit is, is not 100% perfect, but I'll go into cursor here, which is what I use and show you how we maintain it. We have the same, just a folder for AI tooling that sits within the rest of our design system. So we have a lot of design system packages. We kind of house all of these templates within the code. So they are actually like production code or at least they pass most of the linting and type checking apparently aside from that one in our code base. And this kind of helps us maintain it so that if we actually change like our theming, for example, this will break. And if we want to update our theme and you know, it's no longer the refreshed version of our topography theme, people will update this and then we can actually go and deploy that to replit so things don't break over time. So a lot of it's just literally use the design system. Now some of this we're going to be transferring into our CSS and JS library shortly. But for the most part we kind of break this down into yeah, a lot of different components, like a lot of different apps. Currently most of these, as Lewis showed, are just basic sort of this is how to use the template. We might be building these up more. So Trello is actually like a proper clone of the template as opposed to, you know, just a hello World page. And then we bake in all of our like our feature flags. So we have a bunch of feature flags which basically control functionality that effectively is required to be turned on in order to have an Atlassian like experience to some extent. Not that many, but like our new logos, some of our visual refresh stuff or topography stuff, we take all of this and effectively we bundle it up and we do a distribution. So these guideline files are generated actually from a very large amount of other content. So these guidelines files, for example, if I go into the avatar here, everything in our avatar package here that we have defined is actually automated and maintained from within our code base. So instead of going through and if we want to document something documented in five different places, which is what happened, we went and we're like, okay, Atlassian Design, atlaskit.atlassian.com, lM's TXT MCP. Now let's add it to prototyping. All five of those places were different content, and at least two or three of those places were vibe coded content. So it's like, okay, how can we make it so this actually represents what we tell to our customers on Atlassian Design. Make sure these are actually our usage guidelines. I think we are in a state of making them better. I don't think they're in perfect parity. We're working through this in the next half, but our approach so far has been some sort of structured content. So we're not going down like the full ditaxml sort of approach, but we are going with something a little bit simpler that we think we could, we could use to maintain, I guess, the couple thousand packages that live within our monorepo. A lot of these are just strings or even markdown files in some cases. And we basically define what our component is, how to import that. So with this, we grab all the types. With this, we actually kind of give it enough context so that the LLM ideally understands what an avatar component is in its own language. Like it's a profile photo or a representation of a user. We give a description, we have examples, we even have examples that are purely for AI. In this case, rather than all of our internal examples, which are a little bit less clean, we have ones that are just like, this is what I want AI to build with very simple, clean, not all the 4,000 different cases we have for this component, but just basically the three relatively basic versions. Because we realized if you give it all of those different types, it will hallucinate more and more and more and think, well, I can do this mixed with this, mixed with this, right? And then the answer is, no, you really can't. So we try and describe, I guess, the 80% mark for most of these components. The same goes with our content and usage guidelines. This is not everything we care about on this component, but it is probably 80% of usages, which is kind of our target in prototyping, to be honest, is the 80% mark. We take all of that, these offerings, JSON files, and bundle it up and just distribute it. So we have a bunch of sort of code, gen and script, basically to crawl the entire monorepo and gravitate a very large amount of files. So we'll just run sort of the distribute command. And then what this does is it distributes effectively a template for figma, make and replit. So we have a fast and full version of our templates. We have a couple others, but they're not really hooked up right now. So if I go in here, everything that you would see in here, all of the examples, we generate this, yeah, from the avatar component, all the guidelines, we generate that from that avatar component, all of these apps, we generate that directly from our monorepo. And what we do is we just copy all of these files. We're looking for a way to sync them a little bit more directly to maintain them even better. But yeah, the goal is really just been about, like, how can we automate the content that AI needs? Because we've had very poor success with just asking AI to go to Atlassian design and read our components or index them or expecting that it knows what our lozenge or our avatar means. Especially whenever we get into the nitty gritty of all of those type interfaces or our usage guidelines, or especially when we're talking about like translating from tailwind whenever it just hallucinates and it can't understand that, oh, this image should be an avatar. The best way we can change that is just going in and being like, well, let's handle this edge case and let's fix it.
- [41:49]
Interviewer
Zooming out for a second.
- [41:50]
Rid
How long have you two been in.
- [41:52]
Interviewer
Design systems roles and how big of a departure has the last eight months been from what you're typically used to? Because it feels like you're almost inventing an entirely new sub discipline within what it means to run design systems at a large company.
- [42:06]
Kyler Hall
I've been here for the past three and a half years on the Atlassian design system. In the past, I've had much smaller design systems, but definitely nothing to this scale. So I would say AI is really a step change for us. So for example, I presented at Config, I think in 2024 around how we do adoption, basically across all of Atlassian, how do we roll out our visual changes, our new navigation, our Dark mode, that sort of stuff. And with AI, it's changed a lot. Where it's like, okay, how would I do adoption? With AI, it is absolutely night and day. With AI, it's a lot more about like, okay, how do we document this so that it's available in the AI, in the LLM's memory at all times, as opposed to typically with the design system. The way I see that we would do these things is through programs and people and cultural reinforcement, design reviews, that sort of stuff. Now it's like, okay, can we just tell it exactly what we care about? Can we tell it exactly what it needs to do? Let's kind of cut out all the fluff and just give it the bare minimum. And then the other side is like, okay, it primarily knows, let's say, Chad CN Lucid, Radix, whatever it's trained on. How can we make our components more closely aligned with that? Like, why do we call it a lozenge component if the industry doesn't have a lozenge component? Or why do we call our prop appearance versus variant or things like that is kind of where we're getting into, where it's like, can we just shift our system to be more, I guess general, for lack of a better word, like the rest of the industry?
- [43:42]
Lewis Healy
I was an adoption person, just like with Carlo. Carlo and I worked together on adoption. I've been with Atlassian three and a half years now. I was kind of the FIGMA plugin guy, then the FIGMA guy, and then I was the kind of design adoption guy. And I really saw AI as a adoption story, just like Kylo in terms of, like, it's a threat to adoption, essentially. It's like I saw all these things being generated that weren't using the elastic design system. And I actually wasn't an early adopter of AI. I was probably actually quite behind on a lot of stuff. And now I'm like lead design technologist on the AI pillar. Because my passion and my drive was from an adoption lens. Like, how do I enable product managers, product designers, content designers to generate Atlassian experiences with the design system? That's always been my mission. Before, it was very much like, how do I increase adoption in figma and how do I increase confidence that then translates into code. This is now just the next step for me. And now obviously I am a lot more involved in the overall AI, but it's for me, at the beginning, it was very much that just like design system, adoption lens and increasing that fidelity. Interestingly enough, I was actually Initially asked to be a tool lead for FIGMA make as we were kind of upskilling FIGMA make. And then it massively scope spiraled into like me essentially being responsible for the design system integration for the entire organization with KYLO to enable these thousands of people to do AI prototyping. So it's crazy how things can just in the space of eight months, just completely shift your entire mindset. Where I went from being a lead designer working on adoption to, to a lead design technologist leading up an AI pillar with this like massive scope of work in such a short space of time. So I'm very grateful for it. But sometimes I look back and I'm like, oh wow, six months ago we were just doing that. Like that's crazy how mature you can get so quickly.
- [45:40]
Interviewer
Well, I mean six months is an eternity in today's day and age with how fast things are accelerating. So I guess before I let you go, I want to use that as a launching point to maybe look ahead into the next six months and beyond. Because you two are thinking about this a lot and you're seeing where the bottlenecks exist, I'm sure probably asking these questions, man, what if we could do this? Maybe this would unlock a different set of workflows over here. Like when you kind of just look into the future, what are some of the things that are rattling around in your mind that you get excited about?
- [46:11]
Lewis Healy
We're trying to look at like what does an AI native design system look like? We don't know the answer necessarily, but it's very much important to look ahead. Like what does the future look like? 3, 5, whatever years. And my opinion is it doesn't look that too different to today because it's just the AIs are going to get better, the tools are going to be more kind of holistic. I'm of the opinion, like what can we actually do today to bring the future forward? And what agents can we create? What tooling can we spin up? What context can we create to leverage that? And also how can we make our own team kind of AI native? So the future is actually really interesting because it's kind of you look ahead of where you need to be, but there's actually just so much you can do today to get you there. But it's probably going to be like duct taped together or like there's going to be an agent that just does one thing really well and then you have to switch to another agent that does one thing really well and that's maybe like today's model. What I feel like is going to be in the future, it's going to be this end to end where maybe it's just one agent, one tool that will just kind of like handle the entire software delivery life cycle. I don't know what tool that would be. I don't know if it would be a third party or people build it as a first party. Who knows how powerful AI can be in three to five years? I can imagine incredibly powerful or the tools that people create are incredibly powerful. But I'm very excited for the way it's going. Where I feel like teams are going to be able to or organizations are going to be able to truly create velocity for the way that they want to write, create and deliver to customers. Where at the moment we're in kind of a fixed mindset or a fixed tool set. Where traditionally the software delivery lifecycle was very much you have requirements, they get translated into design. It's usually Figma or another design tool and then an engineer then has to read the design and then build that. Now we're kind of going into a fluid model where anyone with any tool can essentially ship to a customer and we need to figure out how to support that like design system remit has just blown up up into like anyone in the organization can essentially ship. And that's a really challenging problem to solve. And I think we need to build more tooling, more linting everything to actually tackle that new kind of Persona. Because I feel like design system is the kind of core of an AI native organization or a truly high velocity organization.
- [48:42]
Kyler Hall
The way I see it, as Lewis said, is the remit of a design system is blowing up and I don't think this is new for other companies. A lot of companies design systems are front end platform teams and they are the entire front end platform and they own the front end platform and they happen to have a design system within that or they ship design stuff. For us, a lot of it is how do I enable a designer to ship to a JIRA customer? And that's a very scary thing, scary question for a lot of people. But also at the same time we've got a couple designers that have shipped to production in possibly smaller apps. I can't think of an explicit one in jira, but we're going in that direction. But the further we go in that direction, even the further we go with prototyping, people are no longer asking about how do I work with the design system. The design system's almost like check, we've done that we did that in six months. I wouldn't say it's perfect. We can maintain it better, we can fill in a bunch of the gaps. There were bugs in the stuff we showed you, but for the most part it's good enough to sell that idea. But prototyping for us, I think it will not stop it just showing an idea. Ideally it will go all the way, you know, further along that, like, oh, can we take that idea put into a pull request? Can we take that idea? Can we put it into design canvas? Can we take the idea, put it to production? I think the context that's required for that, for an LLM to know how to build within Atlassian, what's inside my head and hundreds and thousands of other engineers head in order to actually land that is a little bit scary. So the way I look at it, just some, some raw numbers, is like the Atlassian design System is like 75 packages in a front end Monorepo, which is not the entirety of Atlassian front end, which has about 5 to 10,000 packages. So if you want to build within Jira, you not only need to have the context, so the LLM needs to have the context of R75 packages, but also the 5,000 packages, all the different libraries, the tools, the engineering, the content accessibility standards, all of those that are expected of you when you're working in jira. So we have a long road ahead of us, I guess, to go outside of just this 1% box that is the design system. I know the design system is the largest 1% box. It probably makes up 50% of sort of the react code that you might see in Jira. But for the most part that it's that other 50% that will be very, very hard that we have to go and document, I guess second layer systems or even the third layer systems, how to write tests, how to use, you know, our version of CSS and js, how to, how to do all those technical things. Because an engineer can vibe code and say, oh no, you should use compiled instead. Oh no, you should use this internationalization library. It's specific to jira. But a designer has no clue of that. Even myself as an engineer, I don't know how to work in jira. So I think that's where we're going in a lot of ways. That's how we'll get to the end vision that I think we all have, which is can we empower anybody to at least open a pull request and get peer review on that, customer review on that and you know, experiment that into production. I think that's kind of the five year goal.
- [51:50]
Interviewer
Well, you've taken a heck of a first step here. Really impressed by everything that you all have shared. Definitely changed the way that I'm thinking about the role that design systems play and and I'm just appreciative that you guys came on here and pulled back the curtain. And I'm sure that you have inspired a lot of teams out there. So we appreciate you taking the time.
- [52:10]
Rid
Today before I let you go, I want to take just one minute to run you through my favorite products because I'm constantly asked what's in my stack. Framer is how I build websites. Genway is how I do research. Granola is how I take notes during the and Crit Jitter is how I animate my designs. Lovable is how I build my ideas in code. Mobin is how I find design inspiration. Paper is how I design like a creative. And Raycast is my shortcut every step of the way. Now I've hand selected these companies so that I can do these episodes full time. So by far the number one way to support the show is to check them out. You can find the full list at Dive Club Slash Partners.