← Dive Club 🤿
Loading summary
Transcript41 lines
- [00:00]
Rid
We talk a lot about craft in B2B SaaS, but what's it like designing frontier interfaces?
- [00:06]
Ruz Madhavian
I think there will be new interaction models in this world that are, yeah, just very different than having to use our physical hands to, like, articulate.
- [00:14]
Rid
How does the future of brain computer interaction shape the practice of design?
- [00:19]
Ruz Madhavian
What we today call vibe coding. You know, in the future, like, the time that it takes for a model to actually produce an artifact that you described will, I think, come down to a frame. This will be a very exciting and wild experience where you can just sit down at a computer and basically daydream with it, and things will pop up on screen that are a direct sort of extension of what you have in your head.
- [00:40]
Rid
Welcome to Dive Club. My name is Rid, and this is where designers never stop learning. This week's episode is with Ruz Madhavian, who's the design engineer at Neuralink. So we're about to get pretty nerdy and talk about what it's like designing neural interfaces. Like, how do you allow someone to control a cursor using just their mind? And one of my favorite parts of the episode episode is hearing how Ruiz has had to think through every single detail of human computer interaction from first principles. The level of design thinking is really impressive. But before we get into all of that, let's hear a little bit more about Roos Journey, because it actually starts at Apple.
- [01:23]
Ruz Madhavian
I think the earliest point in that journey for me was freshman year of high school. There was this, like, research that had come out of Berkeley that was truly wild. They had basically put folks through an FMRI machine, and they had them, I think it was watch YouTube videos, and they just collected hours and hours of this data. And then they ended up doing some form of, like, quote unquote, machine learning where they could take this FMRI signal that would happen while they were watching these movies and then predict what frame of the movie that they were watching. And then in, like, real time, they could then reconstruct all of these frames into what, to our eye, looks like then a movie of what they're thinking. The results were super wild, like, super cyberpunk. It was like this composition of frames from which you could see these, like, semantic things coming up. So you could see, like, oh, there would be, like, a bird, or it would be like, you know, a beach. Like, you could see some shapes emerge from this noise. And then they took this extra crazy step of, like, them having them go, like, take a nap inside of one of these FMRI pods. And then you could see in effect, like what they were dreaming. This was like something that once I saw was like, hard to like, let go of. It was just like unbelievable. Up to that point, I'd spent a lot of time actually making movies. So the idea that you have these images in your mind that you can now show other people directly was like, really, really magical to me. For that reason. It's like you spend all this time trying to recreate that when you go through the process of like making a short film. The idea that I could just show somebody that directly was just, you know, incredible. That was like, I guess my first exposure to the idea of some form of a, of a neural interface. And then separately, you know, I was also. I think computers in general are a magical thing. So I was. It was really going down that rabbit hole. That's another way in which you can take something that is in your mind and then show it to somebody else and create an experience for somebody else. So fast forward five years. I had this crazy opportunity in college to basically intern on the Apple Watch team, the Faces team. This was like one year after the watch had come out. So it was really the early days of what the interface for Watch might look like. And the Faces team is also a super amazing place to be that is like the thing you actually see all of the time as opposed to being sort of more buried into the layers of the interface. My interim project was what would go on to become the Siri Watch Face. So I was just, yeah, generally really excited about what a proactive computer might feel like. Something that is with you all the time. We have this notion of drive by on watch which was you're checking the time. That is in some ways the fundamental value proposition of a watch is that you go to check the time, but now there's this like kind of opportunity to surface something else while you're checking the time. So the computer in a way is not something you're actively using anymore, but something that sort of fades into the background and, and can hopefully just do useful things for you while you're out and about in the real world. The first version of the sir to Watch Face was sort of aimed at exactly that, which is like again, more primitive forms of machine learning than I think we have today, but a way for us to try and figure out what is the most relevant piece of information that we can give you. Whether it is like a calendar event that you have coming up, the sort of cheesy examples, but also more contextual things like, you know, if you have been going For a walk. Like we can surface the ability to start a workout right? That was just a ton of fun to go through that whole flow at a company like Apple. I guess like my first experience, going end to end from just a concept that you have in your head, to the initial explorations of what that might feel like, to the actual prototyping work, building it, and then living on that, and you sort of go through this loop over and over again ultimately to ship something that feels awesome. So I came back for another year to build that out further and then I joined full time after graduating to just sort of continue that work. So after a year there, I was still thinking about that initial sort of exposure to neural interfaces in high school was definitely like in the back of my mind still. And of course Neuralink had publicly announced a couple years prior and was still doing a lot of this work. So after their first demo, this was like July 2019. It was just watching that it was clear that this is actually going to be a real thing that people get to use. It was like late 2019 that I made the jump from one frontier interface on watch to another. In this case neural interfaces.
- [05:17]
Rid
Real quick message and then we can jump back into it. I saw a scroll stopping tweet the other day. The creators of Tailwind are working directly on paper to train the output to be perfect. They even invested in the company. So just think about the possibilities for a second. In the future you could design something in paper and then just right click and copy the perfect Tailwind as if the creators themselves wrote it by hand. Or maybe you take an existing code component and import it into paper to make edits directly on the canvas. I mean this is going to totally, totally change how we Design and deliver UIs for the web. And it's just another reason why I'm betting big on Paper as the next great design tool. You can try it out today. Just head to Dive Club Paper. By now you know that Jitter has been my go to tool for animation for years now. But they're still shipping like crazy. I mean, just this summer they've released comments, Pen Tool, morphing text, gradients, Google fonts, and a bunch more. So if you haven't yet, I promise you will be shocked at just how easy it is to bring your designs to life in Jitter. So go ahead and give it a try today. Just head to Dive Club Jitter. I've been designing products every day for the last 15 years, but in the last six months everything has changed. With AI in the mix. I'm cranking out ideas faster than ever. But none of that matters if I can't get the feedback that I need to get the team aligned. And right now, getting async feedback still kind of sucks. So I'm building the product I've always wanted, and it's called Inflight. I use it every day to share ideas and get feedback from the team. It's totally changing the way that I work, so I'm excited to show you. Right now I'm only giving access to Dive Club listeners, so head to Dive Club Inflight to claim your spot. Okay, now onto the episode. I love the phrase frontier interface because on LinkedIn as a line item, you know, we've had similar job titles and yet you're working on fundamentally different types of products and user experiences. And like, every piece top to bottom, is unique.
- [07:26]
Ruz Madhavian
I think what that really means to be like a frontier interface, is one that you have no idea what the actual interaction model is going to look like historically. Like, the interaction model for a quote unquote computer is what ends up defining all the things that you can can do with it. Looking back like 60 years, that interaction model is usually a function of just what the input mechanism is. And, you know, is it the light pen originally, like just some point on the screen? Is it the cursor, which is like, you know, that plus the ability to click and some number of, like, clicks, or is it direct touch, like the iPhone? So, like, you know, all 10 of your fingers at the same time tracking, like, all 10 of those points, that ends up really defining the shape of what the computer can do. Because how you express your intent and at what cadence and like, resolution you can express your intent is going to define, like, what you can do the computer. In many ways, we can go, you know, potentially beyond just the physical. We can capture this intent that right now we have this image in our mind. How do we go about recreating that? Right now we, like, decompose that into thousands, tens of thousands of motor intents where we move our hands to effectively color a pixel on the screen at some level of abstraction. And we go through that process very manually up until finally we can then see some form of what originally was in our minds on the computer. If you have a neural interface, you don't have to do that necessarily, just like looking far down the horizon, because that intent is there much further up the stack than my hand is. You could potentially read that directly. So that's what I mean when I say frontier is we really don't know what the interaction model could look like, but the. The sort of device itself can potentially change that model. So it's just about exploring that whole space of ideas.
- [09:02]
Rid
So fast forward a little bit, then. What was the point where you realized that there was an opportunity for you to step into more of a traditional.
- [09:10]
Ruz Madhavian
I started doing just a ton of prototyping on what it would feel like to, you know, download an app, like a neuralink app, and connect to your implant and sort of calibrate a model for the first time. So, like, the. The earliest steps in here was just, like, getting some feel for. Yeah. What a human experience could be. And if this is like, North Star that we're all really, really stoked about. Spent a lot of time doing that purely on iOS, because that was the focus at the time. It felt like those early days at Apple again, where it was like you had just a truly a blank canvas, and then you're just going through the loop over and over again of, like, some concept that you have initially in your. Then, you know, 10 different iterations on that thing, and then ultimately something that you can hold in your hand and play with. So that was really cool. The flip side is, though, that because it was so early in the process, there were almost, like, too few constraints to actually do something really meaningful here. It was like you could do a lot of cool things, but ultimately, like, you're guessing a little bit too much on what the interaction model is going to be. And obviously, it is something that we're not designing for ourselves. We're designing for, you know, whoever that first participant is going to be. But it's going to be somebody with some sort of motor disability. So it's like they're going to have a very different sensory experience than we do. So it's really hard to actually try and directly empathize with that and actually optimize for that if you can't feel it.
- [10:17]
Rid
Can I drill in on that point for a second? Because that was something that was coming to mind while you were talking, where you're doing all these different explorations. Yeah. Maybe it's a little bit early, but the part that's so fascinating to me is, like, we talk a lot about empathy as designers, but this is a whole other level. You know, it's, like, very difficult. You're not just imagining a job that you've never had before, imagining a state of being that you've never really come close to before. What's that like?
- [10:46]
Ruz Madhavian
I would go so far as to say that it Is not something that you can imagine well enough for that to be useful. We have our own, I think, internal models of, like, what it would feel like to. Yeah, basically just try and imagine something without moving. That works well in the sense that you can just sit there, obviously, for something like a calibration task, when you're actually just trying to imagine moving, you can do that step of it. And I think you can do that step of it well, because you don't have to move. So you just kind of visually try and focus on something on the screen and get a feel for. Is there some thing happening in your mind when you're watching something on the screen where it feels like almost like mirror neurons. You know, I use that word loosely, but mirror neurons are firing and you feel like there's a connection between you and that thing. That step of it. You definitely can, I think, imagine because you can experience that. That's what we call open loop. So what that means is that if you imagine this cartoon loop of there's a human in there, the human has some intent in their mind. That intent goes down into some action, and then they observe the outcome of that action, and then that goes back into their brain. So it's like there's this loop of control. Open loop just means the loop is open. They are observing, but they don't actually see any outcome on the screen. They're just purely observing what's going on. Closed loop is when you can actually see an outcome, and then that obviously feeds back and closes the loop, the open loop stage. You can definitely sort of use your own experience and your own instincts to sort of create an experience that feels right for the task that you're designing for. But closed loop is extremely hard because you ultimately cannot experience the neural interface directly. So, for example, we have various ways of simulating these kinds of control. When I move the cursor, we have a model that will basically read the velocity of that movement. Movement and then convert it into some spike representation. So, like, what the neural representation could look like for this kind of movement. But if you're actually using that to optimize some aspect of the interface, it's going to fall flat because you're moving a real cursor. So you have all of these additional feedback mechanisms from where your hand is in space to the feeling of the friction that you feel when you move, like a physical mouse, or the friction on, like, a trackpad, to the pressure of, like, a click. You're not going to get any real sense of how that feels to Use that is, I think, actually representative of what it would feel like for somebody who doesn't have any of those things. And it's just looking at what's on screen. So the closed loop side of things is definitely, to answer your original question, I think, not something that you can empathize with directly, and I think you can weakly try, but it's never something that is a reliable indicator of what that feels like. But to even get there, we knew that they would have to connect to an implant for the first time. And beyond that, we knew that they would have to calibrate the model. So we would need something for the open loop side and we would need to have some best guess based on academia and also like other work that had been done in research as to what that closed loop thing should feel like. And then we also knew that we wanted them to actually use this, like, outside of a lab, outside of just any sessions with us. Like, the real magic is all going to happen outside when we're like, not in the room and they're just using their, their BCI to do stuff. We wanted this to be something that they could actually just like live with. And all of the work that goes into actually making something that's like, reliable, has a simple enough interface that they can actually use on literally, like their Mac to just like go around, click on stuff, do stuff with their computer again. So we had a really good sense now it's not really going to be an iOS thing anymore. It's going to be just like a Mac app that enables them to use their computer again and what the basic building blocks were going to be of that experience.
- [14:13]
Rid
I want to drill into some of those building blocks and maybe the first one we could talk about is just the cursor. Like, what are all of the things that you have to think through as someone that's designing this experience? Because my assumption is there's so many things that we take for granted and traditional, you know, B2B SaaS designers just get out of the box.
- [14:31]
Ruz Madhavian
I think the cursor is the focal point of that experience. Ultimately, it is like when we use a cursor, it is the focal point of your intent in, you know, two dimensions. So it encapsulates like, a lot. The thing that I think the B2B people get for free and just really anybody that's building an interface for a cursor gets for free, is that there's such a rich spectrum of sensory experience that just goes into using a cursor that I think, like at this point is so subconscious that you don't really think about it. But it's all of those additional senses that I talked about earlier around, like, you get friction, you get pressure, you get sound. What all of that means is having, like, a cursor that just feels like an extension of yourself and, like, feels like you're just focused on what the cursor's doing. You don't really think too much about what your hands are doing anymore. You don't have to think about any of these things when you're actually building an app. Like, that's just like, you know, you put a button on screen, you can add some sort of depth to that button and, you know, do the really nice things you do to make a button click feel great. But ultimately, when you do that, it's taking into account all of these other sensory mediums. So our challenge was like, our participants are not going to have that. We still want the cursor because it is the focal point of the experience to feel amazing. So then what's the right way to bring some form of that feedback to the cursor? There's a million ways to do this. There's a million ways to do this also that we still have not actually even prototyped or tried. But our first take on this is actually not the version of the cursor we have today. But our first take on this was some form of a regular cursor that you see when you connect to your implant. It does this wild, like, transition. So it'll, like, take over your macOS cursor, and it'll spin into this new matchable BCI cursor. And then the main question was, yeah, we have a model. We'll talk about that later that will take some intents into a probability of a click. So there'll be some probability of a left click, some probability of a right click. The moving is fine. Like when you're moving a cursor around, that is something that you can see, but it's the composition of moving and clicking. How do you make that feel? Not like a discrete event, so, like, you know, the click will just happen, but a continuous interaction. The reason you want that to feel continuous, by the way, is just because there is right now, just at the current stage, there is some latency to that click. It takes some amount of time for our model to ramp up its confidence that, oh, yeah, the user is actually the participant is trying to click here. And what that means is that if the model were perfect, it would be the same amount of latency every time. And actually the latency would just be, you know, whatever the sampling frequency of our implant is, which is like, let's say, 15 milliseconds right now. So every 15 milliseconds, the computer will get some new information about your neural activity. In a perfect world, right, that click would just happen the second you think about the click. So, like, within 15 milliseconds, and then you actually wouldn't really need a UI here because it's. The model's always right and the latency is basically zero. The click will happen on screen. The challenge is that there is some latency. Let's say it's like a hundred milliseconds, 200, 300 somewhere in that range. That probability that it assigns is sometimes it'll be a hundred milliseconds before you get to the right probability. Sometimes it'll be 300. And especially when we were thinking about what the first month of, like, the BCI journey would be like, there's also a very good chance that sometimes it'll just be wrong. So, like, it'll click when you didn't intend to click. That's obviously something we want to solve for, but we want the experience of that to be at least visible and much more importantly to be predictable. So when you are in this flow of, like, trying to click for the first time, that isn't this random thing that just after some amount of time, the click will happen on screen, but it's this continuous interaction, much like when you press your finger down on a track, that there's a continuous interaction there, and you can see the intensity of that ramp up over time. And our intuition was that that would just feel a lot better visually than just like the click happening would, with no feedback at all. So the first version of this used color to perform that click. We had, I think, this really nice blue for a left click and this orange for right click. And we would actually mix the two depending on the relative probability of each click. And then we also used depth and a slight perspective shift on the cursor. So as you were clicking, the cursor would sort of tilt inward just a little bit. The amount of tilt was driven by the probability of the left click. And the vibrancy of the blue was attached to that alongside, like, how much it filled in. It would start kind of at the base of the cursor, and then as it was tilting in, it would just sort of like grow or stretch towards the tip and go from this, like, a nice light blue to a Very bright blue. We can also use like the HDR parts of the display to like make that a little bit brighter. You don't want this. You're going to click a thousand times a day. So like you do want this to be subtle. We didn't want this like to feel like a weighty interaction. But the intuition was that having it be continuous and map directly to the output of the decoder is something that, one, let them just feel like a smooth, fluid interaction as opposed to just this like discrete thing. Two, this is always a challenge is when things aren't working. Give us some visibility, both them and us, some visibility into that. Like you don't want just random clicks happening on screen and you have no idea. And then the third thing is a more subtle point. But the interesting thing about a neural interface, this is true about all interfaces, but you are also learning how to use it. So when you are actually in a closed loop way, again, the last step of that flow is that you actually observe the output and then change your inputs based on the output. So because it's a control loop, seeing each stage of that pipeline, when it goes from 0.1 probability to point 2 to point 3 gives you a way to subconsciously, over time, learn to modulate your own behavior to click better. So if the model is just completely wrong, this is going to be a very, very hard thing for you to do. But if there are subtle inaccuracies in that output, there's kind of like a co adaptation that can happen over time. So we wanted to make sure that that's true of motion. For example, you can learn to move it more precisely over time. And that learning step, obviously imagine like if you had a new arm like today. Like the very first things you do would be like super weird and uncoordinated. But over time you would learn to really precisely modulate every single piece of that arm. But that is a challenge, right? It's like learning how to ride a bike. We wanted the same sort of continuity that you have in motion to apply to the interaction of clicking.
- [20:26]
Rid
What were some of the lessons that you were learning from participants that evolved the way you thought about a click? Like, where are you at now and what is the reason for that Delta?
- [20:35]
Ruz Madhavian
The biggest thing was actually that the space of interactions changed a lot. So we went from, let's say just one click or two clicks that we wanted for the first day to the richer set of all of the things you do on a computer. So scrolling is a huge thing that's very distinct from how you move a cursor, like how you actually scroll something. Of course, dragging and dragging is something you think should just work. A drag is something that you hold down over time. When we hold it, like there's some, again, some resistance that we actually feel right from the literal spring inside of these switches that is kind of our signal to keep applying pressure that isn't fully there with obviously something that offers like no resistance. And it's also like representationally, if you actually look at like the neural data, what you see is like sometimes what it looks like is actually more of a transition into like clicking down and then a transition back into clicking up. That may be a better model for a cursor that drags is it's almost like you get into the click down scene and you get into ClickUp. The flip side is that if you, you go down that route, every other click that isn't a drag gets a lot like slower. It's just like, because you now have to like transition between these two states. There may be a much better modeling approach here. And that's definitely something that a lot of people spend a lot of time thinking about. We wanted basically on the first day for you to be able to move and click. And because you can do that, we also wanted to at least have one other click in there. It's just a nice thing to have. Frankly, right click is like a pretty useful thing for just like popping over something, but not necessary. It was more just because we wanted to test out if you have multiple clicks in this mix, like, can you reliably switch between them. But over time then there's a far richer space of interactions that you need to just use your computer. And the more you stack in there, the less just having one simple model where it's just like you can see what the thing is and it's mixing between the two inside the cursor that just kind of fell apart. Our first participant, like one month or so into his journey, we had a very different signal quality than we did on the very first day. And so we actually took a step back and also supported a dwell interaction because we found that the clicks in this regime were actually harder to decode than the motion. Something about just that continuous ability to see every slight movement you make in a moving cursor made it easier for him to move the cursor than it did for him to actually perform a click. And so what that meant is we wanted this gradient of control that goes everywhere from just moving the cursor. And then if you can move the cursor. You can use motion as your signal for when to click. So that's like what the dwell would do. So if you slow it down and bring it to a stop, then you can basically hold that in a certain place and then perform a click that way. We wanted to support that, and then we also wanted to support layering in a left click, a right click, a drag, a scroll, a zoom, and all of the other interactions that you need to like, you know, as you progressively ramp up. We went from just having this really simple cursor that could show you just using color, these two clicks, to a more circular representation. So this is like a circle and a dot, like a reticle almost. This actually made it a little bit easier to see the difference in your click probability. So in this world, we didn't use color anymore, but we used just this, the outer radius of the circle. As you were targeting something, like with the intent to click, it would sort of scale down and focus in onto that thing, so it would collapse to a point. Basically, the more you ramp up your click probability, perhaps because our eyes are more sensitive to motion than color, this was an easier thing to actually use to perform a click. And it also supported just using dwell more naturally. The circular cursor was easier to actually see through than like a normal cursor is today. And because primarily you're just looking at it as your signal of, like, where the motion is, we could make it a little bit bigger without it obscuring stuff you actually want to click on. So we went from this more traditional sort of pointer style to this sort of circular style, and that opened also the door to going between different interaction modes more easily. So we ended up building a different mode switcher where you could sort of slam your cursor to the right side of the screen, and then it would go on Rails and you would use the same model that you used to move the cursor to pick a new mode and then come out. So you would shoot right, scroll up or down, shoot left. What's nice about this is that it was something that could be really fast, basically, if you learn to use it well, as opposed to something where you have to use your normal dwell, because then there's no time spent actually selecting that mode. And it's just one fluid motion of, like, going to the right to the sort of part of the screen that you learn, like, through this sort of neural muscle memory of, like, where this mode should be, and then popping out back left. And then once we had different modes. We had like, drag in there, we had scroll in there. We could easily change the behavior of, like, what this reticle is going to do and the visual appearance of, like, what it looks like to, like, make it obvious to you what it's going to do now that you're in this mode. A lot of that was, like, harder to do, I'd say, with just like the conventional cursor style and having everything exist in one space within the cursor, the trade off there. And this is definitely, as our models improve, something that I think we'll then go back and revisit is you do have to switch modes to actually switch the way in which you do your interaction. So, yeah, the world of BCI is like an interesting one where there's like, if the model is perfect, it's always going to do what you want. It's like an obvious statement, but in that world you really don't need these interfaces. But our bet is that the way in which you get to that world is like more of walking along this ladder than it is like trying to do everything at once upfront. We want basically you to be able to use your cursor to do all this stuff today. Even if the ways in which we get there are more of, like, design and engineering optimizations than they are what we think the perfect final solution is.
- [25:57]
Rid
Are there rungs on that ladder that you anticipate said differently? Maybe like specific barriers or unlocks in the capabilities that you're really excited to bring to this product?
- [26:09]
Ruz Madhavian
The rung on the ladder that I'm most excited about, if this is something that we can do, I think it's like, definitely an open question would be, can we delete the cursor? So if you think about what the cursor is, ultimately it is just like this focal point of your intent that you have to move around. 90% of the stuff you do is more just like, I want to interact with this thing at this point on my screen. Do you need like a cursor for that? If you can read that intent that I have that I want to interact with this point on my screen, probably not. I would say the cursor is like, amazing for direct manipulation tasks. So when you actually want to directly manipulate something, the cursor kind of fades away when you do that, which is really nice. And you can just like, it feels like I'm rotating this thing. It feels like I'm scaling it up. Whatever transformation I'm applying to it, it feels like the cursor isn't there anymore. It's just my hand interacting with this digital element on screen that is a key interaction model in this new world. But if that's that first step of just like the 90% of the time I'm just trying to interact with something, I don't think I need a cursor for that. So that's like one step of this ladder. I would say that is quite a few runs out on the way to that. I'm personally very excited about a world in which you don't need to switch modes. Some form of that first design that we had where you can see not just the click probabilities like inside of this cursor, but the way in which you want to do this interaction. So for example, if it is something like, I'll give you one cheesy example, like if I want to zoom in on something right visually, we still want to give you some form of continuous feedback about that zoom. The cursor, almost like meiosis, cellular division could explode into two finger points and then those could then be your visual anchor for how you're doing this zoom in interaction. That's another step of a ladder. How do you go beyond basically the current interaction model into something that that doesn't require switching modes to a new style of cursor? That mode switching is something that our model can actually learn. So we pick up on your intent, not in this case of just this crazy high order intent of I want to interact with this thing on my screen, but rather the specific I want to drag. Now that could feel like magic because it's just like when you want to drag, it just switches to the drag thing. You don't have to channel that intent through how we currently do it, which is we have a mode switcher and we also have a quick switch where you can use one of your clicks to quickly switch to your preferred mode in this specific app. So that gets us of course functionally like 90% of the way there. Because most of the time your second interaction is dragging. I'm curious what yours would be, but that's definitely what mine would also be because you can do it on a per app basis. Like if in Illustrator you're mostly zooming, right? You can easily remap that in functionally that gets you 90% of the way there. But it's still lacking, I think the actual magic that that is unique to something like a neural pointing device that these physical pointing devices just can't do, which is that we can read some form of our underlying intent. And I think the next like 10 rungs on this ladder are just bringing the actual raw intent into our experience. So the goal so far, I mean, has been just to enable the use of the cursor and to enable it with the same fidelity that we can use it. So I think the current record, our first participant set a wild bar. It was like nine and a half bps. BPS is just like a metric for the information that we can actually read. Really. You can just think of it as just like a score because you can use the same task that they use to measure the bps of like your cursor, like, so you can use it with a trackpad or like a mouse and just do what they do, which is like click on positions of the screen and it measures basically how quickly you can do that and how accurately you can do that. So yeah, first participant set this wild bar from that first point where the signal quality had like dropped off and he had to go back to using dwelling because he just loved using his neural cursor. He like used it every day for months and got insanely better at using it and went from I think something like 4bps, like 2 at, like, at the very bottom rung was like 2bps. And he just like walked his way up this ladder to I think nine and a half is where he ended up. For context, I can do about like 10, like that's my personal record I'm probably on, frankly, because I use a trackpad. But he ended up around nine and a half. And then this year our tenth participant did ten and a half. It was like 10.38, I think is the final number. The primary goal is to enable the use of the cursor at the same fidelity that we can use it. The reason this is like so exciting for us is like the cursor, beyond just being the focal point of your agency or whatever, is also how you use a computer. So if you think about what that entails, that entails, like we use computers to be productive. We use them to express ourselves, we use them to communicate with each other, and we use them to like, have fun. It's a pretty wide subset of the human experience that can be enabled with just a really sort of fluid cursor. So that is definitely like the goal. And that's kind of why we walk this space of different trade offs just to enable some form of cursor use, because we think that obviously there's this better thing out there that is in the back of our minds of where can we go from here? But the reason not to reinvent the wheel in this case is just because the wheel is how you get on the highway, which is this existing system. That's definitely the focus. But there are rungs, like, beyond that, then where it's just like there are things that are not just a projection onto a cursor that we use today that are very unique to pci.
- [31:30]
Rid
I'm interested in learning more about how the product experience itself has evolved over time. So as you're getting more and more usage from these participants, how are their behaviors shaping some of the experiments and just how you were even thinking about what the product needed to be?
- [31:46]
Ruz Madhavian
One fallback we built into the system in general was voice input. So just being able to say a simple command to recalibrate the model, to change some parameters of the model. Not something we envision as our final sort of experience, but it was great. In those early days when things were far less predictable, it was a really reliable fallback option for them to have. Our third participant actually had als. That means that they cannot speak, and that means that, of course, voice input is not something that's going to work well here. So this actually bled through to every edge of our interface in some sense, like. Like things that we had taken a voice input as, like, for granted for. We had to think a little bit harder about what a good fallback option would be. But I think the coolest thing that actually came out of this directly from our third participant was a feature that we called a parking spot. The cursor is always moving to some extent. This is one of those things in that bucket of, like, as models improve, this will never be the case, hopefully, but in the short term, you still want to use it. So some amount of motion is going to be there that the model's just kind of incorrectly guessing or is correlated to them actually just thinking when, for example, they're talking to somebody next to them or they're watching a movie on YouTube, there's just some amount of activity that our models are incorrectly labeling as motion. Here. You can solve this on the modeling side. Longer term effort, shorter term, it's like you just need some way to park the cursor. Our previous participants could just say, hey, turn the cursor off. And then it would just turn off. And then they could just say, turn it on to bring it back on again. For our third participant, obviously they couldn't do that. So we needed to think of some sort of. Of way to use the cursor to turn the cursor off and then use the cursor to bring it back on. We tried a few things here, but the final thing we landed on was this thing called the parking spot where they could just sort of yeet their cursor or shoot their cursor into the bottom right of the screen. And if they push it there, it'll park it, It'll like this little thing will pop out, it'll lock up the cursor, and then they can actually use a gesture within that surface to bring it back out. So we apply a bunch of little transformations in there to hold it still. We simulate gravity inside of it. They can still by pushing really, really hard or by actually going through a specific pattern, so like a dot, dot, dot, they can pull the cursor back out fully on their own. So that was a really cool one because we did that primarily for the third participant, but when we shipped that, our first two at the time were like, wait, this is like amazing. And then they actually were bigger for the first few weeks, like they were using it far more than our third part was. There's a lot you learn from like each individual participant that comes through and most of the time actually what they give you feedback that often always bleeds out to everybody else's experience as well.
- [34:15]
Rid
What does it mean to simulate gravity on the cursor like that?
- [34:18]
Ruz Madhavian
You could think of it as like, we just do the right math to work this out. But you can think of it as like, when the cursor goes inside this parking spot, it's almost like it falls into a hill. So like the cursor kind of rolls down here and then it's, it's sitting there. We can obviously tune the depth of that hill. So we can tune like, if it's a really steep valley or if it's like just a really flat, narrow hill. But what it means in practice is that they have to push harder, kind of roll it up there and like maintain momentum as they go up. The actual like output of the model at any given point is not a position, it's actually like a velocity. So it's actually like a nudge, so to speak. What that means in this context is like, if we add gravity to drag the cursor down there, they'll have to actually roll it out kind of manually. When I say that our third participant actually wasn't using it as much the first two weeks, it's because that version of it, the gravity based approach, didn't actually work for him, which is a really interesting Rabbit hole. He, like a lot of people with late stage als, he used an eye tracker as his primary way of communicating with the world and using a computer. What that meant is that when he was like watching a movie, for example, your eyes are all like all over the screen. It was a really interesting situation where when he was watching a movie, the cursor would just pop out when he's like, not even looking at it. No matter. Even with really, really strong gravity. But when he really focused his eye on the cursor and tried to move it inside the parking spot with that high gravity, it was really hard to push it out. So really interesting.
- [35:40]
Rid
Yeah, that's, that's a whole level of depth that you have to think through that is really interesting.
- [35:45]
Ruz Madhavian
The solution there actually ended up being that it supports like two modes where you can use it with gravity. And that actually, for our other two participants, it worked so well that we found other issues where like, one person's like, hey, I parked the cursor and I was talking to somebody and like my screen went to sleep. So like he was having like a whole 30 minute conversation where the cursor doesn't move. And on Mac OS, if the cursor doesn't move, your system's like, oh, you're not doing anything and it'll dim the display and then shut it off. So in their case, it worked too well. In his case, it like, yeah, it literally did not do with gravity, at least what you'd expect at all, where when he looked at it, he couldn't move it. And when he looked away, he could have huge velocities as he's like watching something. So we ended up going with a gesture for him and that worked super well. So he could just do like an up, down, left, right. These little dots would light up, it would follow them in order and then he would just like pull the cursor out that way.
- [36:34]
Rid
It's fascinating to listen to you talk because you're obsessing over the smallest pieces of these interactions. I mean, observing and thinking about a cursor from every possible angle, all the different ways that we could make this possible. And the end goal is to basically delete it. To delete all of the work that you've done to an extent. You know, like, that's a really good, interesting tension that not many people get to operate in.
- [36:59]
Ruz Madhavian
I think ultimately the goal is to just build an incredible experience and to do that in, like every step of the way. So, like, if the way in which we get to a World where we could, let's say, like, delete the cursor, like, whatever that means. Our hunch is that getting there is going to require like a lot of data. It's going to require a much more robust understanding of how these mechanics work in that world.
- [37:20]
Rid
It starts to get very sci fi to even think about. But the measuring stick is no longer what a quote unquote normal person can do on a computer. You know, you've blown the roof off of what is possible in terms of interaction with computers at a very high level. Right. Like, thinking about how quickly I'm able to do something on a trackpad will feel archaic in that world.
- [37:40]
Ruz Madhavian
Right now, the interaction model is again, very similar to the ones that we use, but there's no, like, reason it has to be. So I think there will be new interaction models in this world that are, yeah, just very different than having to use our physical hands to like, articulate something. I don't know if it would be like archaic, but I think it would just be very different. Like, it'll just be hopefully a lot more natural and hopefully a lot less indirect.
- [38:03]
Rid
Anything specific that you find yourself thinking about. As someone that probably spends more time pondering where this specific world is heading than the typical person listening, for instance.
- [38:14]
Ruz Madhavian
I think there are a few. For me, the caveat being that this is definitely just like, like the stuff that I'm excited about and not necessarily stuff we're working on. And I also think like, the fun thing is like, at neuralink you'll get a million different answers based on who you talk to. But I think for me, it really, a lot of it goes back to that study from 15 years ago that first sparked my interest in this field, which is like, I think the richest things in my mind are visual imagery. I also think that like, both like, as a software designer and as like a wannabe filmmaker when I was younger, like, that's where so much of the work, so much of the magic of a computer is. And also so much of the tedium of a computer is, is in the. This process of taking an image in your mind and spending a very long time, whether it's in software, whether it's in After Effects or Final Cut, to articulate that into this visual thing that it feels like people just understand more easily. A picture is worth a thousand words. Basically. That side of things is I think what I'm most excited about, you know, looking out into the very distant future, what that would mean. We talk a lot about, you know, bridging this Empathy gap between folks. But, like, that feels like a much more tangible version of that where I can show you how I'm feeling instead of telling how I'm feeling. And then also, just for all of the stuff that people want to make in the world, there's this rich interior space that I think everybody has and everybody shares. I always think it's, like, funny when some people tell me, like, oh, like, no, I'm just, like, not very creative. Do you dream at night? And I think, like, if so, like that, like, look what your mind is capable of making and generating. I think that is a hugely enabling thing in the future where you can sit down at a computer, basically, and in this world where a model on that computer can act on behalf of your intent in, let's say, you know, far down the line, a frame. So what we today call vibe coding. You know, in the future, like, the time that it takes for a model to actually produce an artifact that you described will, I think, come down to a frame. And the flip side of that is, then what is the bottleneck here? It's you actually articulating what it is that you want in all domains in the arts and in work. This will be a very exciting and wild experience where you can just sit down at a computer and basically daydream with it, and things will pop up on screen that are a direct sort of extension of what you have in your head. And what would that look like if that came down to just being one frame? I truly don't know. But I think that that's a really, really exciting world to be in. We are working on something called blindsight. That is the earliest rung in this ladder, which is that for people who no longer have vision, some way for them to see again in the world. What this would mean is our implant sits in the part of the brain not for motor movement, which is where our current implant is, but for. For vision. And we can actually read from, like, let's say, a pair of glasses that they wear what they're seeing, and then create the right stimulation pattern in that part of the brain to try to recreate some form of that image in their mind. Obviously, this is incredible, but also it is like a truly wild design space, because the first versions of this are going to be, to use a really crude metaphor, they're going to be like Atari, as opposed to being like, you know, a PlayStation 5. So the visual fidelity that we can actually create in terms of just the number of electrodes that are in this part of the brain is going to be very small relative to how rich visual information actually is. So then that's its own completely distinct design space of, like, how do you recreate an image that is true to life, quote unquote, in this domain? What features of that image are like, the right ones to highlight here? What knobs do you want to give people, basically, because they should have some form of control over how they see the world? World that can be something that's just like sci fi and cool. It's like being able to zoom into things. Right. But it could also be something far more nuanced than that in terms of how, you know, you want to, for example, just to borrow more metaphors like dithering. If you're familiar with dithering, it's like one way of, in the 80s when we were in this regime of, like, low pixel counts to try and recreate features that we perceive through, you know, algorithms that apply that basically are artifacts of our perception, and use this, like, lower sort of resolution space to still try and create some sense of, for example, like, shadows and some sense of, like, texture in this world. So, yeah, what would dithering look like in this domain?
- [42:17]
Rid
I've heard a lot on this show, but that's one of the most interesting design opportunity spaces that I've ever thought of. Like, it just didn't exist in my brain until you shared that. And it's amazing and compelling, not just because of its novelty, but also just the impact that you are having is really incredible. And I know I mentioned this just a little bit before we started recording, but. But I was reading some of the participant stories and I was reading how Nolan, one of the first participants, got, you know, he's going back to school and he got a job on the Internet. And it's just amazing. Like, it moved me deeply. And so what you all are doing is just about as inspiring as it gets.
- [42:55]
Ruz Madhavian
I fully agree. I think this is the stuff that people get out of bed for on our team and definitely throughout the company. We are so privileged to be in this world now where we have 10 plus folks who are actually using the thing we call them, like the neural knots, because, like astronauts, it's a completely new space that we're exploring together. They've obviously done, like, back to that point of, like, what the cursor used to look like and how things work. Now things have changed dramatically once they could actually tell us how things feel and what pieces of that, like, we actually do need to optimize what pieces of that don't really matter, but ultimately, like, yeah, just the impact it can make on their lives has been super inspiring for all of us.
- [43:29]
Rid
Let's say that somebody is listening and they're inspired by this journey. They want to be a part of it. Y' all have opened up a second design engineer type role, so could you share a little bit more about, like, what's that role going to be like? I'm sure someone is listening and they're like, yeah, that could be really cool. But Also, I don't 100% get what that would look like.
- [43:49]
Ruz Madhavian
I mean, the role would be a lot of what I think I've talked about today, just a lot more of that. We're at this phase now where we have, yeah, 10 plus people that actively use the thing every day. They use it for like, hundreds of hours a week collectively. And one piece of that is, of course, just, like, actually just continuing to make that a great experience for them. We generally, because we are such a small team, we are super focused about what pieces of that experience. Like, we spend the most time on making great, and we have a very high bar for, like, what grade is, and we want to maintain that across that whole experience. So one piece of it is literally just this thing that people use literally every day. Making that amazing and continuing to make that amazing. A second piece of it is all of the rungs in the ladder that we have have not explored yet. The process of doing that now that we have folks who can actually try stuff and tell us is really just like, it's something you have to kind of go end to end on, which means that there may be some initial idea or hunch you have about how things might work, what you might call a hypothesis. There's a lot of actual work in between that and making it something that somebody can actually use. There's obviously a lot of, like, design work there, and there's also just like, the engineering work of turning it into something that they can use and try out for the first time. The nuance is also that that that first experience they have with it is not necessarily going to be the most informative if it's something they actually actively have to use and try. What's awesome about our participants is like, they're so down to just, like, take something and run with it and try it. And they will give you such great feedback after using something for like a day versus, like a month sometimes. It's also not just a prototype to actually get a meaningful signal on. Is this a direction we want to go in like, is this a feature that will work? You kind of need to go beyond. Yeah, just something that will work in a hacky way. And one session with a participant and it turned it into something that they can live with for a month. The vast majority of those things also, like, will not pan out is the reality of it. So right now, we're in this phase where there's both a core set of features that we know that we want to make great because we want that point 1, to exist in the world, and also a lot of blue sky around what this could be. That requires an ability both to just, like, kind of go end to end in both design and engineering. That's the role, but also be somewhat comfortable with things. It's the fact that it's experimental, meaning that, like, there will be a lot of uncertainty into, like, will this thing actually ship? And because you want somebody to live with it, there's also a lot of work that needs to go into actually shipping it in some form so they can try.
- [46:07]
Rid
The blue sky piece is really interesting to me because even just listening to you talk now for a while, I mean, it's so clear what you're doing is the complete opposite of looking on Mobin and trying to piece together the right pieces, the right things that already exist in the world to, like, figure out a solution. You know, like, you are really working from first principles in the truest sense. My question then is, what are the signals that you would even look for in a candidate that would get you to a confidence level where you're like, yeah, they actually have what it takes to move the needle and help us find some of those next rungs on the ladder.
- [46:40]
Ruz Madhavian
The thing we optimize most for is feeling when it comes to what it's actually like to use the interface. So as much as we try to be principled about what the building block should be for space that we cannot experience ourselves, I think there's just a general ambition to just make something that's, like, really awesome, inspiring. I think we're all, yeah, obviously, like, indirectly so inspired by our participants, and we do want what it feels like for them to use these, this magical new toy to be amazing. Anybody who has a clear desire to do those things, that's usually what that means in practice is even for something that you know that you're probably going to delete, there's like, a lot of extra work that goes into just making that amazing. The equivalent of kind of like the polishing phase and the tuning phase that you'll have At a lot of other companies, when something is actually, actually going to ship in a calendar year, we try to put as much of that as we can into just an experiment because a person will use it. And also this is where the prototyping ability, like the engineering side of the role is so key. The stack here is very wide. I think that the tightest interplay is really between the machine learning side of the work and the actual interface side of the work. Because the machine learning side, ultimately some form of that, at least right now, is, well, we need to figure out a way to label some something. So like the label is ultimately what on some level is very tied to what they see on screen, how well that label captures the real intent here, like what they were really thinking at that time. What the models can do defines what the interface can do. And what the interface does in some ways heavily biases what the models can do. So there's kind of like a very direct relationship between those two things especially that makes the primary bottleneck here just like, can you build out something to try these things at times end, because a lot of stuff is just the offline data, meaning like trying to collect data and then just run your analysis offline generally is a dead end for anything on the UI side. So it means you have to make something that is actually fully interactive to actually validate this well and potentially also work with folks on the modeling side. If you have a really wild new idea you want to try that involves a lot. It involves a task to collect the data, it involves some final interface so that they can control it. And again it involves ideally something that they can live with with independently and give you a much richer signal on how well it's going to work. So people that yeah, are excited about doing all of that and also are excited about making each individual experiment you run as amazing as it can be, such that we have confidence that when we hit a dead end, it's not just because it was for example, like an mvp. I generally think that like, the mentality of like an MVP can sometimes be quite self defeating because you've basically done the absolute minimum you possibly could have done and gotten like a pretty mid signal on something. The shots that we try to shoot, we try to like shoot them as best we can because the, the results that come out of that very much dictate ultimately the direction we're going to go in.
- [49:25]
Rid
Is there an example of a hunch that led to some experimentation process that you could talk through from your own experience that would help people Get a little bit better of a sense of what it is like day in the life of, of someone working on neuralink. And maybe if there are also examples of what it looks like to sweat the details throughout that experiment, that would be great too.
- [49:47]
Ruz Madhavian
One good example here is what we currently call the body mapping task. The structure of this task right now is there's an actual, like 3D arm that we render and we have a whole rendering pipeline to draw this 3D arm. The approach there means that we can draw your focus to very specific pieces of it and then combine it with the actual guidance that we tell you with like visual indicators in the guidance that we give you as well. Well, so what does this task do? This is the very first thing. When we sort of spec this out, this would be the very first thing that a participant sits down and does. And before they actually go into the part of the experience where they sort of calibrate their implant for the very first time and get to a moving cursor, they just explore this like, wide space of like, you are now going to imagine moving your arm again, this arm that like you currently cannot move or you have very limited residual motion for, and get a feel for just like, like what works for them. What kind of motion still feels intuitive to them even though they can't move it? Well, which ones don't? And then also get not a feel, but a signal for what motions can we actually read really well. So when they imagine doing this, what does the burst of activity look like and which motions seem to be better here and weaker and sort of compare between the two. Pick one from this wide set of moving your arm up, down, left, right, moving your wrist up, down, left, right, and four other things that you can, could presumably use to move a cursor, getting some sense of the overlap between what feels good for them and also what we can actually decode. The hunch here was primarily that, like, seeing that arm on screen would make that first experience far more natural than us just telling them, hey, like, try doing this. It would give them something to basically like look at and like cross reference. And then the details here were basically like, well, okay, well how do we want to do this one? We could try and pre render all of these motions. That will probably work. But what if we could actually just decode that arm? Then why take this arm and then try to go down to the cursor if they're already back in this world of just looking at that arm? In some ways that's a far more natural thing. To do than trying to think about moving your arm to then move a cursor. So the real hunch here was if we can do this, can we also. Can they also puppet this arm arm basically to do that. That was an extraordinary amount of engineering work. Because we can't just use a pre rendered animation anymore. We need to actually render like a 3D arm. We need to mesh that out. We need to have some shading pipeline so that we can make that not look goofy. There's obviously a huge uncanny Valley in 3D arm on screen. So we wanted that to just look basically like have the minimum amount of detail for you to get a feel for what the gesture actually is without all of the redundant information about what an arm is. And we wanted to also feel natural. So we didn't want to a skeleton, basically. You could also do something where you see all the bones. But like that didn't feel like it was going to be a really cool experience to us. And then finally, like one extra bit of magic that we absolutely did not need to do. But we thought it'd be really cool if now you have this arm. And we also need to give you some text to like tell you what the kind of motion is. So for example, like if you're going to do like a squeeze, like we need to also like pop up something that says, hey, squeeze this arm. It'd be really cool if, because that's on screen, so your eye is going to already be looking there. If the sort of text itself could embody that motion alongside the arm just to like reinforce that whole thing. Then we did these wild text animations where like when you do the squeeze, like the squeeze itself kind of like focuses in, so everything in the background sort of recedes. And then the squeeze actually like we do kinetic type so it like fully squeezes in alongside the motion itself. So that's an example of something we totally did not need to do. It's just cool. The hunch here was actually that, that if this works, basically if there is this mind body connection that we can induce by then just having this arm on screen, having the perspective of it feel right relative to where they are, that maybe we can actually have this simple imaginary space where in their head they're just like, okay, now I'm squeezing the arm. I'm not thinking about all these joints, I'm not thinking about moving each individual finger. I'm just like, oh, I'm squeezing the arm. And then we would be able to decode that. And if we can decode that that's a really rich interaction space. Right. I'm not thinking about moving a cursor anymore anymore. I'm just thinking about moving my arm. And obviously, because this is something I could do before, that's also, like, really, really cool. It's not a cursor, but it's a. It's a full arm. This arm that I no longer have the ability to move, it did not pan out in the sense that we could not on the first week. Like, we tried actually decoding this, and it was just far worse than decoding the cursor, which makes sense intuitively in the sense that it's a much bigger space. But ultimately, like, we just, like, didn't know. So there was, like, a hunch it didn't work. That doesn't mean that it won't work in the future. But at the time, it was just a bet on what if we could decode the full arm, what would it take to get there? And then how do we unify that with what we already know we want to do?
- [54:29]
Rid
It's funny. I went ahead and read the job description for the second designer that you're trying to bring on, and there was a line that I pasted into my notes because I wanted to bring it up and look for an example, which was, they need the ambition to build something that feels magical, beyond the local minima of that which works. And I think you pretty much already answered that question with the squeezing of the text. Totally unnecessary, but also, like, what a fun way to bring delight. And the fact that you did that into an experiment is the perfect example that I was looking for of going above and beyond, even when you have no idea if this is technically even possible, at least in the short run. Well, Ruse, this has been amazing. Thank you for creating one of the most unique episodes, probably the most unique episode that I ever had. Like, just hearing the way that you think about this set of interactions and design opportunities is truly fascinating. Really appreciate you coming on and sharing with us today.
- [55:27]
Ruz Madhavian
Yeah, thanks for having me. This was a ton of fun.
- [55:30]
Rid
Before I let you go, I want to take just one minute to run you through my favorite products, because I'm constantly asked, what's in my stack. Framer is how I build websites. Genway is how I do research. Granola is how I take notes during crit. Jitter is how I animate my designs. Lovable is how I build my ideas in code. Mobin is how I find design inspiration. Paper is how I design like a creative. And Raycast is my shortcut every step of the way. Now, I've hand selected these companies so that I can do these episodes full time. So by far the number one way to support the show is to check them out. You can find the full list at Dive Club Partners.