Story: One Million Checkboxes - The Viral Game That Defied Convention

Summary

CoRecursive: Coding Stories – Episode Summary Title: Story: One Million Checkboxes - The Viral Game That Defied Convention
Host: Adam Gordon Bell
Guest: Nolan Royalty
Release Date: February 3, 2025

---

Introduction

In this captivating episode of CoRecursive: Coding Stories, host Adam Gordon Bell delves into the intriguing journey of Nolan Royalty, a software developer renowned for his unconventional and viral web creations. The episode, titled "One Million Checkboxes - The Viral Game That Defied Convention," explores Nolan's early fascination with coding mischief, his evolution into game development, and the unexpected viral success of his project, One Million Checkboxes.

---

Early Mischief and Tinkering

Nolan's passion for pushing technological boundaries traces back to his high school days. Adam reminisces about Nolan's childhood exploits, highlighting his playful yet impactful tinkering with computer systems.

[02:03] Nolan Royalty: "There’s some through line from that to plenty of the stuff that I do now in terms of, like, how can I misuse this thing in a way that's kind of fun?"

Nolan recounts a memorable incident involving the Student Wide Information System at boarding school, where he and his friend Brody engaged in a playful battle of email rule manipulation. Their antics, intended as harmless fun, inadvertently caused the school’s email server to crash for days. Instead of punishment, Nolan was met with laughter and encouragement from the dorm supervisor, an experience that profoundly influenced his future endeavors.

[05:18] Nolan Royalty: "That's hilarious. That's so funny."

---

Pursuit of Game Development

Transitioning from his mischievous beginnings, Nolan ventured into game development after accumulating enough resources from his managerial role at Jane Street, a trading firm. His approach was methodical yet experimental, focusing on creating numerous small games to hone his skills and gather feedback.

[06:16] Nolan Royalty: "One common failure mode for people who try to make games is that they're like, oh, I really love whatever. So I'm going to build a big MMO and don't think about how that takes years for a team of hundreds."

Nolan's strategy emphasized rapid development and iterative learning, aiming to understand what works by completing and testing each game. This philosophy led him to the Recurse Center, a coding retreat that became pivotal in his creative process.

---

Joining the Recurse Center

At the Recurse Center, Nolan immersed himself in a community of like-minded individuals dedicated to intensive coding and collaboration. Here, he participated in workshops and collaborative projects, further refining his skills and expanding his creative horizons.

[08:38] Adam Gordon Bell: "The Recurse center wasn't just about solo coding marathons either. They had workshops."

One notable project was a Mastermind game developed entirely using SQL queries, showcasing Nolan's knack for innovative problem-solving and his ability to make complex tasks enjoyable.

---

Building and Launching One Million Checkboxes

The genesis of One Million Checkboxes began as a spontaneous collaboration between Nolan and his friend Neil. The concept was deceptively simple: a website laden with a million checkboxes where any action on one checkbox would reflect universally for all users.

[20:39] Adam Gordon Bell: "The pitch to Neil for 1 million checkboxes is simple. It's a website with a million checkboxes on them. If you check one, it's checked for everybody else. That's it."

Driven by a desire to create something quickly and gratifying his urge to ship a project, Nolan developed the site over a weekend. Launching on a Wednesday, the site quickly garnered unexpected viral attention, amassing around half a million unique visitors and 50 million checkbox interactions within the first few days.

[25:20] Nolan Royalty: "I think there's something like 500,000 unique IPs there. It's hard to know sometimes people are going to be like, many users are behind the same static IP at a school or whatever."

The simplicity of the idea belied the technical challenges Nolan faced, particularly in scaling the application to handle the immense load while maintaining real-time responsiveness.

---

Community and Collaboration

The unexpected popularity of One Million Checkboxes fostered a vibrant community centered around the site. Nolan discovered a Discord server populated by enthusiastic teenagers collaborating to draw intricate patterns and images using the checkboxes, effectively treating the site as a massive, interactive canvas.

[42:38] Adam Gordon Bell: "Nolan created 1 million checkboxes without a fixed grid because he wanted to stop people from drawing offensive images."

These community-driven efforts demonstrated the site's potential as a platform for collective creativity and playful subversion, aligning with Nolan's original vision of constrained chaos.

---

Technical Challenges and Solutions

Managing a million checkboxes in real-time presented significant scaling hurdles. Nolan leveraged Redis for efficient state management, utilizing its bit set capabilities and Lua scripting for atomic operations. Despite these strategies, he encountered unforeseen issues such as excessive bandwidth usage and data validation errors, which threatened the site's stability.

[28:04] Nolan Royalty: "Redis has a count of the number of boxes checked, stored separately, just so that we don't have to iterate through a million things to determine how many are checked."

When faced with escalating bandwidth costs, Nolan collaborated with a friend at Facebook to optimize the server performance, transitioning from a Python Flask application to a more efficient Go-based system. This rewrite significantly reduced CPU usage and improved overall performance, ensuring the site's continued operation amidst its viral popularity.

[33:14] Adam Gordon Bell: "Part of the big rewrite was moving from an untyped Python flask app to go."

Despite the technical prowess, Nolan discovered a subtle violation in his data validation, allowing persistent, pattern-based interactions that were initially mistaken for bugs but later identified as sophisticated bot-driven behaviors orchestrated by the community.

[39:42] Nolan Royalty: "This isn't hacked. This isn't. I have a bug. This is like, someone who is being very persistent about writing a weird message into my database."

---

Reflection and Impact

The success of One Million Checkboxes had a transformative impact on Nolan, validating his approach to constrained, interactive design and reinforcing his belief in creating platforms that encourage playful engagement within defined boundaries.

[46:58] Nolan Royalty: "This felt like it really validated this thesis that I had had from back when I made Stranger Video."

Nolan's journey from high school digital mischief to creating viral web phenomena underscores the profound influence of early experiences on one's creative and professional trajectory. The episode concludes with Nolan reflecting on the emotional and professional fulfillment derived from seeing his creations bring joy and foster community interactions.

[48:45] Nolan Royalty: "Providing people a very low stakes way to troll, where the worst thing you can do is uncheck a checkbox is great, right?"

---

Conclusion

Adam Gordon Bell wraps up the episode by celebrating Nolan Royalty's innovative spirit and his ability to transform simple ideas into engaging, viral web experiences. Nolan's story exemplifies the powerful intersection of creativity, technical expertise, and community engagement in the realm of software development.

Adam Gordon Bell: "Thank you, Nolan, for pushing the boundaries. You're clearly one of the world's experts at twisting things beyond their intended use for our amusement."

Listeners are encouraged to explore Nolan's projects at eieio Games and join his growing community to stay updated on his latest creations.

---

Notable Quotes:

• Nolan Royalty [02:03]: "There’s some through line from that to plenty of the stuff that I do now in terms of, like, how can I misuse this thing in a way that's kind of fun?"
• Nolan Royalty [05:18]: "That's hilarious. That's so funny."
• Nolan Royalty [46:58]: "This felt like it really validated this thesis that I had had from back when I made Stranger Video."
• Adam Gordon Bell [25:20]: "50 million checkboxes. Half a million people. These numbers are wild, right?"

---

This episode offers a deep dive into the innovative and sometimes chaotic world of viral web development, showcasing how playful experimentation can lead to unexpected success and meaningful community engagement.

Transcript

[Adam Gordon Bell]

Hello and welcome to Co Recursive. I'm Adam Gordon Bell. Back in high school, during the dial up era, my family had a computer and so did most of my friends. And we all connected through this chat site that was popular in our little circle. One of my friends had this paper with HTML, like handwritten on it in pencil, and if you typed it into the chat app correctly, which was hard because it wasn't clear what all the symbols were, it would launch a bunch of pop ups and it would crash your browser. It was like this secret hack where you could take down the chat site and it was handed around from person to person. Back then, nothing on the computer felt very real or important. It was just like a fun playground for teenage mischief. There was once a fight that broke out at school over something that someone said in chat, but. But mostly it was just. It was just playing and a little bit of trolling, occasionally crashing people's browsers. But now the Internet is just life, right? And when the Internet became serious business, when it became part of reality, did we lose that fun? Today's show is with Nolan Royalty. We'll dive into his wild encounters with the Internet's chaos, how his creation sometimes got misused, and also his worry about being hacked and data breaches.

[Nolan Royalty]

This was happening on Monday evening, and my girlfriend had made dinner. I thought that I had a data validation error that let people stuff strings into this midset because I hadn't really even thought about what it was that my script did at this point. And I had kind of gotten re obsessed with what is going on here, right as dinner finished. And she was like, can we sit down and eat? You've been very busy recently. And I was panicking about like, There are these URLs. I have been hacked. I have been hacked.

[Adam Gordon Bell]

But before we dive into the hack, let's rewind to a moment in his past that reveals something crucial about him. A moment where he pushed the boundaries, expected punishment, but got something entirely different. Something that might just explain why he's drawn to the chaos of the Internet in the first place. Nolan calls himself a deranged game builder. He dropped out of college. He spent some time at aws, and then Jane Street, a trading firm. But today is not about that. Right. It's about the deranged game building part of Nolan's career and his exploration of the bounds of computer systems and what you can do. And it all started back when he was in boarding school where they had something called the Student Wide Information System. It was like email and it was chat.

[Nolan Royalty]

But point being, it was an important part of student life. And also, it was the way that you turned in your homework. It had a weirdly powerful mail rule system. So you could write rules in the same way that in Gmail you can filter your mail or whatever. You could write rules that said, when I get a message from Richard, apply the Richard tag. The rules could take a lot of actions, and in particular, they could send emails. And one other quirk. The reason that I kind of got into doing the thing that I ended up doing was that when you got a new email, it made a pretty loud beep. And I had this friend Brody, who always had headphones on during our study hall. We had study hall for two hours a night, Sunday through Thursday. And whenever he got an email, he would jump because the noise was loud and he had headphones on. I thought that was funny. So it was kind of fun to send him a message and see him jump.

[Adam Gordon Bell]

Then Nolan started tinkering with the email's rule system.

[Nolan Royalty]

So what I did is I wrote a rule that said, whenever I get a message from myself with an empty subject, one, send a message to my friend Brody. Two, delete the email that I just got from myself. And three, send myself two more emails that also have empty subjects. So then I would enable this rule, and I would send myself a message, and then it would send Brody 2 and then 4 and then 8 and then 16, and pretty quickly a lot of messages, and he would get like, ding, ding, ding, ding, ding, right? And jump out of his chair, fall over. And it was very funny to me.

[Adam Gordon Bell]

Brody fought back, setting up rules that would delete Nolan's emails before they could cause a beep. And then he made rules that would send things back. Nolan tweaked his emails to slip past Brody's defenses. And so the nerdiest battle began. Each of them writing rules to outsmart the other.

[Nolan Royalty]

And that was all totally fine and fun. And we did this for a week or something. And then one day, both of us, just because whatever our defenses were good enough, forgot to disable our rules. So normally, you would enable this for three minutes and send a lot of emails, and then it was fine. But instead, we just totally forgot, and we went to bed, and we woke up the next day, and the school email server was done.

[Adam Gordon Bell]

No one could send emails. They got the system back up briefly, but the rules were still active, causing it to crash repeatedly over a couple days. Eventually, they traced the problem to the rules and informed the dorm supervisor that Nolan and Brody's battle was behind the email chaos.

[Nolan Royalty]

And he came to me and was like, what's going on here? And I explained and he was like, that's hilarious. That's so funny. And he made me a T shirt. But I was kind of proud of finding this thing, and I think it certainly made me. I don't know, there's like some through line from that to plenty of the stuff that I do now in terms of, like, how can I misuse this thing in a way that's kind of fun? And so being encouraged then I think was pretty nice.

[Adam Gordon Bell]

Imagine expecting detention, expulsion even, and instead being met with laughter and encouragement. This supervisor didn't just forgive Nolan's digital mischief, he seemed to celebrate it. That moment was a hint of what was to come for Nolan. The New York Times interviews, the media buzz, the Washington Post, the speaking gigs, they all trace back to his knack for bending things outside their normal use. So, fast forward, Nolan is an engineer, then a manager at Jane street, the quant firm. He builds up a bankroll that's large enough to quit and do something that he's always wanted to do, make video games.

[Nolan Royalty]

And I had this big list I had spent all of 2022. Every time I had a game idea, I would, like, put it in the list. And so I had this big list of ideas that I thought I would like be working from for a while. The first thing I made was Tetris, just to kind of like get an understanding of how to put objects on the screen in Godot. And a thing I decided pretty early was that the goal was to build lots and lots and lots of small games that I understood that one common failure mode for people who try to make games is that they're like, oh, I really love whatever. This is a silly example, probably, but like World of Warcraft. So I'm going to build a big mmo and don't think about how that takes years for a team of hundreds. And it's probably not a good starter project. And you learn so much by finishing a game and seeing people play it or playtest it. And the thing you want to do is get a lot of reps in of that. So it's like, I'm going to make games that take me a week or two as fast as I can and blog about them. And there's a sign right above me that says the plan. One, make games as fast as you can. Two, blog a lot that I put on my wall to kind of, I don't know, give myself some goal that.

[Adam Gordon Bell]

Sign is still there. And so Nolan builds games, but it's harder than he thought.

[Nolan Royalty]

It's actually really hard to just go to your little, I don't know, second bedroom and sit at the computer all day and write code and not talk to anybody about it. And you can tell your friends about it, but I don't know, your friends aren't working on it with you and you probably can't talk to them as much as you would talk to your coworkers about what you're working about.

[Adam Gordon Bell]

So Nolan decided to apply to the Recur Center, a place that's kind of like a writer's retreat, but for coding.

[Nolan Royalty]

You go there and you Write code for 12 weeks or 6 weeks if you do a half batch. And when you join, there are a bunch of people who have been there for six weeks to kind of show you the ropes. Then they leave six weeks in, and now you're showing people the ropes. The goal is just write as much code as you can every day according to whatever interests you. So when you apply, I think they do a basic coding test thing, but mostly they're looking for can you write code and do you have an idea of what you want to work on in a very self directed way. So I applied and I told them I was going to make eight games in 12 weeks and I showed up and I started making games.

[Adam Gordon Bell]

The Recurse center wasn't just about solo coding marathons either. They had workshops. Nolan joined one on pair programming where his task was to create a small game.

[Nolan Royalty]

We were solving. Writing a solver for the game. Mastermind.

[Adam Gordon Bell]

Yeah, with the mines.

[Nolan Royalty]

No, not Minesweeper.

[Adam Gordon Bell]

Oh yeah. Mastermind is like the colors.

[Nolan Royalty]

Yeah, the colors. So it's actually, it's very much wordle. Like wordle is Mastermind, but there's only one player and you're guessing words. And Mastermind is a two player version of wordle where you're guessing the colors of some pegs that somebody has hidden from you, but it has kind of similar mechanics. But they told us we were going to write a solver for Mastermind or a backend for Mastermind or something. And I said that I wanted to work in SQL and they matched me with someone else that could work in SQL. And so we wrote a Mastermind thing that worked entirely as a big SQL query. And it was really fun. It was a huge blast. And we showed it to a bunch of people and everyone was like, yes, this is funny, we're glad that you did this. And that was very exciting to me.

[Adam Gordon Bell]

Nolan got inspired to create a game using OpenSearch.

[Nolan Royalty]

That is the way that your computer knows how to search a specific site. So if you've gotten to like, you've typed Wikipedia into your search bar and hit tab and then typed a query and it's like searched directly over Wikipedia, right? The way that works is that under the hood, when your computer visits Wikipedia, Wikipedia serves it a document that says, if the user wants to search my site, hit this endpoint. And the spec happens to include this crazy thing where optionally, browsers can choose to support search suggestions. So just like when you type into the Google search bar, it guesses what you want to search for. Your website can offer those suggestions. So you say, hey, whenever the user types a new character, hit this endpoint and I'll return some ideas of what they might be searching for along with URLs to directly visit that page. So the whole bit was that you can return whatever you want from that. So why don't you return wordle answers based on the wordle guess that they're typing into the URL bar, if that makes sense.

[Adam Gordon Bell]

So Nolan took this idea and created a wordle inspired project. Then he wrote a blog post to break down how it all worked.

[Nolan Royalty]

It's nice to have a website where you can point to everything you've made, but I think it's also just a really nice way to codify what it was that you learned from a project and explain what you were thinking. And so for this, I made a blog.

[Adam Gordon Bell]

Nolan demoed this at a Friday presentation at the Recur center where anyone could grab a slot and show what they were working on.

[Nolan Royalty]

And I think that was the first time that, like, I saw that people enjoyed something that I made as I, like, presented it. And the response is like, pretty good. Getting positive feedback on that was really fun.

[Adam Gordon Bell]

Someone at the Recur center shared Nolan's blog post on Hacker News, and then it gained traction. Encouraged by this response, Nolan kept going to the Rygir center and he just kept making games.

[Nolan Royalty]

I made several more kind of more traditional games. You know, games with like high scorers and guys that you controlled and you, like, shot at enemies. I remember taking a game of mine, a more normal game of mine, to a playtest and being there with some friends from Recurse who had brought games that I thought were much more interesting than what I had made. And being at this playtest event and getting feedback on my game from some people and just being embarrassed to be showing the game, being like, oh, I really not like it was a terrible thing, but just that it so clearly wasn't what I wanted to be doing. I wanted to be one of my friends showing these much weirder, more experimental things. And I was getting feedback on how the numbers in my game could be adjusted and like, oh, man, I just really don't want to think about balancing numbers in games. That's not what I'm interested in, what I'm passionate about. And so I think the big thing there for me was learning that what I wanted to be doing was this kind of weird stuff, and also that I was just better at it. I'm very, very into taking a weird bit of technology and pushing it as far as I can. I have zero patience for picking nice numbers that make a game feel balanced. I don't want to work on games that need nice numbers to feel balanced. I want to work on weird, experimental stuff.

[Adam Gordon Bell]

He wanted to do the stuff that bent the rules, that surprised people, that maybe even broke things a little bit in a fun way, just like that original boarding school email prank. But now he was channeling his creation powers into this. And after 12 weeks, Nolan's batch ended. But he kept showing up at the recurse center every day.

[Nolan Royalty]

I came up with this idea for making a game inside macOS Finder by renaming files and doing a bunch of other hacky stuff, which turned into a version of Flappy Bird that you ran inside macOS Finder. That I think was the first time that I remember making something and realizing after I demoed it at recurse, I was no longer in batch, but you can still show up to demos whenever you want, and people lost their minds. The reception was really, really, really good, and a bunch of people wanted to talk to me about it after. And I had also just loved making this thing. I thought it was so, so fun, and I loved talking about it. Getting to present it and the story of how to make it faster was really fun. I got a bunch of advice from some friends on the problems, and some people helped me solve that, but that was the first thing I think I made that I was like, oh, wow, this is, I think, actually good.

[Adam Gordon Bell]

Meanwhile, Nolan is wrestling with his direction.

[Nolan Royalty]

I like having a plan, and when I don't feel like I have a plan or a goal objective function, it's hard to know that I'm making progress. And I do need to be. Making progress, I think was really, really challenging at points because I would wake up sometimes and be like, what am I do? Like, what in the world? What's the goal? What Am I trying to do? Does anyone like me in any way? What? And I sometimes still feel that way and find that really stressful.

[Adam Gordon Bell]

But then Flappy Durr was released and it went to the top of Hacker News, and then it was on Daring fireball. And then GitHub, they have a YouTube channel, and they covered it on that. From this experience, Nolan learns to refine his taste a bit. There is an appetite out there for these sorts of conceptual, weird games. And so he keeps building.

[Nolan Royalty]

And then I made this game called StrangerVideo, which I think is one of the things that I'm still probably the proudest of. And Stranger Video was a website that showed you live video of a stranger's face until one of you blinked. And that's all it did. It used MediaPipe, which can do. It's a library from Google that has face detection to crop video to only show faces and prevent you from showing anything else. And also to detect whether your eyes were opened or closed. That was it. It was just a website with silent video of somebody else on the website, where all the restrictions were kind of coming from this place of, like, I've used chat roulette. I understand what happens when you just have strangers look at each other on the Internet and what they choose to show. And I don't want them to be able to show those things. I want just only faces. I was also kind of thinking at the time about some social anxieties I have around. When do I make eye contact with people that I've noticed far away and kind of wanted to channel that into a game.

[Adam Gordon Bell]

So Nolan releases this game, and for some reason, it went viral in Japan, like really viral for a couple days. And then even some of his friends were very touched by it. Martin, one of his friends, said he never had so many people look at him and smile in his life.

[Nolan Royalty]

And that was like this profound, like, oh, wow, I have delivered. I've given Martin this, I think, genuinely cool, novel experience and kind of opened my eyes to a new thing type of thing that I could be building. People playing with the restrictions was really interesting. So your eyes needed to be opened, but you could put pieces of paper over your eyes that had little eyes drawn on them and that tricked the camera. And if this was a competitive game, that'd be a problem, but for a weird social experiment that's actually just funny and fine. I think I made this thing and I tried to figure out why it worked and why I liked it so much and how to do more things like it. And that Was certainly a moment of validation and of like, okay, like, I'm proud of this. I want to make more things like this. And people like it when I make these things. And I think it has artistic value.

[Adam Gordon Bell]

So the next constrained collaboration game was Talk Paper Scissors, the phone version of Rock Paper Scissors, but with strangers.

[Nolan Royalty]

So you, like, call a phone number, it asks you to say, rock, Paper, Scissors. Use some voice detection so that you can't say anything but those words. Plays your audio to someone else, plays their audio to you. And that wasn't as much of a hit as Stranger Video, but again, got, I don't know, like, 50,000 calls or something. Like, enough calls that was kind of validated. Again, like, this is. People like to play with strangers. There are not a lot of environments that are obviously restricted enough that you feel comfortable doing it.

[Adam Gordon Bell]

Both games thrive on constraints, right? They connect you to strangers, but they have limits on how you can interact.

[Nolan Royalty]

And I think in particular had this theory of, like, the trick to this is constraints. And the cool things that come from this are how people subvert those constraints. You add constraints to constrain the assholes, but also that gives people material for funny things to do. That wouldn't be possible with more of a sandbox. But since the thesis here is simple interaction with constraints that prevent jerks, it didn't really feel like something I could force. You need to come up with an interaction that you think is going to be compelling and fun and maybe silly. And you also need to come up with a way to constrain the obvious ways of abusing the action.

[Adam Gordon Bell]

Stranger Video and Talk Paper Scissors weren't just games. They were experiments in constrained chaos. Nolan was almost designing for playful subversion, for people to find the edges and to push against them, just like he had in the early days. Meanwhile, he craved community. He craved that interaction that comes with building things together. So he went to NYU's ITP summer.

[Nolan Royalty]

Camp, which stands for, like, I actually don't know, Interactive Telecommunications Program or something. It's pretty old, so they probably used the word telecommunications. But it's a school that's focused on interesting artistic, mixed media stuff, tech, art things. It's like a real. You can get a Master's there. I don't want to get a Master's. That's a lot of time and money. But they have a summer camp that's like, hang out there for three weeks with people that think it would be fun to hang out there for three weeks. Which, like, I don't know, I probably am going to get along with most of those people. So I did that this year because I didn't want to do recurse again.

[Adam Gordon Bell]

At the summer camp, Nolan dove into face tracking tech. It was fun, but it was a lot of work and he couldn't release anything and it left him feeling eager to just get something out the door.

[Nolan Royalty]

I think I feel really good when I'm regularly releasing stuff and when I'm not, I get kind of down on myself. And I got lunch with my friend Neil who makes the website Neil Fun, which is a really lovely collection of fun different games. Infinite Craft is a very popular game of his from this year.

[Adam Gordon Bell]

Feels like you two should know each other.

[Nolan Royalty]

Yeah, we're both in Brooklyn. I think a surprising number of people that build funny stuff like this are from Brooklyn. And at the same time that scene is not that big. It's feasible to know a lot of those people. But I think we originally met via a friend of mine from Recurse. And then, I don't know, we've got common interests and sometimes get together to bounce ideas off each other. And I think explicitly the goal was we were bouncing ideas off of each other. We had originally planned to do kind of a stranger video style thing together and then he made Infinitecraft and that went crazy. That went completely insane. And so he was busy for months and I was doing other stuff and then that had calmed down. We were like, okay, we'll come back and we'll collaborate on this thing. And I pitched 1 million checkboxes just as kind of a joke.

[Adam Gordon Bell]

The pitch to Neil for 1 million checkboxes is simple. It's a website with a million checkboxes on them. If you check one, it's checked for everybody else. That's it.

[Nolan Royalty]

And we chatted about it for a bit and we're kind of like, eh, this is funny. But I don't think there's that much there. And I don't think either of us thought it was like a bad idea. But we were both kind of like, this is not the thing that we should collaborate on. But I thought it was funny and kind of interesting and I knew I could build it really fast. I knew it was just not that much work. I know how websockets work. I know how to put checkboxes on a webpage. And I was super hungry for making something quick. Like I said, I hadn't shipped something in a while. And so near the end of the conversation I was like, hey, do you mind if I just go build this? I think it'll take me a few days, and then we can collaborate on one of these other ideas when that's done. And that was fine.

[Adam Gordon Bell]

After that Friday lunch, Nolan spent the weekend figuring out how to build 1 million checkboxes in a way that would feel responsive in real time. It's a lot of checkboxes, a million of them, and they need to stay all in sync across all the users.

[Nolan Royalty]

How do I only render the checkboxes that you're looking at? I've never built an application that uses websockets of this scale. Like, can I sketch out an architecture that will handle the couple hundred people that I think will play and that I could scale up if it was a little more popular than that. And so I kind of came out of that weekend with a plan, and then I built the site in two days. I actually wrote the code the next Monday and Tuesday.

[Adam Gordon Bell]

So on Wednesday, he launched it, the million checkboxes. And that's when things got interesting.

[Nolan Royalty]

I was excited about it. I thought it was funny. I was proud of the constraints I had added, in particular, in ways we can talk about later, made it difficult for people to draw, because I understand what happens when you let people draw on the Internet. And so I was proud of all of that and thought it was evidence that I had kind of learned how to make websites, which wasn't true at the start of the year. So I was proud of that too.

[Adam Gordon Bell]

Nolan thought it was just a curiosity. Unlike Stranger Video, all you could do is check a box. There wasn't a lot to do, but somehow it took off.

[Nolan Royalty]

I think it was first Mastodon. My theory is, like, it was Mastodon, and then some big names on Mastodon posted about it, and that got several people to post it to Hacker News. And Hacker News collects votes if you repeatedly submit the same link in quick succession. And that got it onto the front page there and then. I think once it was kind of popular, the fact that it was kind of popular drove the popularity. You went to the site and, oh, my God, all these boxes are changing all the time. The site is so much more compelling because it's weirdly popular. And that kept it more popular. I don't know. I think the first three hours were me noticing, wow, there's some activity on the site. And then, like, oh, my God, there's really activity on the site. And then, oh, my God, my servers are going to. This is going somewhere, and if I don't start working, the server's going to fall over and I need to stop checking the load and start Writing code. And that was. I don't know, that was the morning. But it was a very fun, if still slightly confusing thing to wake up and see.

[Adam Gordon Bell]

That week marked Nolan's third at summer camp, and it was time to show off what he had made his new spin on Pac Man.

[Nolan Royalty]

And the camp show was that Wednesday night, like you're supposed to be presenting something that you had worked on there. I was planning on presenting my Pac man game, and instead the game had gone pretty viral by that point. And so I was manning my booth, trying to show people my game, but mostly spinning up more servers, trying to keep the website online.

[Adam Gordon Bell]

How many people did you reach?

[Nolan Royalty]

So the numbers here are a little fuzzy for two different reasons. The first is that I originally. So I logged everything, but when I wrote the logging code, I was like, there's no way that more than a million boxes are checked in a day. So my logs will be truncated at a million lines a day because that's an insanely high number. That's stupid. And I didn't think about that until 5pm or something. So I lost all my logs of what boxes people checked for the first several hours, which is a bummer. So my estimate of players from the first few days is from looking at my NGINX logs, which I have IPs, et cetera, and I think there's something like 500,000 unique IPs there. It's hard to know sometimes people are going to be like, many users are behind the same static IP at a school or whatever. In the first few days, on average, we checked something like 50 million boxes a day, which. Which is just totally, totally nuts and well beyond expectations.

[Adam Gordon Bell]

50 million checkboxes. Half a million people. These numbers are wild, right? Picture the. The largest stadium or music venue you've ever been to. It's more people than that. The largest stadium in the US Per Wikipedia, is Michigan's big house. And that's a hundred thousand people. We're talking 500 here. That's five times that many people over the course of 48 hours. And these half a million people fall into two camps, right? There's the checkers who like to check things and watch the score go up. But then there's also the uncheckers, unchecking the boxes and driving the numbers down.

[Nolan Royalty]

I think you can look at media coverage of the site and plenty of time. It'll say something like, it's a website where the goal is to check all the boxes. And I'm like, I didn't. I didn't say that. I just said, here's some checkboxes. Do whatever you want. And it's fine that that's your goal, but that's your own goal. But I think that that was part of the appeal here, is that I think in general, people are much more attached to goals that they come up with themselves and saying, here's the thing, do whatever you want with it. And you deciding, my goal is to prevent all the boxes from getting checked, or my goal is to prevent box number two from being checked ever, or I want to check every box.

[Adam Gordon Bell]

Pretty quickly, people start having their own stories about how they're using the site.

[Nolan Royalty]

So an example of that is I would often hear from someone who would find a place where someone was checking a bunch of boxes and they would follow that person and uncheck the boxes and they'd be like, yeah, and they were getting so mad, right. I was trolling them and they were getting so mad. And again, it's like, you don't actually know that. Right? You have no idea whether they're noticing what you're doing. Like, you have no idea what's going on with that person. Maybe they find it funny. But because you have such limited information, it's really easy for you to play pretend in this way. And that's just super fun. Like, you get really into that. So I think it really let people tell those stories in fun ways.

[Adam Gordon Bell]

Okay, let's get into the weeds for a minute. Imagine managing a million pieces of shared mutable state with tens of thousands of reads and writes per second. It's a scaling challenge. It needs to feel real time to engage users. And plus, it has some kind of aggregate data. You need to have a count of the number of things each user has flipped and then a total. Seriously, if you've ever had to stop and scale a data store or a database, stop and think about how you would handle this for a minute. What's the hard part? To me, the issue is the number of writes. If you're going to use a traditional database, it's a lot of concurrent writes, and some of the rows, depending how you organize the data, were going to be really hot. Those first 100 checkboxes, I'm sure, are where a lot of people are clicking. So then I start thinking about something in memory. But how do you keep all this state in lockstep?

[Nolan Royalty]

Yeah, yeah. The architecture was like, Redis has a bit set just roughly a million bits with some nice operations for flipping individual bits. So Redis has that. Redis has a count of the number of boxes checked, stored separately, just so that we don't have to iterate through a million things to determine how many are checked, and the website's supposed to display that. Side note, Redis, also, you can write LUA and invoke it via Redis atomically, which is really sick. So you really don't have to worry about races as long as you write your logic in LUA scripts. So it was really easy to say, oh, we're storing this count separately from the number of checkboxes. There's an obvious race there, kind of, you know, depending on the primitives you have. But, like, let's not even think about that at all.

[Adam Gordon Bell]

Redis has a pub sub feature. You can flip a bit, bump a counter, and then broadcast the change all in one LUA script.

[Nolan Royalty]

And the rough architecture was you had a bunch of workers whose job were to accept websocket connections from people on the website. They offered an API that let clients flip bits along with some basic rate limiting. There was also some client rate limiting, but that's obviously trivial to avoid. And then whenever a client flipped a bit, you would write that to the pub sub. And then every websocket worker process was also reading from that pub sub. And whenever something got written, there would update all of the clients that were connected to that individual instance.

[Adam Gordon Bell]

This is smart, right? State management is atomic and fully outsourced to Redis. But scaling the workers to handle the websockets, that's still a challenge. And then there are other things that Nolan didn't even think about.

[Nolan Royalty]

I think it was night one when I realized I hadn't thought about bandwidth cost. I was managing all my servers myself. And so I had originally decided, okay, my costs are capped. I know how much my servers cost. I'm spinning up a bunch of servers. This is probably going to cost me a couple hundred bucks. I'm getting some donations, that's probably fine. And then remembered that you get charged for bandwidth, and I'm sending a weirdly large amount of data. Especially at this point, I hadn't thought about how to reduce the size of these updates that I was sending clients. And so I was pushing out hundreds of megabits a second or something.

[Adam Gordon Bell]

So Nolan started doing the math and estimating costs, and the math gave him an estimate of $10,000 for his bandwidth so far.

[Nolan Royalty]

And the pit in my stomach, it just felt terrible. But I called a friend, my friend Greg, who I worked on Talk Paper Scissors with, and he talked me through it, and we agreed that My math was wrong, but also I started working on like, I can vastly reduce the amount of data I'm sending. That's going to help a lot. And two, I happened to be pretty comfortable using the Linux command line utility tc, which stands for traffic control and is an obscure utility for limiting bandwidth on instances. My job back at Amazon Web Services a decade ago involved using it a lot. And so I was like, oh, I can just add a rule that drops all traffic, all outbound traffic above. I think it was 250 megabits a second. And that let me calculate an upper bound. And that was what really mattered to me was like the upper bound was kind of high. But DigitalOcean's bandwidth pricing is actually quite sane. So that was the one time that I was really scared. Beyond that, I was delighted and baffled and excited and super motivated to keep the site up and keep working on it and improve it in different ways. I had made a mistake that I often make when I make things like this, which was I had very little attribution to myself or any self promotion when I made it. I don't even know that there was a link to my website. There was certainly not a link to my mailing list or a way to donate me money or anything like that.

[Adam Gordon Bell]

Nolan added links to his website and his mailing list and just like that the media came calling and I did.

[Nolan Royalty]

The interview with the Washington Post, which was fun. They asked me how I was monetizing it and I was like, not really, but I think in the question they were like, are you going to sell this to the New York Times? Like wordle? And I made a joke about I will sell it to the New York Times for $1 a checkbox. And 12 hours later I got an email from the New York Times which quickly came with a follow up email saying, we're so sorry, this is not an offer to buy the site. But was just someone there who also wanted to talk to me. And I did all that and that was super fun. That was the first time I'd ever done any media like that and pretty validating. It was very convenient for the family in that they don't really understand what hacker news is or whatever. But hey, I'm in the New York Times, that is meaningful to them.

[Adam Gordon Bell]

Next up, Nolan needs to boost the streaming server's performance.

[Nolan Royalty]

So I had a friend from the startup that I used to work at back in Michigan who these days is a performance engineer at Facebook who is very good at making things fast. And we've been wanting to work on something together for years and hadn't had the time. And I was like, one, you really want to work on this problem? And two, I actually really, really need help here. So I called him up and went over to his apartment at 2pm on a Sunday, and we worked until 2am on a Sunday.

[Adam Gordon Bell]

Part of the big rewrite was moving from an untyped Python flask app to go.

[Nolan Royalty]

During the rewrite, we noticed at several points we screwed up our rate limits in one way or another. There were a couple of pretty embarrassing bugs. It was. We'd just been working on this thing for a long time. It was getting really late. And so with one of the first versions of the Go rewrite we launched, the site went totally crazy because we had broken our rate limits and Redis briefly broke. We were sending Redis too many updates. That was crazy.

[Adam Gordon Bell]

Normally you could only send a limited number of requests per ip, otherwise you could just open developer tools and write a little for loop and start looping through and checking all the boxes. But were people actually hitting that limit? Were people writing bots and hitting the API?

[Nolan Royalty]

You know, sometimes you'd go somewhere and like 30 boxes would clear all at the same time, and you'd be like, that's not a human. But I think the scale of it wasn't clear until we removed the rate limits for a little bit and the bots went crazy. And during the rewrite, I had noticed briefly some data in Redis that looked weird. And by looked weird, I mean it contained URLs.

[Adam Gordon Bell]

Nolan had spotted some weird data before. In fact, he had had the site crash due to weird data on the first big night.

[Nolan Royalty]

And the way that it crashed is that I wasn't validating indices. So I had an API that the clients used to check or uncheck a box, and it took an int. And I didn't validate that the INT was in the range 0 to 1 million. And someone was like, what if I manually call this and send 10 million and check that box? And that totally worked. And Redis behavior around the data structure that I was using was like, oh, I was told to check bit number 10 million. I have a million bits. I guess I need to add 9 million bits of padding to check bit number 10 million, right? So that happened. The site crashed. They did all sorts of weird things. And I woke up to this and quickly figured out what was going wrong, but made a couple of utilities to dump the database. And my goal then was like, I will bring the site back up and I will just truncate the bitset to the first million bits. So I had these scripts that dumped the data in the database and they happened to just based on what Reddit does, when you ask it to display this data structure, convert that data to ASCII. So you have 125,000 bytes, a million over eight. And each of those bytes is a number. And we look it up and decide what ASCII character is. And that is what Redis prints. Unless you're asking it to print a character not in the ASCII character range, because ASCII is only, what, like 127. Otherwise you get like hex digit approximately.

[Adam Gordon Bell]

While running the script to dump checkboxes, Nolan and his friend Elliot spot the URL in the data. But they have bigger priorities to fix.

[Nolan Royalty]

And I saw that while Elliot and I were working on it and we were in the middle of this rewrite from Python to go. My Python had no types. It was wildly unsafe in all of these ways. We noticed it briefly and we're like, oh, you have another bug around. Not validating user input. But I thought that I had a data validation error that let people stuff strings into this bitset, because I hadn't really even thought about what it was that my script did at this point. And so we noticed this and we're like, well, we're rewriting this in a much more type safe way. This bug will just be fixed. So we don't have that much to worry about, right?

[Adam Gordon Bell]

With the new service up and running, they breathe a sigh of relief and they call it a night.

[Nolan Royalty]

And then it got hilariously faster with the GO rewrite. I have these CPU graphs that were pegged at 100 and then we launched the GO version and they're pegged at one. There's just nothing. It's ridiculous. And I woke up the next day and I ran the script again, I guess just because it was still kind of bugging me. I had noticed these URLs. It seemed like maybe I had been not quite hacked necessarily, but something bad was going on. And I ran the script again and I noticed that the URLs were still there. And I was messaging him, being like, they're still there. We're going back and forth and we're like, this shouldn't. We're type safe. There's no obvious. You're just injecting strings into my database bug here. What is going on? And then I remembered something, which is that I had gotten a bunch of DMs on Twitter about OMCB. And many of them were just kind of like, hey, I like this thing. But one had been from someone who was like, hey, can I help you rewrite the site to make it faster? I am friends with a bunch of people who are good at writing, performing code. I was like, maybe. I don't know. I like collaborating with online strangers and giving you access to my code base. A little sketchy, but that is a very sweet offer. And then he had sent me this other message that I had no idea what to make of at the time that was like, look at the end of the site or something. A friend of mine left you a message, and that message popped back into my head, and I had no. I think I had sent him at the time, like, what in the world are you talking about? That's such a weird thing to say, but realized like, okay, maybe there is something there.

[Adam Gordon Bell]

This all happened on Monday evening, and Nolan had started to connect the dots in his head, but also his girlfriend had made him dinner.

[Nolan Royalty]

And I had kind of gotten re. Obsessed with what is going on here right as dinner finished. And she was like, can we sit down and eat? You've been very busy recently. And I was panicking about, There are these URLs. I have been hacked. I have been hacked. And had decided, okay, I'll stop thinking about this for a second. And sat down and then had this thought, and it all came together. And then rushed back to my computer to confirm what was going on. And I think all the pieces kind of came into place. Started looking at the actual site. Oh, my God. There is a pattern here. Oh, holy shit. The data is in my database. Because what they are doing is checking and unchecking boxes to make the binary representation of this when interpreted as ascii. Spell out this URL. This isn't hacked. This isn't. I have a bug. This is like, someone who is being very persistent about writing a weird message into my database.

[Adam Gordon Bell]

The only way this could happen is with a bot, right? Any visitor can check or uncheck a checkbox, but this URL is always there. And encoding that URL takes a lot of checkboxes in a row, right? 192 for this specific URL. That implies a bot is enforcing the data in this very specific part of the checkbox grid.

[Nolan Royalty]

And the URL linked to a discord. It took me, like, five minutes to convince myself that I could click it, since it was CatGirls WinOMCB, and that seemed super sketchy. But I did everything I could to convince myself that it was a real website, that it was safe to visit, and then decided that I couldn't convince myself of that, but that I would regret it forever if I didn't click the link. So I just had to click the link anyway and I clicked the link, went to a discord. Discord is called Check in Boxes, and it was full of teens talking about botting the site who had been kind of gathering because they were all looking at this data. They had all figured out how my API worked. They were writing tools that hit the API on the command line, looking at either the base 64 or ASCII representation of the data for one reason or another, and had all noticed these URLs and had joined the discord of, I think one of the first people to start botting the site.

[Adam Gordon Bell]

That's awesome. How many people were in the discord?

[Nolan Royalty]

I think There were about 20 when I joined and then by the time I shut the site down, about 60, which is just mind blowingly cool. Not remotely what I expected. It's kind of impossible to convey the feeling of going from, I have been hacked. This data is still here. I think it felt worse because I thought that I understood. Yeah, I was kind of owned. Someone had stuffed these strings in, but then we fixed it and then, oh my God, I haven't fixed it. Maybe somebody has root on one of my boxes. Something is really bad to wait. There's this whole community and they're doing something that is insanely cool.

[Adam Gordon Bell]

Nolan created 1 million checkboxes without a fixed grid because he wanted to stop people from drawing offensive images. The number of checkboxes wide was device specific, so you could draw whatever you wanted, but most people wouldn't see it. This tied into his idea of curbing bad behavior. But if you mass check and uncheck boxes and build a CLI app and somehow write to the bitset as ascii, you find a URL and it leads you to a discord. And just like that, a community forms.

[Nolan Royalty]

And their big goal, in addition to gathering more people, was drawing on the site, which they were doing by treating it as a thousand by thousand canvas. So they would have a bot that it would take in an image and coordinates and a size, and from that determine the set of checkboxes that would need to be checked to form that image on this hypothetical thousand by thousand canvas that they were drawing onto, and then listen to events and check and uncheck boxes to make sure that that image stayed drawn on the canvas, which was just insane. And so so cool.

[Adam Gordon Bell]

So does it farm out the work to multiple of them or are they each?

[Nolan Royalty]

I think within the discord, people had a sense of who is drawing what where. So they solved the problem of like, obviously if you run two of these bots on the same territory, you just check and uncheck boxes forever and nothing ever gets drawn and my server crashes. But for the most part, almost everyone who was drawing stuff was in the server. There was one exception to some people that were trying to draw the logo for osu, the game, who weren't in the server and were fighting over territory with the people on the server. And that was pretty funny to see. And then the other big problem for them was I had rate limits. I rate limited people based on IPs, and you could work around them by getting more IPs. And so the people that were the fastest and able to draw the biggest things were the people that could come up with the best ways to get a bunch of IPs to send their traffic from. And these are kids, mostly, they're mostly 18, so they don't necessarily just have a bunch of money to rent a bunch of servers. So they were coming up with clever. There are ways to route your traffic through residential IPs. They were using a bunch of stuff like that. So that was kind of the big constraint for them was having enough IPs and coordinating which check events you farm out to, which IPs you need, whatever, a central server that's listening for those events and then farming out the work to a bunch of different workers, which is kind of an impressive amount of infrastructure to spin up for something like this.

[Adam Gordon Bell]

So what were they drawing?

[Nolan Royalty]

When I joined, they were drawing really simple stuff. I think the first thing they wrote was be gay, do crime. And then after that, when I joined, they had a QR code that linked to the discord, which I thought was really funny. But they started getting more and more complicated over time. One way that they got more complicated is they started coming up with big detailed images to draw. They did a Windows blue screen in death, which I thought was really funny, but all sorts of things that they would try to draw over the vast majority of the grid, which just took more technical work. And I say vast majority there because they were very careful to not touch the top few or bottom few rows because they figured that's where most people were playing and they knew they were kind of degrading the experience, which I thought was pretty sweet of them. But then the big innovation later was, oh, we should start doing gifs and what are animated images that we could put on. So obviously we got some Rickrolls at some point. We got a lot of Jake Gyllenhaal GIFs. There's. Do you know Bad Apple?

[Adam Gordon Bell]

No.

[Nolan Royalty]

So there's this thing, and I don't quite understand this phenomenon, to be honest, but there is a video called Bad Apple. It is a cool, interesting anime video thing. But the point is it's a meme to make Bad Apple run inside stuff. Kind of like how with computers it's always like, can I run Doom on this thing? It's like, can I run Bad Apple inside this thing? So there was a project Bad Apple that a lot of people were devoted to and I actually hadn't thought that they had succeeded. And then when I looked back through all the data after building some of my own tools to inspect it, found that someone had successfully run Bad Apple on it. So that's cool and exciting.

[Adam Gordon Bell]

Nolan's main concern is figuring out the right way to connect with this community.

[Nolan Royalty]

I didn't want to ruin it, right. I think it was very fun that I was there and they were excited. They had a lot of questions for me, they had found bugs, whatever, they wanted to talk to me. But at the same time I did think, this is your fun little hangout for you and you're mostly teen friends and I am a 33 year old software developer and we're in very different life stages and you should have your own space. So I did kind of try to pop in, answer questions. I would pop in and make announcements like, hey, an interview with me is running in the Washington Post. There will be new users. Please limit your botting for a little bit, things like that. And they were happy to listen to that, which was great. But they had a bunch of questions about the site, about how I was solving things and just kind of about my deal. I think they're just interested and I was happy to talk about that. In a few cases, people have gotten in touch with me somehow to ask whether any of the teens in the discord are looking for jobs. And so I have relayed some job offers to them, a couple of things like that. But yeah, we've chatted a fair amount and they're great. I don't know, I'm so proud that they decided to point their talents at my website. And I think they're just crazy talented.

[Adam Gordon Bell]

For Nolan, just knowing this group existed changed things.

[Nolan Royalty]

This felt like it really validated this thesis that I had had from back when I made Stranger video And other things. I had had this belief for a while, based on these smaller but still moderately successful stranger interaction things, that the way to approach a problem like this, or a way that I think works well is to have a small, fun, novel interaction with enough constraints that people can't trivially ruin it, and that those constraints produce really fun, interesting behavior. And this just felt like the purest possible validation of that thesis that I believed this. I had given some talks where I believed this, and I said it in the talk. And I think the talks were okay, but I didn't have enough proof. And this was all of the proof I would ever need. This was so much of a reason to continue pursuing projects with this flavor. Yeah, it was validating to kind of like this core thing that I knew to be true but needed to prove in a bigger way. And that was also really, really thrilling. And I don't know, we've talked a couple of times about clarity of what to pursue and what I know how to do and what the goal is. And I think that was another step in that direction.

[Adam Gordon Bell]

It's funny how this direction that Nolan has refined and this direction that he's still on, it all started off with that playful exploration and boundary pushing around email rules and how they were handled back in high school, because back then, he was the one who was having the trolling behavior that other people saw potential in. And it wasn't just his dorm supervisor person who created him a T shirt because he also emailed his high school computer science teacher about the rules he was creating.

[Nolan Royalty]

I don't think I knew what Turing completeness was, and I'm not sure that the rules were actually Turing complete, But I was kind of like, this is way too powerful for a school emul system. But my memory of it is basically him being like, don't mess everything up. That's pretty funny. I'm sure he was kind of proud. Yeah, I don't know. I think back to that sometimes in terms of like, wow. You could easily imagine a world where the response by the school was like, you're expelled. You broke a bunch of stuff. And the fact that instead I was kind of mischievous, but it was kind of funny. And it was ultimately pretty low stakes and people just kind of were encouraging was really nice. There are a lot of versions of trolling that are really fun. If you go back to the version of the Internet that we had in 2008, I think there are lots of places where people did some kind of large scale collective trolling that is remembered very fondly, like 4chan rigging polls to spell out funny things. The thing that was the most emotionally satisfying to me was certainly discovering this and then just being so proud and excited of them and of myself for making something that they thought was worth their time. Getting to be proud of them and not mad at them was so meaningful to me. I cried a lot about it, which is not something I normally do. I think that when people think about trolls now, this is more like, I don't know, people being really mean on Twitter. But providing people a very low stakes way to troll, where the worst thing you can do is uncheck a checkbox is great, right? Like, it's fun to do that. It's fun to prank somebody in some small way and it's totally okay. And I don't know, I think people. People really enjoy it when you give them a chance to do that.

[Adam Gordon Bell]

That's it. That's the show. Nolan's had multiple projects spreading around the web since we last talked. He's really good at this. Reject Bad apple. Is that Bad Apple animation he previously ment but playing Inside Vim using regular expressions. Everyuid.com is a website with every UUID on it somehow. So thank you, Nolan, for pushing the boundaries. You're clearly one of the world's experts at twisting things beyond their intended use for our amusement. Go to his website, eieio Games, and he can send you a newsletter whenever his new creations come out. And thank you, especially to the supporters who keep me at this. The main benefit of being a supporter is literally showing your support and I do appreciate that. But there is also a Slack channel and some bonus episodes. Really just thank you everybody who joins the Patreon supporters group just to show that they support what I'm doing here. And yeah, we also have a pretty awesome Slack channel that you can find on the website. So I hope you liked this one. And until next time, thank you so much for listening.