The Hutter Prize: Compression, prediction, and the limits of intelligence

Summary

Podcast Summary:

CoRecursive: Coding Stories
Host: Adam Gordon Bell
Guest: Don McKay
Episode: The Hutter Prize: Compression, Prediction, and the Limits of Intelligence
Date: August 4, 2026

---

Episode Overview

In this engaging episode, Adam Gordon Bell and recurring guest Don McKay explore the Hutter Prize—a long-standing contest that rewards advances in lossless text compression on a large slice of Wikipedia. The duo delve into the mechanics, history, and underlying theory of data compression, connecting its evolution to concepts of intelligence and artificial general intelligence (AGI). They experiment with various compression algorithms on the air, discuss milestone figures from computer science history, and debate how prediction, understanding, and compression intersect.

---

Key Discussion Points & Insights

1. Introduction & The Hutter Prize (00:00–01:48)

• Adam introduces the Hutter Prize, a contest challenging participants to losslessly compress a 1GB Wikipedia file to under roughly 110MB. The prize is structured so that greater compression yields more prize money, up to €500,000.
• The premise: "Being able to compress well is closely related to intelligence... The intention is to encourage development of intelligent compressor programs as a path to AGI." (Don, 01:26)
• The contest is framed humorously as: "It's a weird thing to say, the development of intelligent compressors as a path to artificial general intelligence. It feels like saying, you know, like this crossword contest will clear the way for world peace." (Adam, 01:48)
• The practical challenge: Even standard tools like gzip fall far short; Adam notes that gzipping the file only reduces it to 322MB, well above the threshold (03:29).

2. Live Demo: Trying (and Failing) Basic Compression (03:56–08:47)

• Adam and Don discuss basic compression methods:
  • Run Length Encoding: Good only for repetitive patterns, quickly fails on real text ("I actually made the document longer." —Adam, 07:11).
  • Pointer/Dictionary Methods: Improved compression by referencing repeated text chunks ("You’ve widened your pattern recognition." —Don, 10:01), but still not close to the contest benchmark.
• The duo highlight that standard and naive algorithms don't suffice for the Hutter Prize's target.

3. History of Compression: Huffman Coding & Its Origins (10:04–16:34)

• Morse Code Insight: Most common letters get shorter representations—an early form of compression.
• Huffman Encoding Narrative: Adam recounts how David Huffman, as a student, solved an unsolved problem in information theory for a class assignment, beating his professor (Robert Fano) and Claude Shannon (inventor of information theory and the bit).
  • Notable quote: "It was the most singular moment of my life... an absolute lightning of sudden realization." (Don reading Huffman's recollection, 12:37)
• Compression as Prediction: Adam describes how Huffman encoding assigns the shortest codes to the most frequent letters, making text files smaller. Their implementation demonstrates this on sample data and Wikipedia (15:43–16:29).

4. Zipping, LHA, and the PKZIP Story (16:34–20:45)

• Ziv-Lempel (LZ77) Algorithm: Pointer/dictionary approach, foundational to ZIP/GZIP.
• Phil Katz and PKZIP: Story of how Katz created a faster, free, assembly-written ARC clone; legal fight over appropriated source code; eventual rewriting of ZIP as "deflate" algorithm, which becomes the open standard (18:35–20:13).
  • Notable moment: "He left his mark on the world. Everybody's still using the software." (Adam, 20:45)

5. Advanced Techniques: Arithmetic Coding & Kolmogorov Complexity (21:15–41:49)

• Arithmetic Coding: Encodes only what’s surprising (less frequent), achieving sub-one-bit-per-character representation for highly skewed data.
  • "It allows you to get below one bit per character." (Adam, 22:21)
• Kolmogorov Complexity: The ultimate lower bound: a file is only as compressible as the shortest program that outputs it. Adam demonstrates with a random seed generator and discusses the logical limits of compression.
• The Sloot Digital Coding System: A cautionary tale—claims of super-compression (entire movies in 8KB) that likely relied on shared external sources and died with the inventor. Insight into the challenge of leveraging pre-existing shared knowledge.

6. Compression as Prediction: Shannon’s Entropy Game (27:04–36:31)

• Claude Shannon’s Entropy Concept: Essential information in data has an irreducible minimum ("entropy"). He estimated English’s entropy via a letter-guessing game, revealing text is much more predictable (and compressible) than it appears if you understand the language.
  • "The meaning of a message is how surprising it is." (Adam, 35:53)
  • "If you understand the patterns in a giant swath of Wikipedia, how is that different than actual intelligence?" (Adam, 51:49)
• Compression ↔ Prediction ↔ Intelligence: Great compression requires understanding, i.e., intelligence capable of robust prediction.

7. Modern Compression & The Role of LLMs (44:02–49:03)

• Predictive Text and Compression: Modern LLMs (Large Language Models) can act as compressors by predicting text extremely well. For instance, "LLMzip" uses a large language model to reconstruct text, yielding record compression ratios (11.27x)—but only if you ignore the massive size of the LLM model itself.
  • "They did find that intelligence is helpful for compressing things." (Adam, 45:49)
  • The practical problem: decompressors (e.g., a 13GB LLM) are vastly larger than the compressed data.
• Alternative Approaches: TensorFlow Compress dynamically trains a network during decompression—impractical due to hardware and time requirements, but further proof of intelligence-compression linkage.

8. Constraints, Records & Ongoing Challenge (49:03–52:14)

• Cmix and Current Records: State-of-the-art (non-LLM) compressors use hundreds of predictors working together (ensemble voting), achieving current records around a 9x compression ratio.
• Test Limitations and Future: The Hutter Prize's hard file size constraint makes it hard for LLM-based approaches to qualify. Discussion ensues about whether and how the contest should be modernized.

9. The Fundamental Insight: AGI and Compression (50:44–52:14)

• File Compression as a Proxy for Intelligence: Only systems that understand, predict, and encode the semantic structure of data—i.e., are intelligent—can push compression ratios to theoretical limits.
• Adam’s Takeaway: "To make a file smaller, you need to understand the patterns in it. If you understand the patterns in a giant swath of Wikipedia, like, how is that different than actual intelligence?" (Adam, 51:49)
• Don’s Reflection: The episode deepened his appreciation of compression’s intellectual depth—if not his wallet.

---

Notable Quotes & Memorable Moments

• On the Connection Between Intelligence and Compression:
  "Being able to compress well is closely related to intelligence... The intention of this prize is to encourage development of intelligent compressor programs as a path to AGI." (Don, 01:26)

• On Early Compression Lessons:
  "The easiest thing I know of is just like when you have a repetition, you take it out... The whole idea with compression is... you need to find a way to just—" (Adam, 04:18)

• Huffman’s Eureka Moment:
  "Finally, he despaired of ever reaching a solution and decided to start studying for the final. Just as he was throwing his notes into the garbage, the solution came to him. It was the most singular moment of my life..." (Don quoting Huffman, 12:37)

• On Prediction and Surprisingness:
  "The meaning of a message is how surprising it is. So the letters where you got it the first try, not very surprising, and so there was actually very little information there. So they could be compressed very small." (Adam, 35:53)

• On the Never-ending Search for Compression:
  "Kolmogorov complexity... you can measure the information in a piece of data by the length of the smallest program that could possibly describe it." (Adam, 39:48)

• Modern Challenge:
  "LLMzip used a 13 gigabyte decompressor. But if you ignore that, it got 11.27 [compression ratio]..." (Adam, 46:28)

• Summary Insight:
  "If you understand the patterns in a giant swath of Wikipedia, how is that different than actual intelligence?" (Adam, 51:49)

• Comic Relief:
  "I learned a lot about compression, but I didn't get any Euros." (Don, 52:07)

---

Timestamps for Key Segments

• 00:00–01:48: Introduction & Hutter Prize overview
• 03:15–05:41: Live demo of basic compression; limitations illustrated
• 10:04–16:34: Morse code, Huffman coding, and information theory legends
• 16:34–20:45: Origins of ZIP/GZIP and Phil Katz's story
• 21:15–24:30: Arithmetic coding & compressibility limits
• 27:04–36:31: Claude Shannon’s entropy game and mind-as-compressor
• 44:02–46:51: The LLMzip breakthrough and the decompressor-size issue
• 49:03–52:14: Contest constraints, philosophical wrap-up, and the AGI-compression thesis

---

Flow and Tone

• Conversational, humorous, and accessible—Adam and Don use live coding, analogies, and storytelling to explain complex technical ideas.
• Frequent self-deprecation and banter: Don plays the “not-quite-an-expert”; Adam steers with nerdy enthusiasm.
• The episode flows from practical experiments to theoretical elegance, culminating in a philosophical insight into the interplay between compression, prediction, and intelligence.

---

Takeaway

By tracing the arc from early compression tricks to the vanguard of AI-enabled compressors, the episode reveals that compressing information well is fundamentally an act of understanding and prediction—in other words, a form of intelligence. The bounds of lossless compression turn out to coincide eerily with the edges of machine comprehension, making the Hutter Prize a strange but fitting battleground for the development of artificial intelligence.

Transcript

[A]

Hi, I'm Adam Gordon Bell, and this is co Recursive. And today I have here with me, which seems to be a trend. Don.

[B]

Hello, I'm Don McKay and I'm back.

[A]

So I sent you a text yesterday. What did I send you?

[B]

You found us a side gig that pays $8,000. Eight thousand Canadian dollars per 1%. And then I blocked you.

[A]

You blocked me? Yeah. So the thing that I wanted to talk about, I actually printed it out. It's a real contest. It's been around for about 20 years. And there's. There's a prize.

[B]

Losslessly compress the 1 gigabyte file NWIC9 to less than 110 megabytes. More precisely, create a Linux or Windows Compressor Comp Exe of the size S1 that compresses NWIC9 to Archive Exe of size S2 such that S equals S1 plus S2 less than L equals. Yeah. 110 megabytes. It's got like a whole bunch of them. It's like 110,793,120.

[A]

So it's. It's zipping a file. Basically. You're being paid to zip a file smaller than somebody else has zipped it.

[B]

Oh. And then eventually you'll run into problems because you'll have to invent some other compression algorithm that'll do it.

[A]

Yeah, but so the file is a gigabyte of Wikipedia data, and then whoever can make it into the smallest file and then reconstitute it gets money.

[B]

Yeah, I think I figured this out, though. It's middle out.

[A]

There's more.

[B]

Oh. You're eligible for a prize of €500,000. €500,000 times 1 minus S divided by L. Being able to compress well is closely related to intelligence, as explained below. Well, intelligence is a slippery concept. File sizes are hard numbers. The intention of this prize is to encourage development of intelligent compressors programs as a path to AGI.

[A]

It's a weird thing to say, the development of intelligent compressors as a path to artificial general intelligence. It feels like saying, you know, like this. This crossword contest will clear the way for world peace. Step one, really small zip file.

[B]

Oh, it's just. It's just a three step progress process.

[A]

I mean, we may be simplifying it, but. Yeah. So when I was a kid, zipping was very important. I think my first computer had maybe it was either 40 megabytes or 80 megabytes. Yeah, you would run out of space very quickly.

[B]

Well, if you wanted to take anything with you, you used to have to compress things down to put them on disk and then you could, you know, split the file across multiple disks, which was a pain.

[A]

Yeah, and I used Winrar for that.

[B]

I remember they used lha. Lha, Yeah, I used to freeze and thaw things.

[A]

Anyways, here's the thing. I want to try to beat the contest, right? I don't think we can get the whole €500,000, but maybe we can. Maybe we can get a 1% improvement and then we can make some money.

[B]

Cool. Yeah. I've never actually looked into how compression works, so this will be enlightening.

[A]

So I got my computer here, my MacBook Pro. Can you see this or. It's way too small.

[B]

It looks like you've done an LS in Linux on some files that are in a directory.

[A]

So there's nwic8 and nwic9. If I gzip it.

[B]

Have you not rehearsed this?

[A]

I never rehearse. I like to just. Let it fly.

[B]

Yeah, let it fly.

[A]

Okay. Okay, so here we go. So NWIC9, originally, this is our.

[B]

What was it originally? It is 1 gigabyte and they want you to compress it down to 110megs.

[A]

Where did we get to? So we got it down to 322 megabytes from the original 1 gigabyte. That's pretty good. We got. It's like a third of the size almost exactly. Right. We've taken it. So that's our first step. We just. We send this in, collect our money.

[B]

Well, I mean, you didn't meet the requirement. It has to go down to 110.

[A]

Oh, yeah, yeah, you're right. So we're at 322. We need to get it down.

[B]

Yeah, you need to get it down to. Was it 110?

[A]

So we got a ways to go.

[B]

Yeah, I mean, it doesn't look like you're getting close to the target.

[A]

Yeah. In fact, I did try to double compress and it doesn't. It didn't gave me anything. And here I'm using GZIP with negative 9, which I believe is the.

[B]

The maximum.

[A]

The maximum, yeah.

[B]

So you've reached the limits of the GZIP algorithm.

[A]

Yep. So we need to do something else. Okay, so the. I think the key thing to do is to just try to build our own compression algorithm. Cause I don't think we're going to win with just gzip. Seems like somebody might have thought of that and collected it already. The easiest way that I can think of to make a file smaller. Right. Like you just have a bunch of characters and some of the characters repeat. So the easiest thing I know of is just like when you have a repetition, you take it out. Because the whole idea with compression is. Yeah. You need to find a way to just.

[B]

You find patterns and then you replace those patterns with a symbol that's smaller.

[A]

Yeah. And like, it's each like compression algorithm because there's a bunch of different ones. Right. They each look and look for and are good at finding certain types of patterns. And if you actually don't have those type of patterns, then it's not useful.

[B]

Yeah.

[A]

So each compression algorithm is sort of like a bet on the type of output.

[B]

It's like a text file is. Will compress more than some complex binary or something like that.

[A]

Yeah. And like the RAW video frames of a video file can be compressed very well by like mpeg, but if you just took all those RAW frames and tried to run them through a zip, they might not work as well because the MPEG encoder actually understands the type of patterns that are in a video. Yeah, yeah. Like I, I'm showing you basically my, my little algorithm. What are you seeing?

[B]

Um, you have a function that you've written that runs through a byte array and performs some kind of transformation on it.

[A]

So I have a box for my, my little Python algorithm, and then sort of I can feed it input and then I can kind of get an answer or how that compresses. And then I have this big button that lets me run it against the data set from the hunterprize. Okay. So, so here's my first. Here's my first algorithm. Right. So it's just, it's just run length encoding. And then I'm going to run it on this string, which I assume must be in Wikipedia somewhere, which just says no with an exclamation mark.

[B]

So you're. You're going to replace all of the O's with another symbol that's less space.

[A]

Yeah, this is my attempt to beat the prize. Right.

[B]

But isn't that sort of what GZIP probably already incorporates?

[A]

We'll find out. Okay, so if I run it on my. No. Yeah, it changes it to this format. Right. So It'll say like one N and then 24 O's and then one exclamation

[B]

mark, which is less space than all of the 24 O's.

[A]

Yeah, yeah. And so the, the compression ratio on that is. It's four, three times smaller.

[B]

Yeah, which, that's good.

[A]

That's Good. But the. If I run it on. Okay. Like, here's Don laughing.

[B]

So just. Ha, ha, ha, ha. So now there's not quite repeat. Like, there's the same characters, but they're not repeated sequentially.

[A]

Yeah. Okay, so I run on that and I. Yeah, the output format now. Cause for every letter, I need to put the. The frequency of it. And now my. My compression ratio is. Well, I've actually made the document longer. Okay. So it's not looking good. But let's run the thing on the Wikipedia small corpus. Okay. So you can. It actually made the Wikipedia corpus larger. So we. Our. Our compression ratio is 0.53, so we're doubling the size, which is worse than our. Worse than our zip.

[B]

Yeah, the zip's 2.74 times and the record is 9.03.

[A]

Yeah. We got a ways to go.

[B]

You gotta. Yeah. Cause you've. You've made it bigger by.

[A]

Wrong direction. I think we're good.

[B]

Yeah. So it'd be like turning a 1 gig file into a 1.5 gig file.

[A]

Yeah. Now we know how to make files bigger. Okay, so I have another idea. The thing. The other thing we could do is instead of just looking at, you know, letters repeating, we could assume that text repeats.

[B]

Yeah. And it does, because we use words and we use the same words over and over again.

[A]

Yeah. And so my new idea is whenever we find a repetition of something, then instead of putting in that repeated text, we just put a pointer back to where it was.

[B]

So that makes sense to me.

[A]

Yeah. So in this example, which I would like you to sing, like the.

[B]

Row, row, row your boat gently down the stream and then merrily, merrily, merrily, life is but a dream. I don't think I have to read the whole thing, do I? People know row, row your boat, which has a lot of repeated words in a pattern.

[A]

So, yeah, I have my little algorithm. It's just gonna. Basically, it's like pointers from, like, C. Right. It's like. But in text. Whenever we have some text that repeats, we'll just say, like, use that. So if we run this on row, row, row your boat, we end up with something like this. So it ends up with row, and then it says repeat four to eight. So it's. It's taking the two rows out and replace them with a pointer back. And then it's got your boat gently down the stream merrily. And then the repeated merrilys become pointers back. And then it has. Life is but a stream. Oh, it's. Oh, it's Even catching part of letters.

[B]

Yeah, it doesn't know that it's a word.

[A]

Okay, very cool. It works even better than I thought.

[B]

Better than you anticipated.

[A]

Yeah. And then. And then it just ends with this. This final pointer because the whole thing repeats. Gives us.

[B]

I see. Like, so that whole verse is now a pointer that the second verse just points to instead of.

[A]

Okay, actually, we already ran it. So what did we get?

[B]

2.84 times. See, there you go.

[A]

We're getting somewhere. Suffice it to say that that is less good than gzip, but we're making progress.

[B]

Yeah, you're going in the right direction.

[A]

That is a more likely pattern than our just like, repeated specific letters.

[B]

You've widened your pattern recognition.

[A]

Yeah, yeah, but we can do even better. Which dates back in some ways to. To Morse. Morse code. The clever thing that, that he did. They don't all have the same length of dashes and dots. The most common letters, if you're doing a telegraph, are shorter. Are shorter.

[B]

Yeah.

[A]

Which allows you to compress things down.

[B]

Side note, SOS doesn't stand for anything. It was just the simplest pattern to remember.

[A]

Oh, because it's dashes and dots.

[B]

Yeah. People think that the, like, letters, SOS actually stand for something. Yeah. Doesn't stand for anything. It's just that the three longs and the three shorts are the easiest thing to remember. So it's the easiest pattern to transmit.

[A]

That's awesome. Yeah, I thought it was like, save our. Save our ship.

[B]

Yeah, doesn't stand for anything.

[A]

Okay, so the next thing we're going to hit is like an important idea. It's not mine, but an important idea in compression. So in the fall of 1951, there's this grad student, and his name is David huffman, and he's 25 and he's going to Ohio University, and he's two years out of the Navy and he has a class. And in the class they say, you know, you can take the final exam or you can write a term paper instead of. And you want to come up with the frequency, you want to come up with a mapping so that the most frequent things use the least amount of terms. But it's not just come up with a scheme like that, like Morris Code did. It's, you know, come up the provably correct way to take a bunch of text and figure out what's most common and give it a binary mapping. So that was the homework project. The professor's name was Robert Fano. And he doesn't tell them that as part of this assignment that this problem he put to them is one that he can't solve himself. And he was the professor. But it gets worse because he had been working with a colleague on this problem, and his colleague was Claude Shannon, who we'll talk about later. Claude Shannon invented information theory. He invented the bit, the idea that you could transmit information digitally. He. He, like, invented the whole field. Super interesting guy. He could not solve it either. He was a genius. And these two could not solve this problem. And he's like, you know what?

[B]

Give it to the student.

[A]

If you don't want to do the final, just prove this thing.

[B]

Just take this thing that we've struggled with through our career and figure it out.

[A]

I think you have a quote.

[B]

Huffman worked on the problem for months, developing a number of approaches, but none that he could prove to be the most efficient. Finally, he despised spared of. He despaired. He despaired of ever reaching a solution and decided to start studying for the final. Just as he was throwing his notes into the garbage, the solution came to him. It was the most singular moment of my life, Huffman says. There was an absolute lightning of sudden realization.

[A]

I always have this thing, like, especially if I'm working on a problem and I can't solve it, and then when

[B]

I put it away, your subconscious crunches on it.

[A]

Yeah. Do you know, did you ever have. When you were a kid, it was this thing, and it has, like, a whole bunch of needles. They're not sharp, but you, like.

[B]

Yeah, yeah. Caven got one of those. They're like. They're plastic now, not metal, and they're like, multicolored. But, yeah, it's the, the. All of the. The matrix of little pins, and you can push something into it and you can see the impression.

[A]

Somebody told me before, you know, your brain kind of works like that where, like, different areas, different thoughts, they get, like, activated. So you're thinking about something and it's sort of like pushing up on all these areas. If you're trying to brainstorm an idea, like, there might be something in the back of your head, and so that causes an area to, like, light up a little bit. You can imagine, oh, here's like, an idea around the topic, and so that needle goes up, but there's also all these other things going on, like the other things you've thought of and, and whatever. Right. So the, the idea is there and it's pushed up, but so is a lot of other things, and you can't.

[B]

Yeah, there's too much noise.

[A]

So you can't see it. But then if you walk away, you know, the other things you were thinking about, they all sort of settle down and then you can see like, oh, there's that idea. The, the other needles have fallen away and I can see this one is just jetting up a little bit.

[B]

Yeah, that makes a lot of sense.

[A]

Right. So his paper that he submitted became one of the most cited papers in computer science.

[B]

I'd like to see one's like, oh, I mean, I knew you could.

[A]

I did. I tell you that this is an unsolved problem and that Claude Shannon, the. The smartest guy that I have ever met, could not solve this.

[B]

I mean, you did.

[A]

Okay, yeah, yeah, we'll give you a pass on the class.

[B]

Cool.

[A]

So this idea though, Huffman, it becomes called like Huffman encoding. And I mean it's used everywhere. His paper that he submitted became one of the most cited papers in computer science. So this is used inside of gzip? Yeah, he never patented it or anything, so it was just like a concept. He, you know, ended up becoming computer science professor and leading a department. Probably helped that he had solved this great thing for his career. Right. So here's my version of that.

[B]

Betty Beat the Best bet.

[A]

It's like a tongue twister, but it does tie into all this. Right. If I run it, it's going to go through and find what characters are the most frequently. And probably not surprising. There's a lot of B's and E's in here. And so it ends up, oh, and T. Apparently T is the most common. I would have thought it was the B. You can see it ends up with this table.

[B]

Right. And it assigns it a binary code. 10, 01 has a four letter code.

[A]

And like the other thing that's happening here that's very valuable is we're not encoding any of the other stuff. We don't have to worry about and. Or whatever. Right. We only need to encode the things that are actually in the text. It's all, this is all capitals. We don't have to encode lowercase. Like we're only focused in on the most frequent characters.

[B]

Yeah. Including space.

[A]

Including the space, yes. Which we have. But yeah. How much smaller is this version?

[B]

So it says it's raw bytes. It was 8 bits slash for SIM and then it's 2, 7. So 66% versus raw.

[A]

So we're 66% smaller. Let's try to run our corpus. So when we run this against the Wikipedia standard, we've actually gotten a smaller file which Is good. Before we were increasing.

[B]

Yeah. 1.56% times.

[A]

Sweet. So we're making progress. We have a long ways to go before we beat the prize.

[B]

Get that sweet, sweet. €5,000.

[A]

Gotta get that sweet euros.

[B]

But I like how it's in euros. But like when you texted me about it, you did the conversion on my behalf to put it in the indollars, like, oh, euros. He's not going to know about that. So I'm going to do the conversion here. It's about 8,000 Canadian dollars.

[A]

I didn't know what a euro was worth, did you?

[B]

A euro? No, not like, offhand Like, I do know that it's about one and a half.

[A]

So the first algorithm we did was run length encoding, which was basically like. That didn't work well. Then we did the pointer one, which worked well.

[B]

So the pointer one actually did compress something.

[A]

Yeah. And so that was invented by two information theorists, Jacob Ziv and Abraham Lempel.

[B]

Do you know what year?

[A]

1977. That was? A long time ago. Right. And then the one we just did, the Huffman encoding, was obviously by this David Huffman. But you put those two things together, guess what? You get a more efficient algorithm that is very much the case. GZIP or zip files.

[B]

Okay. So like everything that's the common standard.

[A]

That's the common standard is the zip file. And the zip file just uses those two algorithms. So those two algorithms were combined into this compression algorithm, which I believe was called compression program, that I believe was called arc. And it was big in, like the BBS days, there was like a free version of this ARC archive builder. I mean, it was just the most common format. And then there was this guy named Phil Katz. So, yeah, in. In the late 80s, before, you know, the Internet, there was all these BBSs. And so compression was important because you just had dial up. There was this ARC format and it was, you know, it was usable by this company called C. Bill Katz lived in Milwaukee and he was kind of a really good hacker and he liked to write things in assembly. And so he wrote his own version of their archiver and he called it PK ARC with a Phil Katz archive.

[B]

Okay. Is that where pkzip comes from?

[A]

That's where pkzip comes from.

[B]

Okay, I put that together.

[A]

Yeah. And so it does exactly what ARC does. It does the equivalent algorithm, except because he handwrites assembly. It's like super fast. So people start using this PK arc, everybody switches to it on all these bulletin boards. So this C company Gets mad, right. He's just like replaced their thing with a better thing. Right. So they sue him because, you know, he's copied what they've code and released it. And it becomes this big court case. And I don't know all the details of it, but apparently they find that there is some comments in their source code where there's typos and then they get his source code and he has the same typos.

[B]

Oh, to try and prove that he just copied it.

[A]

That he just copied it and then made some improvements. So he loses the lawsuit and out of frustration, I assume he's like, well, f. This nonsense, he rewrites, right? It's just these two algorithms that we just went through. He rewrites it in like a, a text file from scratch and shares it with the world. So now there's an implementation of how to do this that is unencumbered and, you know, has no cost. And he kind of did it, I think a little bit of spite. But yeah, this became PK zip. And because the P and K was through Philip Katz and everybody just adopted it because it was free and also it was fast. Cause he was good at what he did and. Yeah. Did you have PK zip files? Yeah, yeah.

[B]

Of course.

[A]

I never knew what it stood for.

[B]

No, I didn't know what it stood for either. No.

[A]

Yeah. So because it was unencumbered, that's what made it popular, Right. Anybody could use it.

[B]

It was open.

[A]

Yeah, it was open. And so, you know, you get it. Used the WinZip. You get it. You know, when the, when the GNU Project is trying to open source things, they're like, oh, we can use this. Right? So that's where you get gzip, the algorithm. The, the combination of the two of them is called deflate, apparently. Yeah. And then his life kind of went sideways. He. I guess he had a drinking problem and. Yeah, he was found dead in his. He was found dead April 2000 in a hotel room in Milwaukee. He was 37 years old. He did not make it very far.

[B]

That's sad.

[A]

No, it's super sad. He was clearly talented at what he did, but I mean, addiction can be a challenge. But he left his mark on the world, right? Like, everybody's still using the software. So with what we have so far, we can compress a file and do pretty good. We won't win our contest. But like, if I have a file where a character is used a lot, I don't have a way. What's the way to say it Imagine this, right? So I have a 25 sided dice. Does such a thing exist?

[B]

Most often it's a 20 sider.

[A]

Funny side. Okay.

[B]

Then it goes up to. I think, I think you can get a 25. It's odd, but like, I think the next step up is like a 30.

[A]

Yeah. Okay, so we have a D20. We have letters on it, but all the letters are A except for one, which is a B.

[B]

Okay.

[A]

So I roll it, I get like a, a, a, a.

[B]

You got 95% chance of getting an A. Yeah.

[A]

And so if I make text like that, what we've created here doesn't necessarily help. Or there's a different way, which is called arithmetic coding.

[B]

Yeah.

[A]

So arithmetic coding has this idea. The math is a little bit more complex, but it tries to look at, why don't I just encode what's surprising? And so the way they do that for this is very simple. Here's the, here's the compressed format. What are you seeing?

[B]

It looks like it's just noting where the Bs occurred at which position.

[A]

Right. So you never have to say A, A, A, A. You just say there's a B, here's A.

[B]

And everything else is A.

[A]

And everything else is A. Which allows you to get below one bit per character.

[B]

Yeah, I mean that's the stats for this test data. But I mean like the test data is basically like a ton of A's and a couple Bs.

[A]

Yes. So this example is very.

[B]

Yeah, it's very specific to the scenario,

[A]

but it does come up in the real world where something is so common that you want to actually encode it at less than a bit of information. I can show you. Here's my results TABLE. Overall, here's our run length encoding. This is on the big file. So our run length encoding made the file twice the size. So that wasn't very good. Zip makes it more than a third of the size. Right. So 2.74. Uh, this is our arithmetic coding, which

[B]

is 3.96, almost down to a quarter.

[A]

It's funny. So I had this guy, Jan Colette, I had this great episode, right? And he was in France and he ended up building his own compression system that was better than zip. And it's called zst, it's the Z standard and it uses this arithmetic coding concept. And he was trying to explain it to me how you can store a character in less than a bit, which is what, what we just showed. Right. We're able to, to compress all those A's. It's easy for me to understand it in the A example. It's harder for me to understand how it works.

[B]

Context of an actual document, in the

[A]

context of an actual document, how it works. But the way it ends up working out mathematically is that you only use as many bits for a character to do with how surprising it is. And so I showed an extreme example where the A is not surprising at all. So all we're encoding is the B. It's able to use fractional bits to encode things. And so this idea had been around for a while. There was some patent problems that allowed people that made it problematic for people to use it. But he brought it to common usage by making an open source project allowing lots of people to do it. And he went from just being a hobbyist person building his own compression algorithms to working at Facebook. And this algorithm is now used everywhere. That gets us to here. Yeah, we are almost at four times.

[B]

Yeah, that would be like if you're down to almost a quarter. Right. So like around 250 megs.

[A]

Yeah. So we still have a ways to go, I guess. Yeah, I don't know. That's. That's where I run out of ideas. What do you got?

[B]

That's where you run out of ideas. Let's find somebody who's figured it out and bring them on the podcast.

[A]

Yeah, exactly. All right, Don, what do you got? I did the first part, you did the second part where we win.

[B]

Question mark, question mark, question mark, Profit.

[A]

Exactly, exactly.

[B]

I don't know. I guess a combination of approaches. I mean, it's either that or coming up with like a new idea on how to compress something. It seems like compression at its core is just recognizing patterns and replacing them with something smaller or another way to

[A]

phrase that that people have used, which at first I found confusing. The Huffman coding, Right. Is it's taking this idea, it's. It's predicting that there is a uneven distribution of characters. And the run length encoding, you hit that immediately because its expectations are for repetition. It worked very well on, on one thing, but on. On the rest, it failed. On a compression algorithm in some ways needs to be able to predict what the file format is going to look like.

[B]

Yeah. So if you're encoding something that's an English language document, then there's certain predictions you can make around the structure of that document. Because it's using English as a language,

[A]

you are onto something.

[B]

Sometimes I'm smart. Coffee's kicking in.

[A]

Yeah, man. The challenge is always that you're too smart.

[B]

I have to be just dumb enough.

[A]

Yeah. Because I bring you here to try to explain something and you're like, oh, yeah, what if they did that? You're like that. You're like Huffman with the papers. Like, I got. I think I got something here. They're like, jesus Christ. We've been working on this, and you

[B]

just did it in a weekend.

[A]

Okay. I have the next thing, though. So it's 1950, and there's Claude Shannon, right? And Claude Shannon has this problem that he came up with. So he had defined the bit, and he had come up with this idea about transmitting information. He worked at Bell Labs. What does it say? He proved that every stream of information has a floor, an irreducible bottom. The minimum in important information in it, which he called entropy. Right. So the theory says that there's a way to make things smaller. He was never able to figure out if there was a bottom. So he was kind of dancing around this idea of compression. Right. If you're transmitting information, some of it is important, some of it is not. Right. Like if we're talking on the phone and you miss a word, I probably still know what you're saying. But if I miss every other word, sometimes I'll put it together, but sometimes I won't.

[B]

Right.

[A]

And so his idea was, it's entropy. It's this transmission of information. He's an information theorist and that there's essential information you're transferring, and if we remove too much, it's gone. It's an easy idea. Think about. But I mean, the math around is very complex. So to figure out his estimate for this amount of information, he came up with a game. So we have here some text.

[B]

The Big Sleep.

[A]

The Big Sleep by Raymond Chandler. The biggest. Raymond Chandler. Are you familiar with him at all? He.

[B]

No.

[A]

He wrote like these, like, noir detective novels.

[B]

Oh, okay.

[A]

I think he's one of the. Like, there's several authors of them, but he used one of the most famous, right? Where it's like a smoky room and the names on the door and damsel in distress. So here's the game he comes up with. His wife is sitting where you are. He's sitting here. He reads her part of the sentence. So it says it was about 11. Now the game is you need to guess what the next letter is. And going back to where we were before. Just like our text documents, we're gonna. We'll limit it to, like, lowercase letters and a space. Cause we're just gonna reduce the domain to make it a little easier can you guess what the next letter is?

[B]

O for o'. Clock.

[A]

All right, let's try it. That's right. So you got O on the first guess, and he records these numbers.

[B]

Oh, okay.

[A]

So on that letter, you got it on the first. So what's your next guest?

[B]

I think it was o'. Clock. Wouldn't it be like an apostrophe?

[A]

So we don't have apostrophes because I just, I reduced it down to make it simple. C. Got it. L, L, O. C. Got it. Okay, and so you were able to guess all the letters on the first try there. But okay, what's next?

[B]

Space. A for at night?

[A]

Nope.

[B]

I for in the evening.

[A]

Oh, I think you got it right. The wild thing is how much little information is actually needed to transmit the English here. You were able to predict it because

[B]

of common sayings and phrases and the predictable letters cost.

[A]

One guess is what he says. A surprise letter can cost five or six. You didn't even get that high. You only got as high as two. But he says on a demonstration passage, a hundred plus letters of Raymond Chandler's book, the guess was correct a one about seven out of ten times. Okay, Like, I find this part hard to explain. What he's trying to, trying to say is there's actually. If you try to encode this information as like ASCII or binary or whatever, it will take a lot of numbers. But actually a lot of that is unnecessary. The. The actual surprising things are, are very small. You guessed almost all of these from the get, and it took you only two guesses to get to some of them. So here's his crazy idea. In 1950, this is the guy, by the way, who couldn't solve the Huffman thing. So what he said is, imagine there's a second Don. I put the same problem to Don number two. If I assume that Don will always guess the same order, I don't actually need to know what the text is here. I only need to know how many times I need to ask Don to guess.

[B]

Right.

[A]

And so I've just compressed using Don.

[B]

Yeah, but like, you were able to do this because you knew what the correct answer was.

[A]

So you're saying I wouldn't be able to decompress it. Okay, but here's the, here's the tricky part. He's saying, no, that's not the case. Right?

[B]

Yeah, I guess I got, I got lost to like, that's where, that's where I'm. That's where I'm struggling. So like, yeah, I get That I guessed correctly most times because I was given the beginning prompt of it was about 11. I could kind of extrapolate what the rest of the sentence was. And when I guess, you're able to check against an uncompressed version of the document. So tell me I was right or not. Otherwise, how do we know?

[A]

Yeah, so you compress the word o'. Clock. Given. Given it was 11. You compress the word o' clock to six ones. And so his idea is to. To transmit the word o' clock that it finishes, that it follows from 11. I could just write down those six ones. In the beginning, it was 11.

[B]

And then you'd have to have somebody like me on the other end.

[A]

You have to have a dawn on the other end that would be able

[B]

to do what I just did.

[A]

And we played the same game, but I don't actually know the answer this time. I know that when you say your first number, if, if you got the number two before, I have to say, like, no, try again. And that second number is the number that goes in. So I. The person on the other side doesn't actually need to know the answer. They just need to know how Don would guess.

[B]

Would guess.

[A]

Yeah, this is messed up.

[B]

That make. Yeah, that's messed up. But, yeah, I get it now.

[A]

But he's. I mean, he did this in 1950, so computers were pretty simplified, but he called it like a digital twin. You know, if we had a Don on the other side, an exact Don duplicate, I could decode the sentences without knowing what it was because I would know your guesses would be the same as his. In his thinking, what he's saying is he's. He identified a range based on a human, and he published this bracket, 0.6 to 1.3 bits per character. But it's, it's clever. He managed to figure out how he could use, well, a physical person's brain as a. As a compression mechanism.

[B]

Yeah. And it would be that specific person because each code would be individual to a person's thought process.

[A]

Because if it was a slightly different person, they might pick a different letter.

[B]

It'd pick a different letter, and it

[A]

would all be blown out of the water. But the interesting thing is, from his perspective, then all of these compression algorithms become prediction algorithms. Right. Because what you're trying to do is predict what comes next. And you're just really good at predicting English language because you understand it very well. So that means that our other algorithms, the way the Huffman encoding tries to predict things is by looking at all the characters and seeing which are most common, and that's its power to predict things, is it knows this is the most frequent thing.

[B]

Well, I don't think that the. I don't think that it's guessing. It knows what the answer is. It knows that T occurs as many times.

[A]

This is the confusing part. When you encoded that word, you came up with like, 1, 1, 1, 1, which meant it was your first guess. It was your first guess. But if you think of the Huffman table as a prediction, then that means by default, the very first thing it predicts is T for any answer. But if T's wrong, then it predicts the second one. E. Yeah, E. And then, like these ones, it takes a while for it to predict it, so it goes further down. So it ends up with a similar encoding as you, but it takes it a lot longer, I think.

[B]

Yeah, no, I, I understand. You take the. The table it came up with, but I thought that it was making this table so that it could replace the character with a different code.

[A]

It is.

[B]

So it's not making a prediction, it's doing a. It's doing a substitution, I guess. But you could use its same information as a prediction because, like, it's already kind of done the work to find out what the most common thing is. And instead of using that as its prediction, it's using that as a. Oh, I'll. I'll replace this with a small code. And the ones that are least, I'll replace those with a larger code. It's not using it to, like, predict the actual letter.

[A]

Yeah. But there's a way to view them that they are the same. Right. Another way. Is it saying, like, oh, you're just. It's treating you as a lookup table. When it's saying, when Don says one, it's like, well, what. How do I look up one in the dawn table? Like, well, you just ask them, like, given the sentence, what's the first letter you'd come up with? But it gets at this key thing. Right. Which was the guy on that Hutterprise, what did he say? Do you have that first quote, something about intelligence and compression being the same thing?

[B]

Being able to compress well is closely related to intelligence as explained below. While intelligence is a slippery concept, file sizes are hard numbers.

[A]

Yeah. So, like, he's saying that past these certain easy tricks, the way that you compress a file actually has to do with, like, intelligently understanding it, which is what you did as part of that game.

[B]

Yeah, Right. Because I understood English so I could Make a educated guess.

[A]

So the whole point of the prize is that if we pay people to compress files smaller and smaller, they're gonna have to come up with machines, algorithms that actually understand English language text.

[B]

Yeah. Cause I mean, if I didn't understand English, I would just be picking a letter at random. Well, not exactly, I guess, because some letters are more common than others. In English.

[A]

Yeah. Or you'd get a little better. You do the Huffman thing and you'd be like, well, the most common letter is A. So I pick a. That's. That's the relationship. Right. Earlier we had the, the tongue twister about Betty. So Betty was Claude Shannon's wife. And like he tested this theory on her. So Betty was the person who did the guessing.

[B]

Yeah.

[A]

And then he. He made the paper out of it. And then Claude Shannon is a super interesting guy. I should do an episode on him. But he built a flame throwing trumpet. So when he played the trumpet, it shot flames out of it. He built a machine that had a button on the top, and when you pressed the button, a hand came out and then closed.

[B]

There's a, there's a. Yeah, there's a toy you can get now.

[A]

So I think it's based on something he made. I mean, his was very simple. It was just like a switch, I think. And when you flip the switch, like a hand would come out.

[B]

Oh, it.

[A]

Flip it back.

[B]

Yeah.

[A]

But yeah, from this came the concept of entropy or surprise. The. The thing that he got down in information theory was like the meaning of a message is how surprising it is. So the, the letters where you had to. Where you got it the first try, not very surprising. And so there was actually very little information there. So they could be compressed very small. If you had to guess a lot, then that meant that that character was surprising. And so this notion of surprising puts a cap on the size that you can compress things. So our aaaaaas and then occasionally a B. The A's were never surprising, so we only had to encode the B's.

[B]

Like he. In 1951, he figured all this out because he didn't have YouTube. He had to sit down and think and think about something. He couldn't just like scroll a bunch of silly videos.

[A]

Yeah, I mean, I think that he might have been smarter than the average person in general, probably. I don't think if I was back then I'd be like, oh. But okay, here's the crazy thing, right? So this is my benchmark of all the things we ran. So run length Encoding. The actual record for the Hunter prize

[B]

is under a bit, right?

[A]

It's under a bit. It's at zero. But in 1950, Claude Shannon, he's like, I think this is the range on English language. And his range was 0.6 to like 1.2.

[B]

He was like, that's the floor. You can't get any lower than that. Is that what he's saying?

[A]

That's his theory about English language and the amount of information that's embedded in it based on human's ability to pattern match. But, like, it seems to be holding. Right. Nobody's gotten past his 0.6 intentionally. This AGI guy created this Hutter Prize. He's trying to say, can you make something that will understand English? Because he said, oh, it's really hard to measure how smart something is, but it's very easy to measure the size of a file. Like, it's very definitive. And. And so, like, if you can get past all of these levels and it keeps getting smaller and it starts getting towards Shannon's range, that means that whatever's doing that tipping must understand English language.

[B]

Yeah. It must be very smart.

[A]

Must be very smart. Turns out compression is very related to intelligence, at least in this case, where you're compressing English language based on your

[B]

ability to predict what the next letter is going to be in the English language.

[A]

Yeah. But I think I found a way around it so that we could win our contest.

[B]

Let's get that money.

[A]

Yeah, let's get that money, man. I made this file. Basically, I generated a file and a compressor for it. And this is like a sample of the file. We could open it, but it's just 4.1 megs. Yeah, it's 4.1 megs. And it's full of just like random hexadecimal. Yeah, I mean, this is in hexadecimal, so it's not text.

[B]

Okay.

[A]

And then I, I tried to compress it, so zip file compressed at 0%. In fact, it got a little bit larger arithmetic coding, like the zstd, got it to zero. But then my algorithm. Here's where we claim the prize. How did my algorithm do it?

[B]

Got it a lot smaller that 486,000 times. Well, there you go. The, the, the question mark, question mark, question mark.

[A]

Algorithm. Yeah, but it's a. It's a trick, obviously.

[B]

Right.

[A]

I. I used a random number generator and just generated a whole bunch of binary that I wrote to a file.

[B]

Okay.

[A]

And then my compressor, basically, it takes that number, it finds what the seed is. That was used and it writes that to a file. And then. So all my decoder does is read what the seed is and run the generator function again.

[B]

But it's a random generator function from the same seed. Yeah. Oh, well, that's how Minecraft works. And anybody with the same seed can generate the same Minecraft world as long as they know the seed.

[A]

So this is the same trick I've done here. They. I've made a file full of Minecraft,

[B]

already figured it out.

[A]

I can reduce it just to the seed.

[B]

Yeah.

[A]

And then I can re expand it to its seed. And so it. Out of all of the possible data in the world, there is like one very specific set of data that this thing dominates. It makes it like it's very hyper

[B]

specific to one thing. If you could make one that's hyper specific to that one file, we can win.

[A]

It's sort of an example, you know, if you just had this file and you tried all these algorithms at it and you saw it never got smaller, you would think, oh, this data is actually not able to be compressed. There's no patterns in it. But in fact there is a pattern, you just don't know what it is. So Kolmogorov complexity, invented by Kolmogorov. It has this idea that you can measure the information in a piece of data by the length of the smallest program that could possibly describe it. So in my example here, right, I have this crazy amount of seemingly random data, but actually it can be described by this program that's just like use generator seed89. But that's true of a million things. But it can also be shown that, like, there's no way to figure out if you've determined that that's the shortest program. You just can't know. But there's no way to prove that it doesn't exist. But it sets a bound on things, and that could be your compressed form. Like, instead of coming up with a special format, you just send over the program that if Ron reproduces it reproduces the entire document. But this is all going somewhere, I promise you.

[B]

Does it end with me getting €5,000?

[A]

Well, we gotta split it, don't we? Or.

[B]

Yes. No, that's true. That's true.

[A]

So there's a. I did an episode about this guy named Sloot. It's a great episode if. If you haven't listened to it. But Sloot had this idea for a movie playing system so people could watch movies in their homes. It was like in an earlier era. And he said he could compress videos you know, into. I think it was eight kilobytes. That would be the. The size of the movie. He was trying to explain how his invention worked. Cause they're like, yeah, you can't store a movie in eight kilobytes. Like, it's just not possible. And he said, oh, it works like this. If I were to send you a picture of the Mona Lisa, there's a lot of data there, you know, a lot of different pixels. But if you and I had the same art history book and I wanted to send you the Mona Lisa, I could just say, like, hey, look on page 76. And there it is.

[B]

So his. What was his source, though, for movies? Just, like a collection of pictures.

[A]

Yeah. I mean, there's not enough data to, like, encode all movies. It's actually like a clear problem in compression to say, like, eight kilobytes, like, even as a number is not that big of a number. There's more movies than even. Even if you just were storing, like, where to go get the DVD and put it in your machine, like, you would run out of numbers. And he had some demos. He had, like, some sort of home entertainment system where you could take a movie and you could play it from this eight kilobytes. And it became this big thing. People invested in it. And then, you know, he ended up dying. And they never found how the system worked. But the idea makes sense, right? He's saying, like, if we have some shared information, then I don't need to transmit every single detail because we share it. And the, you know, the dawn thing sort of does that. Right. Because you're similar to, like a cipher.

[B]

Because a cipher is. You don't understand this text, but if we both share a common key, then we can reconstruct what the message would be.

[A]

Yeah. Like, I think it is. Right. But it presents a problem to the whole prize. If Wikipedia is already in the system.

[B]

Yeah, we'll just store, like, the link to the Wikipedia URL. I feel like they have to have something in the rules that prevents that from happening. They're not going to give you money for that.

[A]

Specifically. What they have is that it was that weird math that you had at the beginning, which is the program. Its size counts too, at the size.

[B]

The size of your. Of your program. S1 compresses the file to the archive exe of the size S2. So, yeah, it's included.

[A]

Yeah. So if your program gets too big and complex, like, you need to make that up in the advantage of compressing. So if we're Thinking of this guessing game we did where we had some text, and then we have to guess what the next words are. Can you think of anything that's good at that?

[B]

Like all of our phones?

[A]

Yeah, yeah. Like predictive text. And predictive text does, you know, it has a dictionary of some sort that's shared. Right.

[B]

Because it knows English and it knows the. It knows grammar to a certain extent. And it's all been kind of programmed into, like, this common database that they all share. So whenever you're typing, it makes a predictive suggestion.

[A]

Yeah. So then whatever the size of that dictionary is. Right. You need to pay that off in smaller compression or. Yeah, like a smaller file. But what else?

[B]

Like, what else makes predictive decks?

[A]

Yeah.

[B]

AI.

[A]

Yeah. And LLM is exactly this, where you could give it. Here's the characters I have so far. Like, give me some more. And so his idea. I mean, the. The. The contest is 20 years old, but this is what's super cool to me. He thought, hey, this will reveal something about AGI and human understanding. And I assume people were like, due to it, we're just zipping files. I don't know what you're talking about.

[B]

I mean, yeah, that's exactly what I think I said at the beginning of the podcast. It's like, yeah, it's just zipping files. I mean, it's probably something that's required. Right. In the grand scheme of things, but it's not gonna be something that directly relates. And now it does.

[A]

Yeah, but here it does directly relate. Right. So the. This research group used an LLM to. To make a zip file. So they made something called, like, LLM Zip, and instead of using this idea of, like, a dictionary of common words, they just had an LLM in there. And so you can understand exactly how this would work. Right. So they're using. Instead of having a don, they have an LLM, they have an LLM. Do it. So they can say, like, hey, we have this text. Guess the next character, and when it's right, then that's easy. Right. And when it's wrong, guess again, guess again until they get it. And if the LLM is good at predicting that type of text, they now have a dawn on both sides that can decode the thing. And this LLM zip was able to beat, like, all kinds of records. And I think it did well on image compression as well, so blew all kinds of standards out of the water. Except the problem is that that LLM, like, base file is like gigs and gigs of data. Right.

[B]

Okay.

[A]

It's like 17 gigs of data or something.

[B]

So like the S1 in the algorithm there would mean that they wouldn't get any money.

[A]

So they wouldn't get any money because of the size of it. But what they, they did find that intelligence is helpful for compressing things. Like, they proved that Claude Shannon's theory works out. So we're looking at a table of different compression sizes. So our run length encoding made files bigger, zip files made it 2.8, 2.74 times smaller arithmetic coding, which was our ZST. It gets almost a 4. The record is here at 9. This is LLMzip. So LLMzip used a 13 gigabyte decompressor. You needed to give it one of these early LLM models. But if you ignore that, it got 11.27.

[B]

Okay, so it's by far the smallest.

[A]

Yeah. So it made the file way smaller because it has way more knowledge of English language. It's like a. It's gone. It understands how English language works. Maybe not as well as you, but

[B]

if you need a 13 gigabyte file to decompress a 1 gigabyte file, you haven't really gained anything.

[A]

Exactly. Right.

[B]

You've just kind of like offset all of that information inside your decompressor.

[A]

But there's all kinds of ways that you could think it would be useful. Not for this contest, but I mean, yeah, you could. We could all have a 13 gigabyte file on our machines and use it to compress and decompress anything English text. And it would.

[B]

Yeah, if it was a common thing

[A]

that everybody had, it would do very well. But you'd all have to have the same version. Right. It has to be the digital twin. So the. This guy made something called TensorFlow Compress. So TensorFlow Compress, it doesn't really violate any of the clear rules. It takes the Wikipedia file and as it's decoding the file, it trains an LLM on it and then it uses that LLM to do the trick. The problem is it needs a very expensive cluster of GPUs and has to run for like days.

[B]

Oh my God.

[A]

And so, and so it violates the theory, and it's very impractical to take your 1 gigabyte file and to spend.

[B]

It'll be a couple days.

[A]

Yeah, seven days on hundreds of thousands of dollars worth of equipment to decompress it. But. So the current record for the contest is this CMIX, and that's the one that's at 9. And in it, it uses this same idea of trying to predict what the data is, but in much simpler fashion. And it has actually 2000 different little algorithms that try to predict what character is going to go next. And then it has them all vote. And so some of them are really good at understanding like Wikipedia markup language, some are good at understanding this or that. And they all vote. I don't know. I went deep on this Dom. There's people who say that this Hutter guy should change the prize because, you know, he came up with this test

[B]

for AGI to like modernize it.

[A]

To modernize it because these limits that we Talked about, like LLMzip doesn't pass the test because the files are too big. But you could imagine changing, right? You could imagine that he says, okay, here's. I have 10, I have 10 separate 1 gigabyte files of Wikipedia. I'm only going to give you one of them, run your thing on that and then all test it on all 10. And like the cost amortizes over all 10, right? No, I guess that wouldn't work because it's 13 gigabytes.

[B]

If the amount of data that you're using is bigger than 13 gigabytes, then

[A]

you might be able to pay it off. Right.

[B]

Like if it's like a petabyte of data.

[A]

We're creating AIs now and they're just at a different scale. Like there's not one CPU, there's like thousands of GPUs, and it's not, it's petabytes and petabytes of data.

[B]

Yeah.

[A]

And you came up with this very cool, very specific, can't cheat it test because a file size is a file size, but you put the constraints as such that like, nobody takes it seriously.

[B]

They're outdated.

[A]

Yeah, yeah.

[B]

No, I think the modernizing the test would be right because then it gets people thinking about things in the modern context.

[A]

And when we, when we talked earlier about the pre training wall and how the LLMs had consumed all of the Internet and they were out, you know, this is in some ways a different view of things because here he's saying like every piece of data counts. You can't just use all of the Internet to answer this question. You need to learn the structure from just this. And every extra byte you add of looking things up is cost you. So he's on the other side of the coin of like, what is the like maximum information that we can suck out of this data? Yeah, same as that. Claude Shannon was saying, like, what is the minimum we can send and extract the maximum from, like, this is compression turns out to have a lot to do with extracting the maximum amount of data from the minimum amount of instructions. It's also the challenge just that, like, it's very hard to explain why getting better at file comprehension has to do with intelligence. It's very obscure. Right. Or obtuse. Actually figuring out how to make a file smaller somehow bumps up against intelligence. Like, that's, that's a very odd thing. I don't know.

[B]

Initially you wouldn't come up with that, but then after you explained that, oh, well, we can actually make things a lot smaller if we just knew how to predict what would. What would be the next, you know, the next piece of data.

[A]

Yeah, I.

[B]

And you're using the brain, right? You're using some kind of intelligence to come up with. To make the prediction.

[A]

Yeah. And I don't think we're actually going to make any money, Don.

[B]

No, I didn't think we were going to make any money when you texted me.

[A]

It's a, it's a, it's a ruse. I brought you over here because the thing I wanted to talk about was this idea. Yeah. That compressing documents is basically predicting and that, that requires intelligence. This is why the guy running this Hutter Prize is not, in fact a high performance computing guy. He's a AI person. And so he said 20 years ago that he would put, yeah. Half a million dollars on anyone that can compress Wikipedia because he had already

[B]

come to the conclusion that a requirement of being able to compress English Wikipedia would be some kind of command of the English language, some kind of intelligence behind it.

[A]

Yeah. In fact, he was making this bet, you know, that compression and understanding are the same thing in a way that to make a file smaller, you need to understand the patterns in it. If you understand the patterns in a giant swath of Wikipedia, like, how is that different than actual intelligence?

[B]

It took me on this wild goose chase. I learned a lot about compression, but I didn't get any Euros.

[A]

Right. There's no euros involved.

[B]

Very upset that we didn't get any money.

[A]

But I don't know, it's super cool. If you haven't listened to the two episodes, one about Sloot, one about, one about Jan Colette, you know, one legitimately rocked the world of compression. One, you know, thought he did. And. But it's cool how we can take just an idea that seems simple and something you use every day, and if you start pulling on it, it feels like in every area, if you, if you look into it, there's actually a surprising amount there. And it will connect to a lot of other things. And there's just areas you can learn about.

[B]

I don't know.

[A]

Maybe I need a better hobby. Sorry, we didn't make any money.

[B]

No, it's fine. I got a coffee out of it. That's good enough for me.

[A]

Yeah. And until next time, thank you so much for listening.