← Creator Science with Jay Clouse
Creator Science with Jay Clouse
#253: Mark Shust – Will AI kill our course businesses?
00:55:17
What we’re doing to future-proof our course businesses
Loading summary
Transcript46 lines
- [00:01]
Creator Science Host
You've probably heard the phrase you're the average of the five people you spend the most time with. I don't know if things are quite that simple, but here's what I do believe. The fastest way to grow is to be around others who are pushing themselves. And that's what our membership community, the Lab, is all about. The Lab is an out of the box peer group of creators like you who are pushing themselves to expand their reach and generate more revenue. Our standard in VIP tier is application based and looks for creators earning $10,000 per month or more. On average, our Standard and VIP members earn nearly $37,000 per month. But don't worry if you're not there yet. You can join our Basic Tier today without an application. Inside the Basic Tier you get access to all of my educational material as well as my personal monthly behind the scenes videos where I break down what's happening in my own business. My courses in Creator HQ alone are worth $1,600 and you can join the Basic Tier for less than half of that. Above all, the Lab is all about the people. It's the most active community I'm a part of and I'm constantly blown away by the generosity and speed members jump in with to help each other. And if you don't believe me, maybe you'll believe Kat. It's incredible to have a space to learn alongside other people, sharing best practices, asking for help, giving help, lifting each other up. It really has helped me to feel a sense of camaraderie and a sense of momentum as well. Later in this episode, I'll tell you more about our upcoming two day event exclusively for members of the Lab. But for now you can learn more and join us@creatorscience.com Lab. The more I use AI, the more I'm faced with the question, is AI going to destroy my business?
- [01:35]
Mark Schust
That's like the first thing I asked and I got a funny like response like, yeah, it's going to pretty much replace it.
- [01:41]
Creator Science Host
In this video I talked to Mark Schust, one of the first creators I saw becoming an adapter two years ago. We compare notes and predictions about what a future of AI means for creators and how you can prepare for it. And now enjoy this conversation with Mark Schust. You were somebody that got started playing with AI much earlier than I did, earlier than most folks in the Lab were doing. And so I'm curious if you were always like, this is opportunity, or if there was a point where you're like, this is scary. Let me see What I'm dealing with.
- [02:09]
Mark Schust
Yeah, I've been following it since pretty much GPT2. It caught my radar. Something just interesting. It was more of a toy, though. You can write, like, word limericks, but they didn't sound good. And then three came out okay, it's getting a little better. It's starting to be a little more intelligent, but it's still making tons of mistakes and it can't really write anything that well. And then I think three and a half came out, and that's when the game completely changed. With ChatGPT, it's funny because it's basically the same version as GPT3, but with a different UI. And that UI changed everything because it allowed you to interact over and over again with the AI. Ever since then, it's just morphed into. It keeps improving, it keeps compounding, it keeps getting more intelligent. Now there's reasoning models out there that can actually think and think through and process things before it generates a response. So, yeah, it's just amazing how far those have come since the early days.
- [03:07]
Creator Science Host
So was it like a toy for you in the beginning? Because you're obviously an engineer. You have, like an engineer's mind, and so you probably could see some fun aspects of it. But I think a lot of people sitting here today, teaching different disciplines in the creator space, they have a certain level of fear around it, and that's actually preventing them from trying it. I think in a lot of ways, I think fear for a lot of people is actually this friction, this barrier that makes them kind of a denier, that change is coming, which I don't think is useful or helpful. We'll talk about that, but it doesn't seem like at any point you were worried.
- [03:40]
Mark Schust
I've been worried since I saw three get better and three and a half get better.
- [03:45]
Creator Science Host
Okay.
- [03:45]
Mark Schust
Well, I wouldn't say worried, but it was like a cause of concern. Right. I don't think AI is going to take over the world or anything like that and lead to total chaos and destruction, but it was definitely kind of a toy when it came out, but it was. It was like a fun toy. And that's usually, I think, how some of these innovative things start happening there. It's like it's on the edges with developers and they start tinkering with it. It seems like everything starts with developers and then progresses up eventually, because our whole society is really based on tech. I do see this mindset, though, and maybe it's because it started as a toy, but even with developers, I see a Lot of pushback, saying, oh, it's not that smart, it's not that intelligent, it can't think for itself. And yeah, we've heard that over and over again ad nauseam. That may have been true a couple years ago, but it's not true anymore at all. They're sort of stuck in this limiting mindset of, yeah, maybe they're scared, maybe they're afraid, hey, it's going to take my job. It is taking jobs, it's taking tons of developer jobs, at least on the lower end of the spectrum now. But now it's starting to move up, upward on the hierarchy. There's some weird vibe going on with developers. You're either all in with it and you're letting it think for yourself, which probably isn't good at all, or you're on the other end of the spectrum where you're completely avoiding it and you're just spending tons of time writing boilerplate code that would be better off spent thinking and solving about specific problems. How that that whole dynamic is working out is really, really interesting. I've been using it myself to code up tons of stuff. I'm building a custom course platform right now, and I know, and I'm familiar with design patterns and architecture patterns. It's writing really good code if you know how to prompt and work collaboratively with the AI. And I think that's the key, which.
- [05:32]
Creator Science Host
I'm guessing you've only learned to do through a lot of trial and error because as you said, you've been doing this since GPT2. So I'm guessing a lot of your comfort in learning how to write with it has come from direct experience. Is there anything you pay close attention to outside of your own trial and error that's teaching you how to prompt and communicate with the AI?
- [05:56]
Mark Schust
Yeah, it's purely working with it. I definitely have my 10,000 hours in a couple years. I've been using it constantly on the side. So I think that's how you have to sort of see things, sort of treat it like a toy, experiment with it and find out what its limitations are, find out how you can talk to it, how it responds, look at the results, you know, parse it in your head and really determine, hey, why is it giving me that specific result? It's usually because I haven't explained something properly. So you can almost treat an AI like, if you don't feed it any context or any other information outside of your current discussion or conversation, you can almost think of it like a stranger on the street. I'M walking up and I ask it to do something and it might just take the most generic type of response that it thinks about rather than actually thinking about your own situation and scenario. So you definitely have to work with the AI to figure out how how it responds and how you can get better results out of it.
- [07:02]
Creator Science Host
I love that frame. Think about it as as if you're walking up to a stranger on the street. That's so good because you're right. You walk up to a stranger on the street, you ask it like a one line prompt. It's gonna give you some response, but it's not gonna have much depth or nuance and probably miss the point of what you're trying to do. But the more information you give it. This is something I've been learning the prompt there seems to be like next to no limit for how long and specific your prompts could be to get the result that you're looking for. Curious to hear as you've played with it, what has changed about your prompting? Have your prompts gotten longer? Have they gotten more specific, less specific? Maybe I'm wrong and there is a limit and there's kind of parabolic but curious to hear your experience.
- [07:44]
Mark Schust
It's really interesting. The answer is really nuanced. It depends on what model you're using, what temperature settings. I'm working directly with the API in a lot of my conversations, which means you can control some backend functionality like temperature and how long it thinks. I start off with really simple system system prompts or system Instructions I think ChatGPT calls it played around with really simple ones. You know you can the models are getting smarter so depending on the model it may know how you how it should respond based on that really small system prompt. But other models aren't as intelligent or like One of the best use cases I've had is for a personal picture business and personal and business coach that I've set up an elaborate I had a 20,000 character system prompt, knew everything about me, everything about my business. Well, my goals and aspirations were everything gave really good responses. But I noticed as I kept adding context to that prompt it started missing certain things in the prompt. And in AI it's called like a needle in a haystack problem where if you give it so much context it'll sort of and you ask it something, it won't completely understand everything in that prompt. So I actually reverted and made it more simple and I actually wound up getting better responses out of it. And it definitely depends on the type of LLM that you're using though.
- [09:13]
Creator Science Host
After a quick break, Mark and I talk about why AI may destroy online courses. So stick around. We'll be right back. And now back to my conversation with Mark Schust. You use AI for personal and business coaching and I have also experienced either coaching or learning. I am. So I enjoy the process of talking to AI and talking through things because in my experience, AI is infinitely patient. Judgment free meets me where I am. And through that lens, it makes a perfect teacher if you as the student kind of know how to interact with it. Because I think there's still kind of a gap like in the real world today, great teachers kind of know how to judge what a student needs and teach to them. But I feel like at this moment in AI, it's a lot of the responsibility of the user to help give the context of how we need to be taught. It's like an interface problem. But it strikes me that that's probably a solvable problem and there's a future where AI is just an incredible, bespoke, personalized, nearly free teacher to each individual. And for creators like you and I who deal in education, that's a confronting possible reality. How do, what is your reaction to this? Are you feeling the same way or do you see it differently?
- [10:40]
Mark Schust
Yeah, it's funny, once I had that coach system prompt, one of the first thing I ask is, is AI going to replace my business? It's like the first thing I asked and I got a funny like response like, yeah, it's going to pretty much replace it. And I was kind of terrified since, you know, I think everything's evolving and I think the whole educational system is evolving. As content creators, I'm a course creator. I'm used to developing really comprehensive curriculums with tons of lessons and it's hours long. I think that whole concept of the curriculum is either dying or changing. People's attention spans are getting shorter and shorter, even not only on the learning side, but also on the teaching side. A lot of even school teachers have issues and they may love their job, but they just, you know, can't, can't progress in their field. So I think learning is going to start becoming integrated into everyday, everyday work environment. So it's not going to be a separate course platform that you go to or a certain curriculum that, that you're following. It's going to be collaborative as you're working directly with the AI. So rather than having a different, different course platform or different, it's just going to be integrated in your, in your working and doing so. It's funny, Mark Twain had had a quote that it's like, I never let schooling interfere with my education, right? And that's kind of how I see it. Like, I've always loved learning, but I've always hated school for some reason. And I think it's just because I'm not working on something that I'm going to be doing either right now or even years from now. It might be something completely irrelevant. And I think AI is going to morph into more rather than learning or doing, it's going to be learning and doing. It's going to be just mirroring sort of real life. So as even a developer programmer, as you're coding something, you may not know how to do a certain thing. And you could just ask the AI directly as you're coding something live, rather than needing to sort of learn all this and fill your brain with stuff that you may not, may not even need to know. So it's more of a just like a just in time learning capability to.
- [12:58]
Creator Science Host
Describe or to try to illustrate what you're describing. I'm not an engineer, but I recently took a course, ironically enough, about how to code using cursor, how to make an app using cursor. And if you're listening to this, I want you just to imagine like you have a window and anything that you're trying to do in that window, you can highlight a piece of text and say, explain this to me. And then the AI explains what is going on in that window. So if you're coding and you write something and you get an error code, you can highlight it and say, what is failing here? And not only will it explain what is failing, it will suggest improvements to make it better. I agree with you that I think that is the type of experience we'll see in a lot of the work that we do. Any, any tool we use. If something isn't quite working the way it is, it's like having a co pilot or somebody sitting next to you who really understands this, that you can just turn to and be like, what? What is happening? Why isn't this doing what I want it to do? And that interaction though, happens in private. You know, you're not sitting in an office environment and shouting out loud for your coworkers to hear, why isn't this working? I'm bad at my job. It's just like, hey, this isn't doing what I want to do. Can you quickly give me some personalized, private, very patient help? And as a course creator, when we make curriculum as you're describing. I think a lot of what we do is imagine all of the different starting points people may have or the different hangups they may have. And we design our curriculum to try to overcome multiple objections for different avatars, anticipating different problems people might have. But to the individual taking that course, then there's this unnecessary sort of fluff of stuff that is not necessary to me because I don't have that objection. I'm not having that issue. And so it's not the most efficient or personalized experience. And I feel like AI education is kind of promising this very personalized based on exactly what you need. Exactly where you are here is the most efficient means of knowledge transfer. And by the way, you can respond to me directly and I'll clarify anything that is confusing in this moment.
- [15:00]
Mark Schust
Absolutely. I think we're all familiar with course completion rates, right? And how poor they could be. Someone might be a text learner and learn by just reading tons of articles. They might want to watch a video lesson or they might want to even see something just in a visual image. So there's, there's different ways. And when AI starts getting integrated with, with everything and how we, how we learn, it can learn the style and preferences of students and adapt and respond in different ways. So not everything has to be even a chatgpt window or a chat window. Right. It can reply and hey, do you understand this? And then send you an image explaining it. And the, the image generation of AI is getting incredible. It can create complex nuanced charts and diagrams like infographs even, and in seconds it can cater the, the exact lesson to every individual student. And like you said, it's Infinitely patient is available 24 7. The cost is going to nothing, just over time. I remember even getting started, I was spending probably over a thousand dollars a month. When it got started, I was kept grumping and yeah, I froze to tier five really quickly. Right now it's, it's like under $100. It's crazy for almost unlimited prompting. So the cost is just going to nothing. So it's being even more accessible to everyone else around the world. That's what that means. It used to be so expensive and now it can reach anyone across the earth and learn its preferences. And Infinitely Patient, it's just a tremendous teacher.
- [16:35]
Creator Science Host
I'm going to give another example because my hunch is a lot of people listening to this episode have kind of kept things at a distance and they haven't experienced the things you and I have. So this is going to seem obvious to you, but just to give someone else another example. I am a notoriously awful cook. I have next to no skills in the kitchen. And it's something that for years I've wanted to improve because especially now that we have a baby, my wife has a lot of pressure on her. And to cook every night, that's just not a fair expectation. It'd be really nice if I go into the kitchen and cook something, but the pressure I felt or the discomfort I felt in. Okay, I've got to go to Google. I've got to search for a recipe, I've got to figure out what we have in the house, and I've got to go make a shopping list, go buy that stuff, then come back and cook it according to the recipe. But now what I've done, I found a custom GPT around cooking. And I will just talk to it and say, hey, I want to make dinner for my wife. Here's some of the things we have available. We have some ground beef, we have some cheese. What are some recipes that I can use these as the main ingredients and have a little bit of spice. Then it'll give me like five examples and I'll say, okay, this is great. Can you give me a substitution for this ingredient? Yeah, here's a substitution for that. Okay, great. We don't want to have this involved. Okay, great. Then I'm going through the recipe and it will say, preheat the oven to this degree. Okay, I turned on the oven. But should that be bake or broil? And you can just get this real time interaction at the moment that you need it that you can't get from like a static recipe. It's just been a wonderful experience. And one morning I made a frittata. My wife came downstairs from bed and she was like, what happened? How did you do this? Did you do something wrong? Like, what are you asking for forgiveness for? And it's nothing. I just had the ingredients and I had someone who could basically teach me to cook that morning in real time. It's just a wonderful, wonderful thing. So the question on my mind, obviously on your mind, probably a lot of people listening to this. For people who do create education, like you and I, what does that mean for us? Is it going to replace us? Have you found a way where you think, actually, I think I will work with the AI and continue to offer education. What does it mean for your education business?
- [18:43]
Mark Schust
I'll see a question. How good was the meal? It was great.
- [18:46]
Creator Science Host
It was great.
- [18:48]
Mark Schust
I tried that too. And had an awful. Yeah. The ingredients just didn't. I was trying to make a copycat like chipotle barbecue thing and it just, it gave me like so much, so much cumin. Overwhelming. So.
- [19:08]
Creator Science Host
But well, good to know. I've been keeping a log in notion we rank every meal because part of this is I'm tracking this so I like in creator's queue, I'm tracking how many meals have I cooked this month because it's part of my KPIs and we rate them and everything has been rated super well. It's been like really good. But I imagine it could miss. Just hasn't for me yet.
- [19:26]
Mark Schust
But either way. So that again that was on objection that most people might have immediately like oh, the meal wasn't good. Right. But it could have been great. And is it going to get better over time? Like is there any proof that these AIs or alarms are going to get worse? That's, that's the question you have to ask. And then how much better are they going to get? But look at the graphs of a year ago even. How has AI progressed from a year ago to now? Right. It's been tremendous, at least in coding. Coding wise it's been unbelievable in writing. Like even Grok the Xai AI is incredible. The responses that it gives and it did that in just such a short period of time. So I see things getting just tremendously better. And if you're avoiding this, you're probably going to be in trouble. So if you're a cook, how you can use that, like that's probably a threat to your business or you think it may be, but it's not. Right? As a world renowned chef going to a restaurant. Right. We're not going to want robots cooking our food anytime soon, but you can use it as a collaborative tool to improve your own recipes. You might give it, let's say you're Gordon Ramsay and you give it, you know, ingredients and hey, this might be missing a special ingredient. What's something out of the box that maybe I didn't think of or something that's inspired by a certain cuisine or certain demographic in, in the world. Right. You can, you can collaborate and use the context from different industries even and apply it to your own industry, which I found really interesting results. When you take different vertical niches and combine them together. I haven't heard too many using that, but you get sort of original insights at least in your field. So it can't really think or think creatively too much. But it's getting better. And better. And it is getting that point where it can start providing you some kind of creative, original thought based on at least different fields. Because I think even if the collective human knowledge isn't completely, it's, it's. There's very little original and creative thought in the world. Usually something creative is inspired by something else or another industry. So you can almost think of it like that, where maybe it will never have creative thought, how humans think about, but it may be helping you to get inspired and create something new or original that maybe you didn't think of.
- [22:03]
Creator Science Host
The thing that kind of broke my resistance was that as I started to talk, to chat, just as a thought partner, and same with Claude, I realized, oh, I'm a small language model. I'm a very small language model. My brain is that. And this is a large language model. And there's some benefits to being a small language model. Like, I think I am very focused around a certain type of information. And so the connections I can draw are very contextual in that way. And that might take some massaging to get there with something that's a little bit more larger and generalized. But that's probably what we're going to see in AI anyway was we'll have these like very vertical specific models that are trained specifically for cooking, trained specifically for engineering, trained specifically for content creators. It's just undeniable to me. So the question I come across is, well, do I make more curriculum or do I change the type of products and services that I sell? I think the first step that you took is you built a chatbot into your existing course curriculum. Can you talk about that?
- [23:07]
Mark Schust
What's funny is the AI created all of this for me. It created the name Maggie Bolt. It's inspired by Magento Bot. So that's the name of the bot, gave it a personality and. And basically it scrapes my lesson content and injects it into the context of the LLM. So she's very context aware of what you're talking about. And you can. It's sort of inspired by Khan Academy as well. They have interesting bot, I think it's Khanmigo. And it makes you question sort of everything. So it's constantly prompting you and asking you if you understand a certain way and it wants you to explain it in your own words. And it can detect and of course respond and tell you if you're incorrect or correct. So if you extrapolate that out. So that was one lesson. To train it on the collective knowledge of everything is really, really difficult. At least in a specific vertical. But I've been thinking through and I think just courses are too long. And again I think learning is going to be more embedded and integrated in what we do every day. So even when I started building the, building my courses, the first one, I had this grand idea to build a zero to hero course that has everything to do with Magento. The problem was I would still be building it right now, five years later if I did do that. So what I realized is I need to break this down into multiple courses and I need to break lessons down even simpler so everything gets smaller and smaller. It's chunk by chunk, byte by byte. And then I pretty much created an idea of atomic lessons. And other, other people have had this idea as well. But if we want to extrapolate this out further, where AI is going, if it's going to be integrated into our daily life and I want to ask it something specific, it needs to be able to connect and look up that information, that specific thing that you're asking about and pull back data from it and some kind of model or knowledge graph. And that's why I'm seeing more education going towards more atomic units that are just individually consumed and because they can be reassembled in different ways. So if you ask it a certain question, it may be able to look up that knowledge graph and pull out two or three individual atomic lessons that are really small and concise and short and focused and combine them together and give you a reply that you normally wouldn't have gotten in a regular course. So I see just a big shift in this education of where it's going. And I think that may be one place plausible solution that comes up.
- [26:01]
Creator Science Host
The way you're describing like atomic lessons versus like a typical course chapter or something. It sounds like you're saying this would be short in terms of duration. It'd probably have some sort of metadata like a title that is hyper specific to what is included inside of this video. Rather than like a 20 minute video that might include the content of 20 atomic lessons, but would require the machine to kind of transcribe understand all of the information within that video before it could connect that information. It's keying off of shorter, more specific bits. Is that kind of what you're describing?
- [26:38]
Mark Schust
Yes. So you have almost the same needle in a haystack problem I talked about before. So when you give the whole like LLMs are trained on everything in the world and they aren't really good at niche specific topics. Even part of building, even like you said, a small language model, something more nuanced, something very specific to a certain topic. If you can chunk that out into a way that can be categorized and filtered and searched upon and indexed, it may be able to give much better results. Because when you have specific lessons just on one specific thing and no preliminary or it doesn't assume any pre existing knowledge, you'll have just a very focused answer that's correct, that won't hallucinate and it can can be used to combine with different lessons as well, to create something new and personal, even for a specific user.
- [27:31]
Creator Science Host
I see this user behavior with my wife actually, because when TikTok was momentarily banned in the US she is one of the people who decided to erase it from her phone and couldn't redownload it for like two weeks. And she was having a meltdown because TikTok had become her primary search engine. She found that she could search things like Ikea, Couch, Alden and Ivory, which is or Aidan and Ivory, which is the brand of couch that we were thinking about buying at Ikea. And TikTok would serve up exact specific videos for that brand of couch and she would get in a very short period of time exactly what she was looking for. And when TikTok was gone, she couldn't do the same thing on Google, she couldn't do it on YouTube, she couldn't find the same bite size, contextual information, reviews, teaching, you know how to cut a pineapple, you can just find a perfect tiny video for that on TikTok if you search for it right now. So I see people leaning that way because we are a convenience creature. We want convenience, we want accuracy, and we want it in the shortest period of time possible. So I do, I do kind of see this world, but that sounds like that is Counting on these LLMs Indexing your content and bringing it back. And I'm interested to hear what you think about SEO and indexing of content, because we hear a lot of people in this space also who have primarily had their traffic driven by SEO, they saw a big downward shift with the helpful update. Now Reddit and Quora are getting all the traffic, but in general, if we move to searching for information through an LLM like perplexity, how does our content get surfaced or indexed? Do you see the original authors being.
- [29:19]
Mark Schust
Credited in these results with large language models? That's sort of exactly what it is, right? It's not nuanced on specific topics. So you're going to get a collective result of an entire world and it's probably not going to be as good as something, as more of a small language model, something trained, so you can of course create your own model that has your own specific info. It's a little different than your own chatgpt because it, it would have its own knowledge graph and everything. Sort of like LLMs. Yeah, that, that whole shift with LLM search is interesting. It's definitely changing the sea SEO game a little bit and, and you could get surfaced in results, but it's, it's a crapshoot. It's just like how search was at the beginning days. It was very hard to get searched or returned as number one in the search. And that's what everyone wanted. The top, you know, it was top 10 and then top three. So its ability to actually serve up specific content from an LLM probably isn't that good. So like you Talked about with TikTok, that was specific because it was personalized to your wife's browsing habits, what she watched on, what videos she dwelled on. So TikTok knows all of that about you. And then it can serve up a very specific ad or piece of content, depending on what you're browsing. That's why even there's a lot of rumors that Google or Apple recorded your voice and microphone randomly at phones, because people would see an ad on Facebook an hour later of something really weird that they were just talking about in a conversation with other people. And the Facebook engineers are adamant that their stuffy suits and lawyers would never ever allow anything like that. So they just said that's how good the algorithm is, that it knows pretty much exactly what you're looking for and it knows when to serve it up. So similarly even to teaching, that could be applied to even learning. So let's say I'm learning and it's 9am, I just had my coffee, and maybe it learns that I can learn best 45 minutes after I've had my first coffee. So it can ping me and say, hey, it's time to not even. It's time to learn. Here's something you may want to learn. Right. It's just, it feeds into your habit. So it's not like on a specific rigid schedule, it can be optimized for how your brain works. So it's could be continuously perfect and personalized in real time too. So that's sort of the magic of teaching too. A lot of it is knowing when to teach, when to push, push hard on someone, when to back off, when they may not be ready to learn. Learn something new.
- [32:04]
Creator Science Host
So when you talk about these atomic lessons that you want to create for your educational content is the thought that you will continue to build a reputation as the Magento guy. And so people will come to you and your specific language model around Magento because they trust you to give context specific good information around that subject in particular. And, and they will use a language model created by you that is trained on specifically your material.
- [32:35]
Mark Schust
Yeah, so pretty much so right now, like I think the most advanced code thing right now is called cloud code. It's a terminal editor and you can ask it to code anything you want. And it does a really, really good job. To the point where one of the founders of OpenAI established a term called vibe coding not too long ago and he talked about, hey, you could just sit back and let, let the editor code for you. But I sort of see that as nuance because you have to know how it replies back and you have to know what's good and what can be improved from even an architecture level. So similarly, when you're learning a specific topic about anything, the LLMs aren't really trained on that specific data. And I don't know if they ever will be, they'll ever be nuanced. So I think small language models will be on the upswing because you can train it on specific data that's more nuanced. And the more complex topic it is, the harder it is to return a result because it's again that needle in the haystack problem. It hallucinates and everything. So when you break things down and you can create your own knowledge graph on a really complex topic, you can just deliver much better results.
- [33:52]
Creator Science Host
After one last break, we talk about how Mark is future proofing his education business. So don't go anywhere. We'll be right back. And now please enjoy the rest of my conversation with Mark Schust. Do we think with improving models it will eliminate the needle in the haystack problem?
- [34:08]
Mark Schust
Possibly. So the context and the context length that we can feed into the LLMs is getting larger and larger. That means the amount of data that you can have in a single conversation. So I think Google has something like a 2 million parameter context length in Gemini, which is the longest. But it does again have that needle and a haystack problem. But as that improves, it may be able to find those needles. It's just a wait and see game. History is telling me that they will be able to do it because they've been able to overcome all these obstacles so far and advance it. Yeah, it really depends on really what you're working with and how better it can find that data within that large context length. There are rumors that a lot of the LLMs, even OpenAI has unlimited context length, so I don't know how that's going to factor in. So maybe they've already solved it internally. It's more of a wait and see though, because there's a lot of just guessing of how this is going to affect everything.
- [35:16]
Creator Science Host
It's really interesting in my own behavior. So I work predominantly with ChatGPT and Claude. I've tried Gemini a little bit, haven't given a good enough shake for some reason I'm biased against Grok, but people are saying good things so I should give that another shot. I was going to try Deepseek, but I was going to do it with an API into typing mind and ever since I've been trying to try, it's like we're over limits. You can't do that. But anyway, what's really been interesting is since OpenAI rolled out, what is it? 01 the longer thing where it takes a little bit longer to respond to you. I've noticed that with Claude 3.7 and with even O3 mini from chat, when I get an instant response to my questions, I actually trust it a little bit less. It's trained me that I think the best responses come with a little bit of consideration from the product. And it's interesting because I know Claude 3.7 is probably better than 3.5, but it's faster and it makes me feel like it's working less hard. And I don't know what to make of that. But I would have to think that the longer the context window, it should take more compute, more time to contextualize what I'm thinking. It's almost like hubris that I think what I sent requires so much thought, spend more time thinking about it.
- [36:36]
Mark Schust
I think that delves in the delves. That's the first time I've used that word in real life. You are an AI. After a while you start just becoming an AI, I guess. But yeah, it's. That's just human psychology, I think, because even in programming, it's funny, years ago when you hit place order, sometimes that credit card order can go through in a second. Or maybe that's a bad idea, but maybe I'm customizing something for you. So programmers put a loading spinner, had them wait 30 seconds before they got a response because they thought it was actually doing more work when it could have happened almost instantly. They almost thought there was A bug in the problem when it happened instantly or really fast. So I think that's just how our brains work. We think that something that takes longer is going to get a better result and that's not always true. 3.7 with Cloud has been amazing from my experience. And they've actually built reasoning models inside of of the model like reasoning. So it knows internally to think before it responds to something, just does it so faster and cheaper. So that's definitely something to think about. And I it's. It knows when to switch context in the background, which is probably the most intelligent AI right now based on that. And it's funny you mentioned that, you know, like croc. Yeah, it had. I noticed something really weird with that too. I don't know if it's because Elon runs it, but when I asked it to explain something simply, it just can't. It goes into extreme technical detail and maybe I'm not prompting it correctly as well. So it's funny how there's different models can respond in different ways and it's best to just experiment with all of them and find out what works best.
- [38:23]
Creator Science Host
Talk to me a little bit about language marketing positioning. As you think about the future of MDOT Academy, your education platform, do you see yourself marketing courses? Do you see yourself messaging this as education or as a language model? Like how are you thinking about a change in language if you're thinking about a shift in how education is delivered?
- [38:47]
Mark Schust
I think right now it's you can only go on what you know. And right now we don't know where any of this is going to go and how far it's going to progress, how things are going to change. The entire landscape might change in two, three years for all we know. So I have to deal with what I'm dealing with now. And right now courses are still premium grade, right? I need to take a course to learn something and save time. And that's not changing this year or probably even next year. But it could change over time as these LLMs and AIs develop. So what I do know though is that content abundance, like the ability to create and generate and consume content, is going to be ever expanding. And people's attention spans seem to be getting less and less. They're going the other way. So I think the value in a specific membership or course platform comes down to personalizing the educational content for specific students. So you actually learn their styles. And you can do that today, right, with just asking and polling your students, how do you learn? And you can cater your content to them. And then community is a huge aspect. So if you get stuck on an AI issue and it can't solve it, where do you go right now like you're, you're stuck. I, I need to talk to an expert, I need to talk to someone. I need to get advice of. Hey, it gave me this result. I don't understand it. It's not explaining it to me well. Right. I, and a lot of students just won't even be in a terminal or want to use an AI like it is right now. It's a different tool. It's not embedded currently. So it's hardly anyone uses cursor in php world like @ least in my, in my industry, it just doesn't generate good code yet. So I think that may morph and change over time, but we have to wait and see. So there's that personal personalization, the community and then there's the verification that you actually know what you, what you know. So it's more of a verification of knowledge and you can't really get that with any AI. No one has a, hey, tell me how Mark knows how much he knows and what he knows about and what his experience is. There's nothing to do that. So I think a course and getting verified and certified is definitely the way to go. And those three things, I think the value shifts to those rather than the actual course content itself.
- [41:19]
Creator Science Host
How much of your business comes from like professional learning budgets?
- [41:23]
Mark Schust
From me. I used to buy courses all the time. That's. That's ironic. I don't buy it much anymore. But I think that's because of just the phase of my life I'm in right now at the moment.
- [41:36]
Creator Science Host
I mean more like your students are they, are their companies purchasing courses so that their people can take your, your material and become a better employee?
- [41:46]
Mark Schust
Yes. And I, I see that actually growing with companies, it's hard to find knowledgeable workers. There's. I've heard, I've even had story. Just recently when I was over in Florida they interviewed a, a senior developer. Someone's supposed to be senior developer. They had seven years of experience and they, he just completely bombed the interview because he, he stopped thinking for himself and that's part of, he was using AI for everything. So that's part of the big problem. And there's another big problem with junior developers. Right. It's very hard to onboard anyone who's junior right now, depending on your industry because right now cloud code, I can hop into Terminal and tell it to code something and it does A better job than a junior developer would. So there's those two aspects and you know, one is like AI. Call it AI atrophy, where people stop thinking for themselves and can't, can't think creatively and independently, which is a really big problem. And there's another problem where you have developers that may want to learn it, but they can't get a, get a position somewhere because the bots have taken over their job, at least at some level, and they can't upskill their learning. So companies definitely will be onboarding, I believe, more new employees to get them onboarded and trained right now on course platforms and memberships because it's really the only way to advance their skills quickly and deliver a result that they can learn right away.
- [43:18]
Creator Science Host
Yeah, I was thinking about this. I think there are a lot of folks who they do tap into these professional learning and development budgets. And as much as I feel like courses as a business model are changing, I feel like uncertainty is a time when if there is money available, there might be like new unlocks in budgets. So in a world where how people learn is changing, if you're at the front end of that, I feel like you might have an opportunity to unlock some of those learning and development budgets. If you can help these companies, their employees upskill in this very rapidly changing time, I could see a case to be made for that.
- [44:03]
Mark Schust
I think you got to be careful too, of not going, not taking too much of a change. If you're a creator, don't change your entire business because things usually progress over time very slowly. I've learned not to do that. My previous company was actually called Change, ironically. But it's. If you change too quickly and try to do too much, you'll probably fail because there's just, there's that unknown that you get into and you don't know how, how people are going to react to a giant change like that. So I think just, just slowly and iteratively changing and morphing your system to get slowly better. Sort of like how I added a chatbot to my learning system is probably the way to go.
- [44:47]
Creator Science Host
I've seen so many content creators who just go all in now and talking about AI, which for a lot of them was like a complete 180 degree change from whatever they're talking about before. So when you successfully build a reputation, being known for something and you completely change into a different direction, that's a big risk, that's a big cost. So I agree with you that I would be careful to jump in and just grab the latest buzzword for the sake of trying to capture some of the tailwinds. Because yeah, one, it's getting increasingly competitive all the time and two, there are only so many people who are interested at staying at the absolute edge of this constantly. We want to get incrementally better. But you're right, the pace of change in actual professional context is much slower than the pace of change of technology. Things are going to be possible much sooner than they're actually implemented and experienced at the company level. You mentioned community, which also, you know, as I think about the future of this business and how I'm approaching this, I kind of see this dichotomy of one, you could lean into how to work with AI for a new model of education. Because as I shared earlier, my thought is the way to interface with this as a teacher is just not very intuitive yet. So building a better interface behind a trusted brand I think is a great path forward for education. I think a second path is focusing more on community and the human side of things. That's the path I'm more interested in following right now with the lab and doing in person events. And even in my writing I try to just be more human than like, here's the 10 steps and here's the bullet point. What does community look like in your business and how is that changing?
- [46:40]
Mark Schust
Yeah, I run a private developer community for it's geared just for magento developers. I have a contractor who helps out, answer questions and who's been fantastic, very knowledgeable, like an encyclopedia. So yeah, that whole community aspect is so important and it's the main reason a lot of developers enroll in it, because they get a response. That's not a bot, right? That is a human who's thought through things and can deliver an answer at a high level. And you can't get that experience anywhere else other than a private community. Even the public communities are just overloaded with AI bots and spam and comments. We notice that on LinkedIn I see posts all the time, constantly. They're AI generated and comments. I try to block everyone as much as I can. It drives me crazy. I'm sure we've all like dabbled and experimented with that, but I think people do want a more human connection and what they're getting right now. And I think that's a great way to do it. Even in person events may be a good way to add to your membership if you can do something like that. But anything you can make a personal one on one connection with you're probably going to get a better result than just someone who doesn't focus on that.
- [48:01]
Creator Science Host
You also get very specific experiences. We had a post in the community yesterday that was somebody posted, should I spend this many thousand dollars on this program? It was like a coaching program and we had folks in the community who had experience with that and they were able to weigh in and be like, here's my experience with it. Yes, no. Which you ask that to an LLM and it's going to give you a very strategic pros and cons list, but it's not going to be able to tell you this is my experience. It was very negative because of this and I would stay away from it. So I looked at that post and I was like, wow, that's like a, that's a huge roi.
- [48:34]
Mark Schust
Yeah. Even. Yeah. Conversations I've had with you. The LLM will say something, hey, you should go down this path and you talk to someone else. You're like, no, that's a horrible idea. Because this, this and this, right? So you can't, you can't get that with an AI and you probably never will be. Right. It's just human experience. You can't really, you can't really touch, right? You can't replicate a human spirit, their interactions, their experiences, what they've been through, and their advice. It's best to just take advice from wherever you can get and then make your own decision, of course. But it's great to get that feedback from an actual human because it's just becoming less and less available. You can't really touch. Right. You can't replicate a human spirit, their interactions, their experiences, what they've been through, and their advice. It's best to just take advice from wherever you can get and then make your own decision. Of course. But it's great to get that feedback from an actual human because it's, it's just becoming less and less available.