B (3:32)

So that's where the moldable part comes, right? Like, because you're constantly molding your development environment so that you tackle system specific questions when you're investigating how they work, what might not be working, what might be affecting performance, and so on. Right? Like, is that why you chose the word moldable?

C (3:52)

That's exactly it. So what we're saying is, I mean, we started from a couple of observations. So one observation is that you can't perceive anything in a software system except through a tool, which means that the tools are essential. They are not nice to have. They're absolutely, they're fundamental. Like an editor, for example, that you just use out of the box, that's a tool. But the thing that you type, that's not the shape of software. In fact, software doesn't have a shape because it's just data. But we as humans, we need some sort of a shape, some representation in order for us to perceive so that we can reason. Now, the tool is the thing that translates whatever is inside the system to in some form that we can go and perceive. Now if that's so important, then the ability to control that shape is probably.

B (4:41)

Kind of important and define what that shape should look like. And you have a story to share with us, right? Like the telco story. Walk us through that story.

C (4:50)

Okay, so you're referring now here to the. We tell a story in the book. The book is called at the moment Rewilding Software Engineering. The book that I'm writing with Simon Worley and this is an actual case study that we went through. We've been through many case studies, but this is just one of them that we picked. And so this is a. We're talking about the data pipeline in a telecom company. It's the main data pipeline that is fueling the offers that the telecom company was sending to all their customers. So millions of customers are being basically receiving offers, you know, one of those two in one promotions and so on. But they had a problem. So a few years back they realized, oh, we're a little too slow because it took about one day from the moment they had the data until the moment they could send offers out. Now, one day in a business to consumer market, to react is an eternity today. So everybody knew about this. The problem was visible all the way to the board. The CIO mandated this must be much faster. The budget was put in place, the initiative was started. Three years later, somebody started to measure and they realized it took exactly one day to go from here's the date until the moment they send the offers. And then the question was, what did we do for three years? They have hundreds of person years of, of effort being spent on, on an initiative, and at the end of the day you have no result. So the problem question that they ask is that how come that we did not affect reality at all? So this is when we.

B (6:42)

Because, because. And that's important because they actually made a lot of changes. So like this, hundreds of, of many years of effort didn't go into doing nothing. They did a lot of things, they made a lot of changes. But the reality was that three years later, it still took exactly one day to do the exact same thing that took one day three years before.

C (7:02)

That's the problem. Right, exactly. So it's not like the effort wasn't spent. And this is not one of those things where you say, oh, we have a legacy, but nobody wants to modernize it. We don't have budgets, we don't have. Not at all. It was, here's the budget, here's the problem, Everybody wants it to be solved. The energy is being spent, and at the end nothing happens.

B (7:22)

So why, what was going there? Why didn't they find the reason why it took one day to go through that data pipeline?

C (7:30)

Right. So this is what we wanted to find out. So first thing we did, we asked them, so what is the data pipeline? And they drew for us four boxes, just here's some database or some database. And then some transformations were happening, putting into some other places. And at the end, there was a low code platform that was manipulating all this data and doing something else. So there were some transformations, lots of transformations happening through this pipeline in four boxes that they drew. So whenever somebody draws for us a picture of the system on the whiteboard, they don't document the system. They document what they believe the system to be. This is such an important distinction, right? We've made this. We confuse these two things for a long time. There used to be a time when people were saying, oh, let's decide what the system should be in the future. And this is when people say, oh, let's draw, let's catch it on a whiteboard or on a napkin or something. And that's great because you're describing something in the future, but as soon as you have a system, if you're still describing it on the whiteboard, you're not describing the system, you're documenting what you believe the system to be. And this is exactly what was visible, very visible here in this case, because we started to build a tool. This is how we want. This is how we approach any problem in software engineering. We simply say, well, what do you think the problem is? Or what do you think the system is in this case, they said, here are the four boxes. And we wanted to build a tool that will mirror back the boxes and.

B (9:06)

By looking at the actual code and finding where those four boxes are and then showing those four boxes to us by kind of translating the code into those four boxes.

C (9:17)

Right, precisely. Okay. So we started to build, and a couple of weeks into the problem, I said, well, actually the output of the first box doesn't seem to match the input from the next ones. And then we went back and we asked, are you sure you don't have something else here in between? And they found the whole system, which.

B (9:37)

Of course was taking one day to go through, I'm sure.

C (9:41)

But so the point. Exactly so. But the point here is this. So their system had the reality consisted of five things, five boxes, but they were only able to enumerate four of them. Even at that high level of what do you have? What is your reality? Right. Even at that high level, they were able to only enumerate four things out of five. If this is your level of awareness about what is reality around you, you have almost no chance of systematically affecting.

B (10:16)

So I'm sure that somebody listening to us might be thinking, oh, they're just not a very good team. I'm sure that if my team was the one doing that, we would know about the five boxes. Okay, but we need to explore why that is so.

C (10:29)

Right.

B (10:29)

So let's dig into that a little bit. Why was this large team of competent and committed developers unaware of an entire system in their high level architecture?

C (10:41)

Right. So, but before we do this, let me take a little parenthesis on that thing. I thought so too, because I thought, oh, maybe we were just, you know, unlucky or lucky to find all of these cases because we've been doing this for 15 years. And I'm seeing these kinds of cases of 15 years, and I thought maybe these are just some data points that I have. And it's just my perception is so skewed so since we've made this a bit more public, so about half a year or so, I've just had the chance to Talk with multiple CTOs and head of engineers and they said oh yeah, I had that kind of problem. We were doing this migration and I saw there was this special payment system that were kicking off only every six months or so and it was in a special other place, completely different, that nobody knew about or. And we see these stories actually appearing all the time. So I was scared that people will think that oh, this is a little bit too extreme. But it turns out it's actually not that extreme. So now back to your question as to why is this happening. Why are we not able to enumerate the reality of our system? And the problem goes back to how do we get ourselves informed? So the today the main source of information that we gather from system is reading.

B (12:05)

You mean reading code?

C (12:07)

Reading code or other textual artifacts. So some developers are going to read some code and then some of those lessons are being summarized and passed forward. The problem with this is that our systems are typically larger than anyone can read or any and anyone will read. And even if you try to read, so let's say if you do a real, let's say quick calculation. So let's say you have a 250,000 lines of code which is not that large. Let's say you read really fast, one line in two seconds. Let's say you can do it for eight hours non stop. It takes about one person month to read the whole thing. Now within one, you know, within that time this will have changed many times. There's no way, this is no way to get yourself informed. You will not know what the reality is if that's how you're going to approach the problem. So it doesn't work, it doesn't escape because the systems, they change all the time, which means that everything you knew from yesterday, everything you know from yesterday, it's actually not a useful, it's only a useful source of hypothesis. It's not a useful source of answers.

B (13:25)

Okay, so we now have the problem in two layers. So first is that when we describe a system, we're describing what we believe the system to be. Then the next question is, okay, but why don't you read the system? And then that's the second layer, which is even if you would read the system, it would take so long to read that once you have the full understanding of the system, say a month later for a 250k line of code system, the system has already changed, which means that your then current understanding is already out of date. So we have this infinite loop of always missing some critical information. Right, and that's what you are trying to tackle with glorious toolkit.

C (14:08)

Glamorous.

B (14:09)

Glamorous. Sorry. It could be glorious as well though.

C (14:12)

Exactly. We take any of these. So this is first of all the amount of time that developers spend on reading accounts for the more than half of the time total time of development. This is the single largest expense we have. And in fact. So I know because we have lots of good studies about this going back decades, but I asked thousands of developers, do you agree that you spend more than 50% of your time reading code? And everybody says yes. And by the way, we're talking about AI, all of a sudden people realize, oh, development has fundamentally changed because now I find myself reviewing code all the time. You were reviewing code before, you were just maybe not necessarily aware of it. Because I'm asking people the second question. And the second question is, when was the last time you talked about how you read code? This is kind of important because it's a very weird question. Nobody knows how to answer it because nobody thought about it.

B (15:15)

And there's also another aspect which I want to bring up in this, how you read the code. Because when we described the first case study, the telco story, we talked about understanding the system from a box perspective. So high level architecture perspective. But that's only one of the many perspectives you can have on a piece of code. Like performance, number of calls to certain systems, how long. Yeah, like all of those different. There's a million different perspectives. And even when you're reading the. Let's assume you could read 250k lines of code in one day. That would only give you one perspective. You would need to read it the next day, again with a different lens to find the other perspective and so on. So we also have this additional complexity that systems, dynamic software systems have many different perspectives and a high level architecture is but one of them.

C (16:12)

So yeah, so exactly. So this is why I'm asking people, how do you read code? And when now when people don't know how to answer is because they haven't thought about that problem. But if you don't talk about it, if it's not a subject conversation, then it's not explicit. If it's not explicit, it has never been optimized. That's one thing we know from agility, is that if you want to optimize anything, you first of all have to make it explicit without it being explicit, you have no chance of it.

B (16:44)

Except by accident.

C (16:45)

Of course, except by accident. But it turns out that the accidents rarely happen. The problem here is, and this is really, on the one hand, we have a humongous waste because we're talking about the single largest expense we have. This is not like a tiny thing. That's a tiny problem, Right? So more than 50% of the time being spent on a single activity nobody talks about. But on the other hand, you can look at it as an enormous opportunity because, like, we've just discovered a huge amount of our energy being spent in the least productive possible way. Because that's what reading is. Reading is the most manual possible way to extract information in other systems.

B (17:28)

Okay, okay. But before you go further, reading is the least optimal, most expensive, slowest way to extract information from a system. Now, this is a very important statement because especially today in the age of AI, the very first thing that I'm sure you've heard before is like, we don't do that anymore. We just ask the AI to read it, and we ask it the question. But you have a specific perspective on that use of AI to read code, preconception. Tell us a little bit more about that.

C (18:00)

Right. So the idea is that when I asked, for example, if we go back to the telco example, when I asked, people draw the architecture, they gave an answer, and the answer came actually quite quickly. It came like within minutes, they could draw the four boxes on the whiteboard. But it turns out that answer was an opinion because we didn't know why the boxes. We didn't know whether the boxes are all the boxes. We didn't know whether all the dependencies are the ones that they described. And we didn't have the explanation as to how they got the answer. And when you ask the AI, you will find that you get exactly the same kind of same type of answers back. The answer is going to come quickly, but you will not know whether this is accurate. You will not know whether this represents the whole thing. And you definitely do not have an explanation as to why the answer is the way it is. So you'll get the answer.

B (19:01)

So that's a very important realization. If we give a system to the AI and assuming that the AI is really good and is able to correctly read the system, when you get an answer, you get the filtered answer. Whatever the AI thought was important, that does not necessarily mean that those are the things you thought were important. But there's another layer which is that you ask the question from the AI, only one Question. And what you talk about in multiple development is completely different, which is to have, as you called it, thousands of tools that represent the system both statically and dynamically in many different ways. And when we work with these complex software systems, we don't have the luxury of looking at it only from one question's perspective. As soon as we start looking at the answer for that one question, more questions will come up and we need to answer those questions. So we might actually be in a case like for example, the Telco. We might actually be in a case where we need to go into different levels of detail at different steps in order to understand the system. And if I understand you correctly, what you're trying to do with the glamorous toolkit, which could also be glorious, is to be able to provide, not instantaneously, depending on whether we're doing a dynamic or static analysis, but quite quickly, multiple different understandings of what is happening in the system. Right, right.

C (20:25)

You mentioned a few points here. So first of all, let's talk about the different levels of granularity at which we might want to have the answer. If I come back to how did we draw the five boxes now diagram of the system? Well, we had to first of all do in their key important or their key hypothesis was oh, we are producing too much data along the pipeline that we are not using anymore. So we said, okay, let's actually see what the data lineage is. But of course here you have now data lineage that goes through like four or five different technologies. So there is no tool anywhere. And one of the technology was a homegrown Excel based transformation system that was translating Excel files in which people were programming transformations and transforming those into Oracle instructions that were populating various tables and those were being transformed further. And then on the other side, you had a low code platform with 22,000 scripts in 54 different templates. So that is the thing that you have to now figure out how things relate to one another. Because the problem was the team at the end, the local team, didn't know what they could change because they didn't know where it come from. And then the other way around, like the people on the first team, the core team of the data team, they didn't know how to change because they didn't know what implications, who's using those. So at the end we had to do a data lineage, but there was no tool, there would be no tool possible out of the box because nobody knows that combination. Definitely didn't know about the homegrown language that they had now. So we built a tool. But now, of course, in order to build that high level view with the boxes, you have to literally do it. Data lineage on the property level. So very, very, very low. Very, very low level. But to do that, you first have to reverse engineer the language, because on the one engine, some language here that nobody knows about, it's just some script that translates something into something else. So we have to reverse engineer that. On the other side, on the low cod, you have a local platform that was actually advertised for creating real things really quickly, which they did. So they ended up with thousands, tens of thousands of scripts. But on the other hand, there was no tool to analyze it in any other form. So we had to put all those things together. And so this requires multiple layers of abstractions. And now the question is, and the problem is that until now, the industry is optimized around the idea that all of these different kinds of questions requires completely different kinds of tools. And that's the thing that we are showing not to be true, because we are saying that multiple development is uniformly applicable if you have an appropriate environment. In Glamorous Toolkit, which is, by the way, free and open source, anybody can take a look at it. In Glamorous Toolkit, we are showing how we tackle very, very deep, apparently, different kinds of problems in the same way, using the same skills and same way of decomposing problems over and over again. And the consequence of this is that we can start approaching problems that require combinations of techniques very easily. Okay, so once you have such an ability to create arbitrary perspectives about arbitrary problems at arbitrary levels, then you want to, at the extreme, you want to tackle every question you have about the system through such a tool, all the way from the simplest developer question that might occur maybe dozens of times per day per developer, all the way to how do I transform my system from A to B, like strategic level.

B (24:25)

And the tools that you build to add to the Glamorous Toolkit are contextual because they require the understanding of what you're trying to analyze. Right?

C (24:35)

So that's the second I talked about the observations that we started from. So one observation was tools were important. The other observation was developers spend most of the time reading code. The reason they read code is because actually they don't want to read code. They want to just extract enough information to be able to make a decision. So the actual activity is decision making. The extraction of information is just a means to get the data and extract what you want. But the third observation is that software is highly contextual. So because software is highly contextual, it means that we can predict classes of problems people will have, but we cannot predict specific problems people will have. And because we can't predict specific problems people will have all clicking tools, all the tools that you just, they just give you a button to click on out of the box will most likely fail because, or let's put it the other way around, all deterministic tools that do that will fail. So for example, to maybe take a contrast here, so let's take for example a test as built in test driven development and compare it with another kind of a test, like a static test, let's call it some sort of a lint rule or. So they are both tests, right? In the sense that you run them against something and the test against the dynamic system, the static analysis against a static system and you get maybe a green, red, yellow type of answer back. So it's an analysis. Now the problem is how are they created? How do people create the test? Well, they stop and they have a problem and they built the test within the context of the system. And so you have the problem and then you document the problem in a deterministic language and then you run against that. Now the static analysis though, how do people work typically with that? Well, they download it from somewhere else. The people say, oh, there's like this list of a thousand rules and these hundred of them apply to me. But then of course if I download it from somewhere else, by definition that rule must be applicable to me and you. But if my system is, you know, telecommunications and your system is in automotive, like what do they, what do they have in common? The technology? Is that the base? Is this the level of the value that we're interested in? No, that's not the interesting part, it's just the easy part.

B (27:10)

But actually that's a good point because that led to the development of TDD as a development practice, right? Because it led completely contextual and for decision making. Where should I put my focus in this case? You know, if something is broken, where should I put my focus? So when we talk about multiple development, I think, and this is my understanding, tell me what you think is that we're going back to understanding that we need to change the developers experience, that the tools that we use in this case decision making aid, are extremely important and they should be developed with the developer's experience in mind, not just as default high level, everybody types text into an editor kind of understanding.

C (27:54)

Yes. So let's continue the comparison to test driven development because one of the Things that Simon and I came to as a way to explain things is that. Well, in software engineering, test driven development is very interesting because you start, you have hypothesis, it's hypothesis driven, and then you need information from the system, and the information is always generated, and it's always generated contextually for the system. It means that nobody downloads unit tests from the web. Everybody creates them within the context of the system. And the reason they create in the context of the system is because they stop and encode value in the test. So at the moment, a single test fails. No, people don't deploy. Right. That's kind of interesting. At the same time, you might have tens of thousands of static warnings, but in fact, there is nothing wrong with the static analysis technology. What's wrong is how we approach it, because only in the testing part do we take into account the reality of the system, which is that it is highly contextual, which means that the only deterministic tool that will work will be those that are built to match the context. And so that's exactly what we're doing in testing. And then now how do we do it? Well, we need some sort of an engine that makes the creation of the test very, very low. Right. Very low cost. That's the key. Because once the cost of the test doesn't matter anymore. Right. Then we end up creating tens of thousands.

B (29:26)

And that's what TDD is all about. Right. Because with the TDD systems or frameworks, what we're doing is we're putting the focus on the decision making, what do I want the system to do? And then the test harness and the test framework will do everything else.

C (29:42)

Precisely now on the other side. So this is what we are doing for decision making regarding the functionality of the system. But there are many other kinds of questions that we might have for which a functional decomposition is not useful. But the approach, the way we're tackling this is fundamentally great because we're essentially approaching every single development problem as a little tiny empirical scientific discovery problem. That's it. We can apply the scientific method for every one of these cycles that we have. And now there used to be a time that's kind of interesting. If you go back like 25 years ago or so, there used to be a time when people were saying, oh, are we ready to release? This was the decision to be made. Okay, so how do they know? Well, they would look at the report. What was the report? Well, it was somebody putting things in Excel. Okay, how do they put things in Excel? Well, they had. They had to run the test case, how did they run the test cases? Both manually and every test. And this is kind of also important. Every test was really, really different. Like, oh, for this test I have to go and start the database and replay the other database first and then set the user for this one. And then I'll click, click, click here. The next case will be completely different. I'll run some scripts and so on. Completely different, very ad hoc. And then it turns out that all of those can be completely industrialized in a systematic fashion and that you can apply to any arbitrary functionality, which is kind of interesting. Now it turns out now what we are doing today when it comes to, for example, how do people optimize the data pipeline in the telecom company? Well, they are asking someone to draw the diagram, which is manually put together a report. And then how did that report get created? It was basically sourced together by somebody inspecting the code manually. This doesn't make sense. This is like testing 25 years ago. We can take the testing lessons and simply generalize it to everything else. Now, of course, this requires a new kind of platform so that you can create these tools very quickly. We call them micro tools. Just like a test is actually a little tiny micro test. And it's the totality of them that gives us a model of the system and encourages us to move forward. So this is basically what we're saying with multiple development is you need a new kind of a development experience that should be focused not on what exists out of the box, but on how quickly you can create a contextual tool. Exactly.

B (32:25)

To solve specific questions. And implicit in this, and I want to make sure we highlight this, is that we are constantly developing these tools, just like we are constantly writing tests. And I think that this brings us back to one thing that I want to discuss and understand your perspective on, which is this idea that software emerged in a world, let's say 60s and 70s, more like the 80s, because that's when really massive scale software development started. Software emerged in the context of other industries. And for many years, for decades, we've been trying to apply the lessons from those industries to software. And I think that the glamorous, which is also glorious toolkit, is finally giving us tools so that we can look at software as software done by specialists in software development. Understanding that there's many different perspectives into a software system. Dynamic, static, looking at only data, looking at only performance, looking at memory usage, all kinds of different questions we can ask from a system. And we need to have those answers because without those, we are actually Developing software blindly.

C (33:39)

Yes. So this debate of whether software engineering is a special thing or not has been going on for a long time. But one of the characteristics that is rather unique here in our case is that the tools, so that the thing, the object of our work is invisible, it's not perceivable except through tools. This is rather unique. You have similar things. For example, if you want to study microbiology, you cannot perceive it except through a tool. If you want to go and take a look at the deep space, you cannot perceive it except through a tool. So there's, there's parallels there. Except now the subject of the perception is different from the nature of the tool. And in software they are the same. This presents itself with a unique opportunity and most likely a different kind of lesson. So in this case, the construction of the tool is indispensable from actually the nature of manipulating systems. And that's the lessons that we have learned. And so we've been working on this for 15 years on this hypothesis. And it turned out that it works and it works at scale. And we've tested it on very, very different and sometimes ridiculous kinds of problems from reverse engineering a real time distributed system, from a large company that is not allowed to trade with China today, to optimizing a data pipeline, to looking at performance, to doing migrations.

B (35:29)

People can see these examples in the glamorous toolkit. We'll put the link in the show notes. And one important thing is the book Rewilding Software Engineering that Tudor and Simon are developing at the moment. So make sure to check it out@multipledevelopment.com. but Tudor, if people want to know more about you, the work that you're doing, the glamorous, pardon me, Toolkit and your company think where should they go?

C (36:00)

So we're doing a couple of things. So indeed, multiple development is. If you want to learn about multiple development, which is the most important thing that we are producing is you can find more details about it@multiple development.com the book that I'm writing with Simon Worley. We're writing it in the open, so we're actually looking for feedback and we are encouraging people to read it. And we are also tackling explicitly, for example, the AI. So how does multiple development fit in an AI in an AI world? And it does fit now to not just talk about this. Our industry has been kind of dominated by people that talked about things first and then implemented them. And maybe sometimes they worked and sometimes they didn't. So we did it the other way around. So everything we are Talking about has been, we're talking about just first from firsthand experience. And a lot of it is encapsulated in glamorous toolkit. You can learn it about it@gtoolkit.com and everything here that you see that I described here is free and open source. Now the way we get funding to the way we get funded to make all of this free and open source is through Fink. Fink is a company and the way we get funding is we go and solve hard problems. And we work today with two kinds of two kinds of companies. On the one hand, companies that have some sort of a crisis related to legacy systems like the telco problem. On the other hand, we work with companies that seek a competitive advantage in software engineering. If we are correct, then we can optimize decision making in software engineering with at least an order of magnitude. And if this is correct, this can be directly transformed into a competitive advantage. And we already have a very large case study company that actually was a flagship case study in TTT by example in 2002. At that time they were the largest test suite that Ken Beck has seen with 4,000 tests. Today they have 150,000 tests and they have just moved and deployed and employ today moldable development. Is there main or centric practice and adopted glamorous toolkit in their case as well. So that's what we do at Fink as a way to get our funding so that we can produce the research further. So you can learn more about fink@fink.com.

B (38:40)

Yeah, and the link will be in the show notes so make sure you check it out. Fink.com and the book multipledevelopment.com and of course we'll put also Tudor's LinkedIn page in the show notes so that people can go and connect and ask questions and maybe discuss this further. Tutor, it's been a pleasure. Thank you very much for being here and for your generosity with your time and your knowledge.

C (39:03)

Well, thank you very much. I really enjoyed this conversation.

B (39:09)

All right, I hope you liked this.

A (39:10)

Episode, but before you hit next episode, here's the deal. This podcast is powered by people like you. The members who wanted more than just inspiration. They wanted real tools and real connection to people who are practicing Agile. Every day we're talking access to over 700 hours of agile gold, CTO level strategy talks, Summit keynotes, live workshops, E courses, Deep Dive interviews, books. And if you're into no estimates, we got the pioneers of no estimates in those Deep Dive interviews as well. Agile Business Intelligence, creating product visions, coaching, your product owner courses, you name it. You'll get invites to monthly live Q&As with agile pioneers and practitioners, plus a private Slack community where which is free.

B (40:08)

Of all of that AI slop you see everywhere.

A (40:11)

And of course without the flame wars. It's a community of practitioners that want to learn and thrive together. It's the best place to connect with community and learn together.

B (40:25)

So if this podcast has helped you.

A (40:27)

Before, imagine what you will get from this podcast membership. So head on over to scrummastertoolbox.org membership and join the community that's shaping the future of Agile. We have so much for you, so check out all the details@scrummastertoolbox.org membership because listening is great, it's important. But doing it together, that's next level. I'll see you in the community.

C (40:59)

Slack.

B (41:01)

We really hope you liked our show. And if you did, why not rate this podcast on Stitcher or itunes. Share this podcast and let other Scrum Masters know about this valuable resource for their work. Remember that sharing is caring.