Unknown Host (3:02)

And what time is it for you right now?

Kakeru (3:04)

It's 5:00pm all right. So I know you are in early morning, right?

Unknown Host (3:08)

It's 9am for me it's not too bad.

Kakeru (3:10)

I'm sorry to wake you up earlier.

Unknown Host (3:12)

It's fine. It's 9am I am actually. It's funny I live in Sweden, but I am in the very far north part of Sweden. I'm in a ski resort very, very far up north. So wherever I look, there is just snow everywhere right now.

Kakeru (3:23)

Nice. In Tokyo it's rarely see the snows and I like snows.

Unknown Host (3:27)

Oh yeah, I seen. I have a friend who went to Sapporo and I think it snows in Sapporo, right?

Kakeru (3:32)

Yeah, Sapporo is really cold place and it's good place for the skiing maybe. But here in Tokyo it's relatively warmer than Sapporo.

Unknown Host (3:41)

Got it, got it.

Kakeru (3:42)

Really see snow recently. Got it.

Unknown Host (3:44)

All right, so. All right, let's talk about this tool that you guys have open sourced recently, the Kubernetes History Inspector. Can you tell us what's that?

Kakeru (3:52)

Yeah, sure. It's a leech log visualizer designed for troubleshooting Kubernetes issues. Before this tool existence, we needed to troubleshoot the pod troubleshoot or something by checking the content log. But if the problem can't be solved content log alone, they needed to rely on various kind of logs like Kubelet log container log Container D log Kube API server log Kube Container Manager log. There are so many various logs needed to be used in troubleshooting. So it require high expertise to craft the log filter or something to gather these logs and investigate them because it just generate least tons of the logs. And we needed to understand what's happened in the past around the port just from the lines of logs. It was so hard for us. So then this tool will provide us detailed timeline visualization or resource relationship diagram just from logs available on your log backend. Currently this is only supporting cloudlogging, but we are expanding this support to the other clusters, especially for the open source Kubernetes class.

Unknown Host (5:10)

Nice. And so one important detail actually that I want to talk about is that it says history in the name, which means that the tool actually allows you to go back in Time.

Kakeru (5:20)

Yes. So when I say that inspector or something related to Kubernetes, maybe users think this is a kind of agentful tool like Promises or something. But actually this is just a log visualizer. It's visualize the history of the cluster resources just from logs by parsing all of them.

Unknown Host (5:41)

Yeah, we're going to talk about it, but it's important to understand you don't need to install anything in your cluster. I tried this yesterday. I just fired up the cloud shell in the console and started the docker container. And as long as it has permissions to pull the logs, it will just pull them, right?

Kakeru (5:56)

Yeah.

Unknown Host (5:56)

And give you this rich visualization as you said. Where did the idea of open sourcing the tool came from?

Kakeru (6:02)

Well, I am working as a support engineer in Google and I needed to troubleshoot customer cluster issues when I got the ticket. But the problem is maybe that would be my first day to see the customer cluster or even the customer said my port was dead yesterday or something. But when I touch the cluster, maybe the cluster is running healthy without any issue. But the troubleshooting the current ongoing incident is easier than troubleshooting past issue because I can interact with cluster if customer allow it. However, the troubleshooting past issue require me to see through many kind of logs and it takes a really long time. So I wanted to understand the macroscopic view of the cluster. So that's why I needed to create it. And after I created the prototype, I showed this to my colleague or other support chain members and it gained popularity among my support team or many other teams in Google. And I decided to make this available to the other customers on the Internet.

Unknown Host (7:20)

Nice. Nice.

Abdel Sighiwar (7:22)

Yeah.

Unknown Host (7:22)

So the tool is obviously on GitHub, so everybody who's listening to us, you should check it out and maybe give it a little star if you want. It's very easy to start. It's just a docker container. So I did a little bit of troubleshooting and support back in my days before my current role. And I remember yes, like troubleshooting past problems is difficult, but also troubleshooting problems that doesn't happen very often. Right. When you have like a transient issue. So how was this like tool helps like support engineers particularly like how does that help them in troubleshooting these kind of problems?

Kakeru (7:56)

Well, this is a very important for the support chain because declaring the logs is also declining in the skills. Like at first we needed to understand the explicit time or the incident happens. But once this tool was used in support chain, after querying the logs by one Support agent. The support agent can show the visualized timeline with the other support engineer handed over. So they will continue troubleshooting with detailed understanding of the cluster.

Unknown Host (8:30)

I see, I see. And so can you talk about some issues that the tool have helped you actually solve that Kubernetes history Inspector have helped solve?

Kakeru (8:38)

Well, let me introduce a little difficult complex ticket before. All right, so this is about the GKE with workload identity, which is a feature to get the access token to access the GCP API from the port with checking the permissions granted for the Kubernetes service account. So my customer told me to troubleshoot the intermittent error over a workload identity. I mean, customers say my port got the authentication error intermittently, but it only happens very few times in the month or something. And I realized that is caused by third party security product restarting container D because workload identity needed to communicate with containerd to verify the port is actually running on the node to return the access token. But to troubleshooting this kind of issue, I needed to check customer port log, container D log, third party security product port log, and also I needed to check the workload identity system workload logs. So the crafting the log query with checking all the logs would be a little hard. And even if I could get the list of the lines of the logs, it won't make sense. But this tool will help me to show how these logs happen. And I can see the kind of line of the dots of the logs happening at the same time over the visualization. So I could easily see, oh, this is caused by container DD starting or something triggered by third party security product. So this kind of difficult problem involving multiple components on the cluster can be solved with Ketchi easily.

Unknown Host (10:41)

Yeah, and so I think what's important in what this particular use case you're talking about is the correlation part. Like how can you correlate events happening in multiple parts of the system using logs and understand that those kind of events are related to each other. Like you are talking about like workload identity, which has its own pod. Then you have the customer pod, then you have container D and then you have the pod of this third party security tool. And you have to like pull the logs across all these things and kind of like try to understand how they line up with each other kind of. Right. So I was using the tool yesterday, as I said, I like, I fired up the docker container, I launched the interface. It has a graphical interface, a rather interesting graphical interface because it's built in WebGL. I haven't heard of WebGL for a very long time, but we're going to talk about it. But how does that correlation work? Like is it the tool that will correlate these things together or is it you that you have to like try to find that correlation?

Kakeru (11:35)

Well, is it meaning users needed to customize correlation settings or something?

Unknown Host (11:41)

Or does the tool find like those events and then says okay, these events are related in time? Like how does that work?

Kakeru (11:49)

So basically KHI has various parsers implemented already on its code base. So it passed the structured log fields and it decides which resources is related to the logs or something. But the log parsers are not so such simple. Like some log parser needed to be run before the other log parser or something. For example, containerd log parser shows a container behavior with container id, but it won't show any port name or something. Yeah, but Kubelet also show container ID and port name. So I can correlate the container ID with a port ID because I'm running the container ID parser after getting the port name from the other logs. But this is a little hard to, you know, because many parsers depend on the other parsers. So KHI is basically based on the directed acyclic graph. It's DAG based log parser system. So that makes KHI to be extensible. Currently we are not publishing any document how to extend KGI log parser but I will do that later and that helps customer to extend their log parser to support their own custom control or something.

Unknown Host (13:17)

I see.

Kakeru (13:18)

And this extensibility makes KHI to support many kind of dialogues.

Unknown Host (13:23)

Okay, cool.

Abdel Sighiwar (13:24)

Yeah.

Unknown Host (13:24)

So then I want you to talk to us quickly about how it actually works behind the scene. So I tried it. So it's essentially you start the docker container, you select, well in our case you select like a project id, then you select which cluster you want. Then there is like a little thing happening on the interface that says I'm running, I'm pulling some logs and then you get an interface. But how does it actually work behind the scene?

Kakeru (13:48)

Well, each parser has a dependency of the form, like okay, this parser needed to have the project ID before querying or something like that. This dependency is defined on each parser as a DAG based graph. So the cache is just running this graph based task runner and generate one single log bin file and then that would be parsed from the front end and showed that a diagram.

Unknown Host (14:17)

I see. And as I said, it uses WebGL for the rendering of the interface. Right.

Kakeru (14:22)

Yeah.

Unknown Host (14:23)

Did you learn WebGL to build this.

Kakeru (14:25)

Or did you work with WebGL before joining Google? Actually I joined Google at New Grad, but my hobby was doing some open source work especially for the WebGL framework.

Unknown Host (14:39)

Okay.

Kakeru (14:39)

So I made a WebGL framework before joining Google. So that's why I had experience with WebGL.

Unknown Host (14:45)

I see.

Kakeru (14:46)

So for me the usual web development is harder than the WebGL for me because I needed to learn AngularJS to build this application. But for the WebGL side, I know really basic and I could realize this performance visualization with existing my knowledge.

Unknown Host (15:08)

I see, I see. All right. And so I think I have to ask this question because we are in 2025 and AI is all around us these days. So before I go there, the tool is pretty much in memory only, right? So when you fire up the container, all the logs are in memory, right?

Kakeru (15:24)

Yeah, yeah. Actually that is intended to be in memory. So if the tool was for the storage, I think that should be done in backend. But I build the application for investigate quickly. So that's why I wanted to take all the logs on the memory on front end side. Yeah, that design is intentionally and yeah, that works on the memory.

Unknown Host (15:52)

Nice. And so do you see a future in which an LLM could be integrated into KHI and help troubleshoot issues?

Kakeru (16:01)

Well, maybe I can consider about it. But the important role of the KHI doing taking part in AI era would be like even LLM said. Okay, this problem happened because this configuration issue or something, and this is intermittent issue happened by Tollyguard by this board or something. Maybe the people couldn't be convinced so much. So they wanted to understand why that happens with visualization not just by text.

Unknown Host (16:36)

Yeah.

Kakeru (16:37)

So that's why this visualization is still important for the LLM.

Unknown Host (16:42)

I see, I see. Nice.

Kakeru (16:44)

Cool.

Abdel Sighiwar (16:45)

Well, this is actually pretty cool.

Unknown Host (16:47)

So I highly recommend people to go check it out. It's on GitHub. We'll leave the link in the show notes. Go check it out. Go try it out. Give it a star on GitHub. If there is any features missing, either implement it or open an issue, I guess. Right. Kakeru is on Twitter. But your Twitter is mostly in Japanese.

Kakeru (17:04)

Yeah. So you can follow me. But maybe it's only limited for. No, no. It will be a little hard for the non Japanese speakers.

Unknown Host (17:16)

Yeah. So that's fine. Maybe then can talk to you on GitHub.

Kakeru (17:19)

Yeah, yeah.

Unknown Host (17:20)

Well, thank you for joining us on the show, Kakuru.

Kakeru (17:21)

Thank you.

Abdel Sighiwar (17:24)

That brings us to the end of another episode. If you enjoyed the show, please help us spread the word and tell a friend. If you have any feedback for us, you can find us on social media ubernitespod or reach us by email at kubernetespodcastgoogle.com you can also check out the website kubernetes podcast.com where you will find transcripts and show notes and links. To subscribe. Please consider rating us in your podcast player so we can help people find and enjoy the show. Thank you for listening and we'll see you next time.