David Sztykman on Streaming Data, AI, and the Future of Real-Time Observability | Ep 1178

Summary

---

Episode Overview

Podcast: The Digital Executive by Coruzant Technologies
Episode: David Sztykman on Streaming Data, AI, and the Future of Real-Time Observability | Ep 1178
Guest: David Sztykman, Chief Architect & VP of Product, Hydrolix
Air Date: January 7, 2026
Duration: ~10 minutes

This episode dives into the evolution of streaming data, the increasing scale and complexity of real-time observability, and the vital role of AI in anomaly detection and security operations. David Sztykman leverages his decades of experience to outline recurring technical challenges, principles of AI-driven operational intelligence, and predictions for the next wave of data platform advancements.

---

Key Discussion Points & Insights

1. Persistent and Evolving Technical Challenges

Timestamp: 01:48 – 02:34

• Constant Challenge—Scalability:
  Sztykman reflects on the growth in streaming system scale over the past 20 years. Today’s demands dwarf earlier events, such as the first Super Bowl streamed online.
  • “20 years ago, the scale of the event that we had was really tiny compared to what we have now… scaling is I think the hardest part of anything that we.” — David Sztykman (01:48)
• Growing Complexity:
  Increasing users and data volumes make sustaining performant, responsive systems especially difficult.

2. The Imperative of Streaming Data Architectures

Timestamp: 03:11 – 04:43

• Real-Time Insights are Critical:
  Modern enterprises require immediate visibility into operations—delayed analytics are no longer acceptable.
  • “Technically when you’re watching a video, for example, you don't want to know that the video had issue half an hour after—you want to know in real time what's going on to be able to accommodate that for your users.” — David (03:18)
• Dynamic, Unpredictable Load:
  Systems must gracefully handle unexpected spikes (like DDoS attacks or flash events), not just steady-state volumes.
  • “When you have something that's not planned, something that's massive, how do you scale, how do you manage that influx of new data that's coming in? And how do you do that dynamically is really hard.” — David (04:21)

3. AI & Anomaly Detection: From Reactive to Proactive

Timestamp: 05:22 – 07:23

• Preventing ‘Breaking the Bank’:
  Applying large language models (LLMs) or AI to every data point is computationally (and financially) unsustainable.
• Hybrid Approach for Efficiency:
  90% of data should be processed using cost-effective rules and heuristics, reserving expensive AI for the remaining anomalous 10%.
  • “When you have something that's unexpected, that's where you leverage an LLM and that’s where you leverage AI.” — David (05:54)
• Smart Routing to AI:
  • “If you build something smart enough that manages most of the data... but whenever something doesn't fit, that's when you leverage LLM. That really gives you the benefits of both worlds…” — David (06:38)
• Strength of LLMs:
  LLMs excel at interpreting the unexpected, given their diverse training sets.
  • “That's where LLM are really great at, because they have such a variety of training data that they can really figure out what it is, that it's abnormal and what it is about that data.” — David (07:23)

4. The Future of Observability: AI-Driven Agents & Democratized Data

Timestamp: 07:51 – 09:31

• Rise of Autonomous Agents:
  AI agents will drive automated detection and remediation—actionably querying data and executing responses.
  • “Agents are the future of automatic detection and repudiation... you're going to have a huge data platform that’s going to allow you to query that data and have agent querying information about it. And then... generate action directly into different vendors.” — David (08:01)
  • Example: Automated firewall rule adjustments triggered by AI.
• Agent-to-Agent Communication:
  Agents will increasingly interact, automating multi-system responses to security and operational events.
• Natural Language as Data Interface:
  AI is simplifying data access for non-experts; natural language queries enable anyone to gain insights without specialized training.
  • “AI assistant allows you to ask question in plain English and you can generate a query and get a response out of that is really brilliant. It's opening use cases and new things that we don't even think about.” — David (09:15)

---

Notable Quotes & Memorable Moments

• “Scaling is I think the hardest part of anything that we.”
  — David Sztykman (01:55)

• “Technically when you’re watching a video, for example, you don't want to know that the video had issue half an hour after—you want to know in real time what's going on to be able to accommodate that for your users.”
  — David Sztykman (03:18)

• “You can't really send it to an LLM or to any agent like that all the time because that’s going to cost a lot of money for that LLM to run, query and analyze that data all the time... when you have something that's unexpected, that's where you leverage an LLM and that’s where you leverage AI.”
  — David Sztykman (05:33–05:54)

• “Not everyone is a data scientist, not everyone likes to write SQL queries. And the fact that AI assistant allows you to ask question in plain English and you can generate a query and get a response out of that is really brilliant.”
  — David Sztykman (09:09)

---

Key Timestamps

• 01:48 — The challenge of scaling streaming architectures
• 03:18 — Real-time data needs: user experience and incident response
• 05:33 — Cost and strategy for AI-driven monitoring
• 06:38 — Hybrid design: rules for the known, LLMs for the unknown
• 08:01 — The future: autonomous agent-driven detection and resolution
• 09:09 — AI enabling natural language access to data

---

Flow & Tone

David Sztykman covers complex technical ground with practical, conversational clarity. He emphasizes the sometimes-overlooked operational costs of AI, the critical importance of fast responses, and exciting AI-powered directions for the field—particularly the democratization of analytics via natural language interfaces. The host’s questions keep the discussion focused and accessible, delivering actionable insights for technologists and executives alike.

---

For New Listeners

This episode is a succinct, insightful primer on the evolution of streaming data platforms, real-time observability, and the imminent transformation AI is bringing to security and data operations. You’ll come away with a clear understanding of the perennial technical pains, emerging AI-powered solutions, and how future platforms will empower a broader array of users—well beyond data scientists.

Transcript

[A]

Foreign.

[B]

Welcome to Coruscant Technologies, home of the Digital Executive Podcast. Do you work in emerging tech? Working on something innovative? Maybe an entrepreneur? Apply to be a guest at www.corazon.com brand welcome to the Digital Executive. Today's guest is David Stickman. David Sickman is a chief architect at Hydraulics and has two decades of experience in designing and building complex solutions for streaming content, web development, caching, security, observability and data analytics. He loves to tackle challenges and is curious to a fault. Prior to joining Hydraulics, David worked at cloud and security companies including ACMI and Elastic. His roles have included solutions architect, principal solutions engineer, senior technical architect, technical product manager, and now the Vice president of product where he leads Hydraulic's product team in designing and managing the company's streaming data lake used for observability, security, AIML and real time log analytics. Well, good afternoon David. Welcome to the show.

[A]

Thank you. I'm glad to be here.

[B]

Absolutely my friend. I appreciate it. And hailing out of Paris, France right now I'm in Kansas City, so we've got a little bit of a time zone triverse, but I really appreciate you making the time. So David, jumping into your first question, you've spent two decades building systems across streaming, caching, security and observability. What recurring technical challenges have stayed constant and which ones have fundamentally changed with the rise of real time data and AI?

[A]

So one of the biggest big thing I think that's constant is really the scale. Funnily enough, like 20 years ago, the scale of the event that we had was really tiny compared to what we have now. If you think about the first super bowl that was streamed online in 2012, it was a tiny event compared to super bowl events right now. And so one of the theme that really is striking is how scaling things is difficult because historically when we work on something we think that yeah, this is easy, it's going to work, but. But as we have more and more users, as we have more and more data, that's where it becomes really problematic to ensure that the system can continue growing properly and responds properly with more and more demands basically. And so yeah, scaling is I think the hardest part of anything that we.

[B]

Do totally understand and what's cool about your platform is you do a lot of that. You know, you can scan petabytes of data in no time, which is really cool. But yeah, scaling events from 20 years ago, probably a little bit of a challenge back then, but it has massively grown obviously and as long along with challenges of course. So appreciate Those insights. David, at Hydraulics, you lead product for a streaming data lake used for observability, security and AI. ML, what makes streaming data architecture so critical for modern enterprises compared to traditional data warehouses?

[A]

So if you think about the use case that Hydraulics is solving, people needs to have information in real time about what's going on. Technically when you're watching a video, for example, you don't want to know that the video had issue half an hour after you want to know in real time what's going on to be able to accommodate that for your users. And so one of the challenges part of really like big data analysis like that is as the data is growing, typically you usually add delay into ingestion and into analysis of that data, because like handling a terabyte a minute of data and handling 10 gigabytes a minute of data is really different obviously. And so the challenge really is like, how do we keep going down as fast as we can and going as real time as we can with an evolving amount of data that keeps coming. And so that's really one of the big challenge and why people care about it is because people really need to see in real time what's going on at the edge of their platform. If you think about the security event, you can't learn about a security incident half an hour after it's ongoing. And if you have a DDoS event, for example, you need to be able to react quickly about it. And the problem is, accommodating to a huge amount of logs that are ongoing is difficult. You typically build your infrastructure to manage your daily basis. But when you have something that's not planned, something that's massive, how do you scale, how do you manage that influx of new data that's coming in? And how do you do that dynamically is really hard.

[B]

Absolutely. That is certainly a challenge. Businesses, people need insights in real time today, especially if there's issues, they need that now versus after the fact, as you mentioned, the challenge is the amount of growing data to scan quickly, you know, within minutes or even seconds. But having a platform that can do this, a technology that's dynamic and being able to give insights real time is really the key here. So I appreciate that. David, what role do AI driven pattern detection, anomaly recognition play in helping teams move from reactive troubleshooting to proactive incident prevention?

[A]

So there's a lot of things that are ongoing right now in the industry. One of the key thing for me is really to understand how AI is helping us without breaking the bank, by the way, because if you have a lot of data that's ongoing and incoming all the time. You can't really send it to an LLM or to any agent like that all the time because that's going to cost a lot of money for that LLM to run, query and analyze that data all the time. So what you want to do is build something that basically allows you to get 90% out of the way. So you know, that's the incoming data, that's the expected format, that's everything that you expected. But when you have something that's unexpected, that's where you leverage an LLM and that's where you leverage AI. And I think really that's the key part here, is really to build a platform that allows you to accommodate most of the use cases and most of the things that you expect. And when you have something that's unexpected, that's when you need to leverage AI tooling to be able to analyze that data quickly and figure out if it's a normal event or if it's something that needs to be triggered as abnormality. And really the idea of sending all your data points to an agent or to an LLM becomes really expensive and slows down the whole process. But if you build something smart enough that manage most of the data that's incoming, but whenever something doesn't fit, that's when you leverage LLM. That really gives you the benefits of both world in the sense that you have your platform that can scale to a massive amount of incoming data that you know and you can manage. And when you have something that you don't know, that's when you leverage AI to figure out what it is.

[B]

Thank you. I really appreciate that you talked about. The key is how AI is helping us with that without that large investment breaking the bank, as you said. But again, it's all about how we review that data. Like you said, if it's 90% of it, structured, formatted, you know what it is, that's great. But that other 10%, when you have some unstructured data maybe, or an anomaly or something, is where you said to leverage that.

[A]

And that's where LLM are really great at, because they have such a variety of training data that they can really figure out what it is, that it's abnormal and what it is about that data. And so you need to leverage your right tool for the right thing. For sure.

[B]

Absolutely. Thank you, I appreciate that. David, looking ahead, how do you see AI reshaping the future of observability and security operations over the next Three to five years. And what skills will engineers and product leaders need to keep up?

[A]

One thing that's becoming clear is I think agents are the future of automatic detection and repudiation. So if you think about it, you have a big data platform that contains everything or a lot of the data that you have. But at the end of the day that data is not being activated or is not being triggered. And so if you have agents that are querying information about that data and then generating action out of that, it's really what the solve was supposed to be for security event where at the end of the day it would automatically trigger remediation rules. And I think agent to agent communication is really going to enable us to do something similar to that use case where you're going to have a huge data platform that's going to allow you to query that data and have agent querying information about it. And then based on the type of agent and the type of rules that has been configured for it, then they're going to generate action directly into different vendors. So for example, blacklisting an IP on your firewall list or things like that are going to be fully automated at some point through AI communication and agent to agent communication here. The other big piece of things which is really important to us is how AI is opening data to everybody. Not everyone is a data scientist, not everyone likes to write SQL queries. And the fact that AI assistant allows you to ask question in plain English and you can generate a query and get a response out of that is really brilliant. It's opening use cases and new things that we don't even think about. It's going to open data platform to really users that were not used to get access to that data or used to have like data scientists to work on that with them.

[B]

Thank you, that's awesome. A couple things that you highlight or I highlighted here from what you think the future, obviously agent to agent communication, that's been a big topic recently and I definitely see that coming into play. But the fact that AI is opening data to everyone, it's allowing for natural language processing, right? For everyday user to become basically a data scientist, which I think is amazing. So I appreciate your insights on that. And David, it was such a pleasure having you on today and I look forward to speaking with you real soon.

[A]

Nice meeting you and nice talking to you.

[B]

Bye for now.