#724: Accelerated computing: From fraud detection to AI innovation

Summary

AWS Podcast Episode #724: Accelerated Computing – From Fraud Detection to AI Innovation

Released on June 9, 2025

The 724th episode of the AWS Podcast delves into the transformative world of accelerated computing, exploring its pivotal role in powering advanced AI and machine learning (ML) applications across various industries. Hosted by Shruti Koparkar, the episode features insightful discussions with two AWS experts, Ray and Sudhir Kaldendi, who shed light on the challenges, architectural considerations, and real-world applications of GPU-accelerated computing on AWS.

---

Introduction to Accelerated Computing

Shruti Koparkar opens the episode by introducing the concept of accelerated computing on AWS, emphasizing its significance in AI and ML workloads. Accelerated computing leverages powerful hardware like Nvidia GPUs and AWS’s proprietary AI chips—Trainium and Inferentia—to enhance computational performance for complex tasks.

---

Segment 1: Accelerated Computing with Ray

Ray’s Role and Expertise

[00:00] Shruti Koparkar: "Hello everyone and welcome to another episode of the AWS Podcast... Today we are going to dive into a couple of different use cases for accelerated computing."

[01:06] Ray: "I'm a container specialist, solutions architect... My primary role is to now work with customers that are trying to build these massive systems on Kubernetes... especially for machine learning and generative AI solutions."

Ray, an experienced solutions architect at AWS, specializes in container orchestration and helps customers architect scalable ML applications using Kubernetes and Amazon EKS (Elastic Kubernetes Service).

Key Challenges: Maximizing GPU Utilization

A significant challenge highlighted by Ray is the maximization of GPU utilization. Unlike CPUs, GPUs are costly, and underutilization can lead to substantial financial losses. Ray states:

[05:37] Ray: "The biggest challenge... is that the GPUs... are very expensive. Anytime you're using the GPU you need to ensure that you're maximizing its usage."

He points out that applications often fail to fully utilize GPU capacity, sometimes achieving only 30% utilization, which is inefficient given the high costs associated with GPU resources.

Architectural Considerations for GPU Workloads

Ray discusses critical architectural considerations for deploying GPU-accelerated workloads on EKS:

• Storage: Utilizing fast, distributed storage solutions like Amazon FSx for Lustre to ensure rapid data access and minimize latency.

  [13:51] Ray: "FSX for Lustre is a great service for anyone that's looking to do distributed training... It allows hundreds of instances to read simultaneously."

• Networking: Ensuring low-latency connections between GPUs using Elastic Fabric Adapter (EFA) to facilitate efficient data exchange during distributed training.

• Resource Management: Balancing cost and performance by selecting appropriate GPU types based on workload requirements.

Customer Success Story: Rivian

Ray shares how Rivian, an automotive technology company, leverages accelerated computing on AWS:

[15:24] Ray: "Rivian has built a stack on top of AWS data on EKS... They use the Jark stack—Jupyter, Argo, Ray on Kubernetes—to run distributed jobs efficiently."

Rivian utilizes Argo Workflows and Ray to manage and orchestrate large-scale ML tasks, ensuring high GPU utilization and streamlined workflow management. By optimizing job scheduling and co-locating related tasks, Rivian maximizes the performance and cost-effectiveness of their GPU resources.

Nvidia NIMS and Karpenter Integration

[19:37] Shruti Koparkar: "Nvidia has launched NIMS... How do those work with EKS?"

[20:13] Ray: "NIMS addresses maximizing GPU efficiency by providing optimized containers that adapt to different hardware configurations."

Ray explains that Nvidia Inference Microservices (NIMS) offer pre-packaged, optimized containers for various ML models, simplifying deployment and ensuring optimal performance across diverse hardware setups.

Furthermore, Ray introduces Karpenter, an open-source Kubernetes autoscaler:

[23:02] Ray: "Karpenter optimizes compute scaling by dynamically provisioning the right EC2 instances based on workload demands."

Karpenter automates the scaling process, allowing EKS to adjust resources seamlessly in response to application needs, thereby enhancing resource utilization and reducing costs.

---

Segment 2: Accelerated Computing in Financial Services with Sudhir Kaldendi

Sudhir’s Role and Expertise

Transitioning to the financial sector, Sudhir Kaldendi, a principal solution architect specializing in payments at AWS, discusses the application of accelerated computing in fraud detection.

[27:03] Sudhir Kaldendi: "Financial institutions face the challenge of processing vast amounts of transaction data in real time, requiring robust infrastructure and efficient data processing."

Fraud Detection with Accelerated Computing

Sudhir highlights how financial services leverage GPU-accelerated instances to build sophisticated fraud detection systems:

• Data Processing: Utilizing Nvidia Rapids integrated with Amazon EMR to accelerate data processing pipelines.

  [35:55] Sudhir Kaldendi: "Nvidia Rapids speeds up data processing and machine learning pipelines, enabling faster fraud detection and cost savings."

• Machine Learning Pipelines: Implementing frameworks like Nvidia Morpheus and Triton Inference Server for real-time transaction analysis and model deployment.

• Scalability and Cost Efficiency: Combining AWS services like Amazon SageMaker with Nvidia technologies to achieve up to 14 times faster data processing and model inference, while significantly reducing costs.

Architectural Considerations for Real-Time Fraud Detection

Sudhir outlines key architectural components essential for building real-time fraud detection systems:

• Data Storage and Retrieval: Efficiently storing and accessing vast volumes of historical and real-time transaction data using Amazon S3 data lakes.

• Data Correlation: Integrating historical data with emerging trends to identify patterns indicative of fraudulent activities.

• High Throughput Processing: Leveraging Triton Inference Server to handle up to 350,000 transactions per second, ensuring swift and accurate fraud detection.

[30:19] Sudhir Kaldendi: "Using the Triton inference server, we could process close to 350,000 transactions per second, which is crucial for identifying fraud in real time."

Customer Success Story: Feature Space’s ARIC Risk Hub

Sudhir shares the success of Feature Space and their ARIC Risk Hub:

[33:25] Sudhir Kaldendi: "Feature Space processes over 100 billion events annually, leveraging AWS scalability and Nvidia GPUs to deliver impressive fraud detection rates."

By utilizing AWS’s elastic computing power and Nvidia GPUs, Feature Space has developed a dynamic platform capable of real-time fraud detection with high accuracy, effectively mitigating financial risks.

Convergence of AI Technologies

Sudhir elaborates on the synergy between various AI models in enhancing fraud detection:

[38:26] Sudhir Kaldendi: "Graph neural networks analyze complex transaction patterns, while large language models process unstructured data like invoices and emails. Together, they provide a comprehensive fraud detection system."

The integration of Graph Neural Networks (GNNs), Large Language Models (LLMs), and Large Transaction Models enables financial institutions to detect fraudulent activities more accurately and efficiently, reducing false positives and enhancing customer trust.

---

Conclusion

Throughout the episode, Shruti Koparkar facilitates a deep dive into the intricacies of accelerated computing on AWS, highlighting its critical role in driving AI and ML innovations. From optimizing GPU utilization with Kubernetes and EKS to empowering financial institutions with real-time fraud detection capabilities, accelerated computing stands at the forefront of technological advancements.

[26:44] Shruti Koparkar: "Thank you so much, Ray... you provided a great overview of what folks should be thinking about when they are running GPU workloads on EKS."

[40:56] Shruti Koparkar: "That's it for this episode, everyone... until next time, keep on building."

For more insights and updates, listeners are encouraged to connect with Shruti Koparkar on LinkedIn or X, and provide feedback via email at [e-mail].

---

This episode underscores the pivotal role of accelerated computing in modern AI applications, providing actionable insights for developers and IT professionals aiming to harness the full potential of AWS’s GPU-powered resources.

Transcript

[Shruti Koparkar]

This is episode 724 of the AWS podcast, released on June 9, 2025. Hello everyone and welcome to another episode of the AWS Podcast. My name is Shruti Koparkar and today we are going to do something fun. We are going to dive into a couple of different use cases for accelerated computing. This is essentially the compute on AWS that is accelerated using Nvidia GPUs as well as the AWS AI chips, AWS Trainium and Inferentia. And it's purpose built for AI and machine learning workloads. And so we are going to discuss how excited computing is enabling success for several different customers across different verticals. And we are going to do that by chatting with a few different guests on this episode, starting with Ray. Ray, welcome to the show and can you please introduce yourself?

[Ray]

Thanks for having me, Shruti. So I'm a container specialist, solutions architect. I've been with AWS for about eight years and for most of my journey I've worked with customers trying to architect applications in the cloud. And yeah, so my primary role is to now work with customers that are trying to build these massive systems on Kubernetes and a lot of them is accelerated computing, since everyone is currently now trying to run some kind of machine learning solution or some kind of generative AI solution. So a large, large part of my work now is working with customers that are trying to build ML applications within their organization. So they're trying to build these platforms that enable hundreds of machine learning experts to build models and do things like distributed training or do model serving or serving large language models, et cetera. So that's gonna do what I do, right?

[Shruti Koparkar]

That's really cool. So just to elaborate that on a little bit, there could be teams who are directly using Amazon Bedrock, for example, and just building generative AI applications with that. But then we also have customers who have internal teams who are building their own AI ML models or sometimes fine tuning, sometimes pre training and then deploying. And you work with the teams that help these other teams at a customer do that. So the teams that you work with typically are the ones that own the AIM and infrastructure piece, where they are responsible for securing the GPU capacity or the accelerated instances powered by Trainium or inferential capacity, and then figuring out how to manage that, the cluster management, sort of tracking utilization, all of that. Is that sort of what you were trying to describe?

[Ray]

Yeah, yeah, exactly. So there's a vast array of customers that are using AWS ML solutions or building machine learning applications on aws. And on one hand you do have customers that really just don't want to invest anything on building net new applications on aws, but would still like to integrate some kind of machine learning solutions. And for that, a turnkey solution like Bedrock is perfect where you don't have to understand infrastructure. You can just say, hey, I just want to be able to use a large language model for that Bedrock. It's amazing. And then you also have customers that want more control, that are building their own models that really would like to tune the underlying hardware for the specific use case that they may have. So for example, we have many, many customers that do object detection when they may be simulating like a car going through a highway and what happens when all of a sudden it sees a pedestrian crossing. So that may not be a real world solution situation. That entire scenario, driving scenario may be a generated scenario. So for that you may need really, really a lot of compute. Sometimes it's a distributed compute jobs. We are talking about tens of GPUs. This is very similar to how customers also train large language models or vision models. So these customers would benefit from having the underlying access to the underlying hardware because their needs are very unique and you know, not what most customers would use these models for. You know, they're building their custom models.

[Shruti Koparkar]

Yeah, yeah, that makes sense. So you know, as organizations scale their AIML workloads, as you said, many of them are turning to Kubernetes and specifically Amazon eks. So what are some of the key challenges you are seeing as customers try to orchestrate these GPU accelerated workloads at scale?

[Ray]

Yeah, so I think like the biggest challenge that the entire compute community faces with these new workloads is that the GPUs that power these workloads are very expensive. That means anytime you're using the GPU you need, you as a user need to ensure that you're maximizing its usage. And that's where we see a lot of use cases suffer where the GPU may only be utilized 30% of the time. Right. And that may not sound quite a lot because you know, if you come from the old infrastructure world, where I came from, we go for a CPU percentage. And CPUs are okay. You, you know, they're not that expensive. If you run 30%, that's fine. GPUs are expensive. You know, we're talking about thousands of dollars that get wasted because our current applications are unable to maximize the value out of GPUs. Okay, so that's the current Challenge that's face across the board. Anyone that's using GPU for machine learning, whether they are training models, where they're hosting models, they are struggling with maximizing the, the utilization of the gpu. Okay, so and Kubernetes community is not unique. We're still running the same application. So that's where the challenge is, is to how to maximize the gpu. How do you make sure that the GPU cycles are fully utilized so you're getting max, max the value out of your money. And the benefit with Kubernetes is also that there are many, many frameworks that are tuned and designed to optimize GPU utilization. Like their whole concept is how do we make sure we serve a workload. For example, let's say you may say I want to be able to build a service that translates this text from English to French, right? And you may build this and thousands of people are using this across the board. But each person may be translating different lengths of text. You know, you may be translating a sentence versus somebody else, maybe translating a whole page. Now if you stack them up and want to serve them together, there's no really good way to easy way to optimize it because you don't know how much a user may be trying to translate. So you now need to have some kind of intelligent way to batch them together so that you maximize the amount of GPUs you may have. And so these are complex algorithms and there are frameworks out there like Vllm that are designed to maximize the GPU GPU utilization. And these products are increasingly easier to deploy on Kubernetes. So where we see a lot of customers that have, that already have significant expertise in Kubernetes, they have teams that already manage tens of Kubernetes clusters in the environment. For them, moving to Kubernetes, moving to deploying EKs for their ML solutions becomes just a natural progression of their, of their architecture.

[Shruti Koparkar]

Makes sense. Now what are some of the architectural considerations then when they are using Kubernetes for these GPU accelerated workloads, what are some of the design decisions that the team need to be thinking about regarding, you know, around networking or storage or resource management as they use EKs.

[Ray]

Yeah, yeah. So if you're going into the self managed world, which is where I live, it's you're architecting an end to end solution. And so whenever you're building a solution like that, you're not thinking in isolation. You're also keeping things like cost in mind. So firstly it's it's, it's very difficult to give a generic, generic answer because a. We're still learning, this is still new for many of us in the community. But also it also depends on the type of workload that you may be running and the amount of traffic you have. I think the, the biggest way you look at it is what do. What is the need for my, for my use case? Am I building a chatbot? Am I building a document processing solution? What is the solution and how would I consider my solution to be success? Is it going to be speed? Is it going to be accuracy? And so that is really going to define what is going to be the metric by which you're going to gauge success. And once you're going to get that metric, you would find out what is the actual. Then you would figure out what is how much money am I going to make from building this solution? Because GPUs are hard. You'd have to have a solid plan on how are you going to recuperate this cost based on the amount of benefit that you have downstream from building this application. You would then define the architecture. So there's a wide AWS gives you a huge variety of options from compute to storage to, you know, the type of storage to type of GPUs. So you do have a bunch of options like, you know, we have customers that are building the next generation LLMs on AWS and then we also have customers that are not trying to do the most advanced thing, but would still like to benefit from having the option of having different types of GPUs being available. So I have the options of using not the latest generation Nvidia gpu, but I can use maybe a few generation old because it's fine for my workload. So first you define what that metric is, how much, what kind of compute, what kind of performance you would need, and then you work backwards from that. So if you really need fast storage, fast distributed storage, you would then start working through like what are the storage options in aws. And I would go through each one of those options. As an, as an architect, you would go to each one of those options and say, you know, this is the most appropriate option, but hey, this is also the most expensive option. Where do I find that balance? So it's, it's the kind of thing that you do, but in, in terms of distributed computer, you are looking at the fastest. Whenever you're talking about multi GPU solutions, you're, you're looking at having the fastest link network wise between the two GPUs available. So on AWS, that's EFA, the elastic fabric adapter. So you definitely want to make sure that there's the lowest amount of latency between your GPUs. You definitely want to make sure that they're sitting as close as possible. So in terms of aws, it's the same az, you want to reduce any kind of storage latency. So GPUs often have to copy data from disk into memory. And if your GPU and disk. But the way this operation occurs is there's also CPU in between. So if your CPU is a bottleneck, if your disk is a bottleneck, then your GPU is just waiting, just trying to extract data from the disk. That's not a good situation to be in. You want a GPU to be like always be processing. You know, it never waits for data, it never waits to exchange data between other GPUs when you're doing distributed training. And it never has to wait to. To load data from the disk because you're wasting dollars there.

[Shruti Koparkar]

Right. Is that when something like Amazon FSX for Lustre comes in is where it can really feed data very fast to the GPUs?

[Ray]

Absolutely, yes. So FSX just recently added the support for GPUs to directly read data from a distributed storage. So that is really, really helpful because you bypass the cpu. And FSX for Luster is a great service for anyone that's looking to do distributed training, because in that situation, what you're looking to do is you have a vast corpus of data, basically a bunch of files, and these could be terabytes to petabytes of files, and you may have hundreds of GPUs trying to create data from each one of these files. And so that's why you need distributed stores. That's where you need the ability to scale to that kind of hundreds of instances reading at the same time. So FSX for Luster is great for that kind of use case. Yes.

[Shruti Koparkar]

Awesome. Okay, so let's dive into a customer example. And I know that you've worked closely with Rivian, and they are obviously using a lot of GPUs and together with EKs. Can you walk us through that use case, perhaps? And like, sort of, to the extent that you can share, what were some of those sort of criteria in terms of performance or budget or time to market that they were trying to hit? And then what did that mean for their solution architecture that, you know, AWS and them designed together?

[Ray]

Essentially what they have built is they have these, these massive jobs that may be doing things like object detection. Okay, that's like, let's take that as a generic workload. You may be running a car and simulating that through a scenario where you may say that a car goes from, from, from spot A to spot B and during that spot there are a bunch of objects that it has to detect. Maybe it has to stop for pedestrians or stop at a stop sign or slow down at a school crossing, things like that, you know. So these tasks are typically multi GPU tasks and sometimes these may take tens of instances. P5 instances and P5s are, are pretty, pretty expensive instances. They have eight Nvidia H100 GPUs if I'm correct. But these, these are massive instances and they are also very very difficult to get in terms of availability. So when you do get them, you want to make sure that you're maximizing the utilization. So what, what Rivian has built is that they built a stack on top of this project that AWS started, which is data on eks. And as part of this project we have an ML reference architecture implementation called the Jark stack, which is Jupyter Argo Ray on Kubernetes. And so Rivian has extended that stack and named the GTC talk there is. They've gone through the, the architecture in, in detail but what that stack does is it runs this scenario across a distributed Ray job. And that distributed Ray job is ARC is orchestrated by Argo. So it runs in on a bunch of EC2 nodes that are Kubernetes cluster worker nodes. And the way it's it does it is that they have a bunch of EC2 instances and they have a bunch of jobs running. And their job as interest at the infrastructure team is to ensure that a the number of jobs remained low and there are not many pending jobs in the queue because at that time a developer is waiting, wasting their time and at the same time try to not make sure, try to ensure that there's not a P5 instance per developer. So they want to ensure that whatever the number of instances they have, they're utilizing it it fully and efficiently. So the way they, they use this, they use Argo workflows, which is a workflow engine that runs on top of Kubernetes and they use Argo and EKS to then make sure that hey, I have let's say 200p5 instances, these are the number of jobs that I have. And Kubernetes then makes sure that they have the loads the jobs get loads of scheduled as needed and so a lot of this is also bad scheduling. So they also implement bad scheduling to make sure that the jobs are co located and if they need to intercommunicate, the latency is low, et cetera.

[Shruti Koparkar]

Awesome, awesome. I think you mentioned Jupyter Argo for the ARGO workflows, which is open source container native workflow engine. And then you also mentioned Ray, which is a distributed computing framework that is designed to build and scale AIML applications. Is that right? Yeah. And I think Ray by any scale, anyscale is one of AWS partners.

[Ray]

Yes, yes, yes.

[Shruti Koparkar]

Awesome, awesome. So one other question I had since we are talking about containers is Nvidia has launched nims, which is the Nvidia inference microservices, which is essentially sort of a container that comes packaged with the LLMs and all the optimizations and libraries that might be needed. How do those work with eks? Have you seen customers sort of use NIMS with EKS and across, you know what verticals? Have we seen those?

[Ray]

Yeah, so, so what is nims? First of all, NIMS is also, it's addressing the same problem that we started with, which is how do I ensure maximal maximum efficiency of a gpu? So that's one challenge. The other challenge also is that how do I ensure that for I can run models on the limited hardware that I have? Okay, so not everyone has the luxury to, to get a P5 instance, I, I, I can't afford it, but I still want to be able to, to run, let's say an open source model. And so a lot of times what you have to do is that you have to do a lot of model customization to be able to run it on a commodity EC2 instance. You know, it's it bedrock or like chat tools really make it really super easy to start chatting. But if you were to deploy your own large language model, it's not that easy because you may not have the latest hardware, you may not have the best hardware, you may not have multiple EC2 instances. So a lot of times you may have to do a series of optimizations to be able to get it to run on your own hardware, which is okay, you can do it, but not everyone knows the best techniques and not everyone knows what to trust, you know, at this point. So Nvidia has these containers that are optimized that you can say hey, these are optimized for this kind of hardware and it will maximize the utilization. And these are tuned to run on most, most hardware. So depending on your, the type of GPU you're running, you may be getting a different setting. So, so Nvidia has made sure that you don't have to be running those per model optimization. Like Llama may have a different set of optimization and then a different model may have a completely different set of optimization. How do you know which one is the best way to do run? And so you. There's a lot of research that's needed and so Nvidia Nim basically eliminates that, that research. You can take the NIM container and then run it in your environment. And the beauty is that it plugs in with a lot of other backends, so you can use it with the tool of your choice.

[Shruti Koparkar]

Awesome. Since we are talking about containers, there's one more thing that comes to mind which I've seen a lot of customers use, and that is Karpenter. And can you maybe chat a little bit about what Karpenter is and you know how it helps with sort of optimal resource utilization?

[Ray]

Okay, that's. I would have preferred a longer conversation, longer time, but let me give you a very, very quick summary of what Karpenter does. So a lot of times the people that are running applications in the cloud versus the people that are building applications in the cloud are different people. Okay. And so I may be a developer and essay. I'm writing my code, and I write my code, I build my application and say, okay, operations guy, run it. Now the operations people are like, okay, fine, I'll run it. But how much capacity should I. Should I allocate to your application? And I would say, you know what? I. It's not my job to find out what the application capacity is. So give it the maximum like 50,000 GPUs. But that's not practical, right? So that may be too expensive. So what ends up happening is you have a lot of wasted capacity that's not really fully utilized. So a lot of times we have to do as infrastructure architects. You may be able, you may have to do right sizing, you may have to right size. You may have to say for this application, this is the best site of EC2 instance. And that may be good for one application. But how do you do it? Like with 500 application level impossible. So the right way to do it is hey, let the environment decide when to scale. So what Karpenter does is in Karpenter or with EKS Auto mode, you just go and create containers. And then creating containers you'll say this is the amount of CPU and GPU or Memory I want. And Karpenter will then decide and EKS auto mode will then decide which what type of EC2 instance you should create. So I don't have to decide whether this should run on C5 instance because as an operations person, why should I decide what this runs on? And developers most of the times they don't know what type of instances should run on. They're like, give me an instance with a disk, you know. So Karpenter does optimizes that kind of compute scaling and it does two things. One is it makes sure that you're, when you scale up, you scale up reacting to the workload that you need to deploy. So if you're, if you just want two VCPUs and two gigs of RAM for like a small workload, the Carpenter will deploy an EC2 instance that just matches just that amount over a period of time. If you deploy many, many instances. And Carpenter thinks that replacing them with a larger instance will be cheaper for you, Karpenter can also do that. So you can let Karpenter decide that and say, hey, it's okay if you disrupt my workload between this time and that time just to save cost. So Carpenter is basically a Kubernetes autoscaler which now is open source by the way, and also across available across other clouds. And so it's designed to make scaling really seamless on on EKs and Kubernetes in general.

[Shruti Koparkar]

Awesome. This was really fun, Ray. Thank you so much. I know that personally I would like much more time with you to dive deep and understand some of these technologies, but you really provided a great overview of what folks should be thinking about when they are running GPU workloads or excited workloads on EKS and the various different frameworks and tools like Ray and Karpenter and how they can help them. So thanks a lot for joining us.

[Ray]

Yeah, thanks for having me.

[Shruti Koparkar]

Okay, so for the next segment in this episode we are going to look at accelerated computing use cases across the financial services sector. And joining us to talk about this is Sudhir. Sudhir, can you please introduce yourself?

[Sudhir Kaldendi]

Sure, absolutely. Thank you for having me, Sudhir. Hello everyone. It's great to be here. My name is Sudhir Kaldendi. I work as a principal solution architect and worldwide financial services at aws, specialized in payments.

[Shruti Koparkar]

Awesome. Welcome again. Can you maybe talk a little bit about how is accelerated computing which is our compute instances powered by the Nvidia GPUs or our purpose built AWS AI chips train them and Inferentia. How is this Portfolio of compute being used by customers in financial services. What are some of the challenges that they are trying to solve using these instances?

[Sudhir Kaldendi]

When we are trying to build real time fraud solutions, what we see is there are several key architectural considerations especially will come into play, especially when leveraging Nvidia technologies on aws. So financial institutions, right, they face the challenge of processing vast amounts of transaction data, especially in real time, which requires robust infrastructure and also efficient data processing. What we have seen is using Nvidia's Rapid suite, which is integrated with AWS services like Amazon emr, which accelerates data processing and also using machine learning pipelines which enables faster fraud detection and the use of graphical neural networks in Nvidia's AI workflow, which enhances fraud detection accuracy by analyzing complex relationships within transaction data. And we also see right customers today using Nvidia's Morpheus framework and also the Triton inference server, which are very crucial for real time transaction analysis and model deployment which ensures swift decision making. And also AWS services like Amazon Sagemaker and Amazon easy to support model training and deployment which allows for scalable and efficient fraud detection systems. So what we encourage customers is by again combining Nvidia's AI solutions with AWS cloud infrastructure to the financial service customers, they can achieve up to 14 times faster data processing and model inference alongside significant cost reduction. So this integration allows for real time fraud detection and also reduces false positives, improving customer trust in building the fraud detection solutions on aws.

[Shruti Koparkar]

I see. So a lot of customers in financial services are using these GPU X rated instances to combat fraud. And this requires as you said, a lot of real time data processing, instant decision making. And that's where the Obviously the Nvidia GPUs but also some of this other software that Nvidia has built comes into play. Could you break down maybe some of the architectural considerations when building these solutions? Like especially given that there is sort of this real time processing required. What are those architectural considerations? What does that solution architecture look like beyond using the GPUs and some of the software services you used? Because AWS provides the wide array of services that kind of go around this accelerated compute, right?

[Sudhir Kaldendi]

Yeah, so that's a very valid question, what we hear from the customers, right? When we are trying to work with financial service customers to build a production system, the several key challenges, what they said is especially the huge amount of transaction data which they handle daily. Again, this definitely requires efficient processing and systems to process it quickly. And again, it's not about today's data Right. We are also talking about years of historical information as this history is very valuable to identify the patterns and which also had presents its own set of challenges. So the one of the first challenge what we see is how do you try to store and retrieve this vast volume of data information which is very efficiently. And the second thing is how do you try to correlate the older data with emerging trends. So coming to the architecture patterns here, right. The first thing is where you need to move your data. Again if you're trying to run from your on premises we do have many services where you can move the data into aws. And the second thing is if you have your own data residing in the silos, we can also store efficiently meaning process the data effectively and then push it into again S3 data lakes. And through S3 data lakes what you can do is you can able to design where is the cross storage coming to the volumes. What we have seen is using the Trident inference server we could able to process close to 350,000. Sorry, 350,000 transactions per second which is a huge skill in identifying the fraud.

[Shruti Koparkar]

Awesome. That's really great. And that's a helpful sort of guidance on what type of architecture patterns you've seen and what helps. It sounds like data is. I mean we've all known that data is central to AI but this is where it really becomes apparent where you have not just like live new data but also a lot of historical data. And how do you process all that data at scale really becomes important.

[Sudhir Kaldendi]

That is correct. And we also have the workshop related to this. Again we encourage customers to get into this public git repository so that they can start building the architecture. What we have built on top of the.

[Shruti Koparkar]

Yeah, no, that'd be great. We can actually include a link for that on our show notes and so listeners please check out if you're interested. We will include one link. Let's dive into a sort of customer example maybe like specific customer story. We've, you know you've worked with Feature space and their feature spaces Aric Oric Risk Hub is something that they are building on AWS. They're processing over 100 billion events annually with really impressive fraud detection rates. Can you walk us through maybe how they are leveraging AWS as well as Nvidia technologies and achieving these results and maybe some of it is things that you've already mentioned but it will really be helpful to understand from a very specific use case perspective.

[Sudhir Kaldendi]

Sure, sure. Shruti. So Feature Space ARIC WISK Hub.

[Ray]

Right.

[Sudhir Kaldendi]

Which is Definitely a great example of how cutting edge technology can tackle financial product scale, especially today. Feature space, right? They are processing close to, as I said, right, over 100 billion events annually, which they have built a robust system with remarkable fraud detection rates. So the big part of this success which comes with usage of both AWS and Nvidia technology. So coming to the AWS side, right, the feature space leverages again, the scalability and reliability, which definitely need to handle vast volumes of transactions in real time. And again, especially the feature space, they use our elastic computing power and storage, which means ARRAG risk cup can seamlessly scale up to again thousands of transactions per second, which during peak times without even compromising on performance. And also it ensures uptime and low latency, which is essential for fraud detection where every real, every second matters, right? Because whenever a transaction is happening at a point of sale or at an E commerce transactions, customers do not really want it to wait at the transaction part, right? So they immediately the transaction has to make sure that it is not fraudulent and the transaction has to happen seamlessly. So that is where AWS is really helping the feature space to build the live transactions on AWS. And coming to the Jira, the Nvidia part. So they've been leveraging Nvidia GPUs and here they use Nvidia GPUs to train their deep learning models which power their adaptive behavior analytics. And again, GPUs are incredibly efficient at handling complex mathematical calculations. Right. Which is required for AI training. And again, this results in faster model development and also more accurate fraud detection. So by combining Nvidia GPUs, acceleration and also with AWS scalable cloud environment, Featurespace has created a dynamic fraud detection platform that not only reacts to fraud attempts, but also learns and evolves to predict new threats.

[Shruti Koparkar]

Awesome. I think you may have mentioned Nvidia Rapids and Rapids accelerator for Apache Spark on Amazon EMR earlier. Can you maybe talk a little bit about that particular solution and how it's helping institutions, you know, tackle some of these more challenges while keeping costs down, which is always important.

[Sudhir Kaldendi]

Yeah, sure. Especially, right. The costs which are tied to the feature engineering and model machine learning models. Right. They can really add up. Again, if you're not really trying to optimize, the cost is going to be significantly high. So that is why finding ways to cut those costs is very super important, I would say. So one of the game changers in this space is Nvidia Rapids. Again, it's a tool that speeds up data processing and machine learning pipelines which means they can handle huge amounts of data much quicker than with traditional CPU based systems. Again, this speed is absolutely critical, right, when it comes to real time fraud detection because it lets institutions react to emerging threads right away. Again, but it's not about the speed, it's also about saving money. So here the traditional CPU systems often need more resources and take more time to process data, which again drives up cost. So that is where the GPU accelerated solutions like Rapids, they can process data much more efficiently, cutting down on the need for extra hardware and also reducing overall expenses. Things get even better with Rapid Accelerator for Apache Spark on Amazon emr, which boosts the speed of model training. This is crucial because fraud tactics, tactics which are always evolving and institutions, right, they need to adapt fast. So by using these technologies together, right, especially the financial institutions, they can create fraud detection systems that are faster, more scalable and can handle new threats while keeping costs under control. In short, right, what I would say is these solutions are reshaping fraud detection, making it faster, more affordable and also more scalable. And that's exactly what financial institutions need to stay ahead of potential fraud and protect themselves from big financial losses.

[Shruti Koparkar]

Yeah, so true. I mean, it sounds like these institutions have to constantly innovate because, you know, the fraud, you know, keeps evolving and being multiple models. One last question. You know, as you mentioned, like they have to constantly evaluate and we are seeing this convergence of different AI technologies. So there's graph neural networks, I think you mentioned those earlier. There's large language models, there's large transaction models. How are all of these working together to kind of create sort of that innovation or that more sophisticated detection systems.

[Sudhir Kaldendi]

Yeah. So when we look at how AI technologies like graph neural networks, which we also call as GNNs, and also large language models, LLMs and large transaction models. Right. So they do work together again for fraud detection. What we see, it's very exciting. So, so these technologies are combining to create very advanced systems. So if you look at graph neural network, what they do is they do analyze large complex transaction methods, again through networks, which I would say to find patterns that might show fraud. So the example I can give you is again the LinkedIn. Right. Again which we are connected to each other. So that is how HGNs work behind the scenes. And they are good at catching unusual things that other models might fail. So when it comes to large language models, they are good at processing lots of unstructured data like invoices or emails to find anything suspicious. So again, this helps business or even financial institutions Right. To catch problems faster and also work more efficiently. Coming to large transaction models. They also play a big role by processing a lot of data to find patterns that indicate fraud. But the best part is when all three technologies work together, they give a complete picture of transactions, reducing false alarms and also catch fraud in real time. I would say obviously by combining graphical neural networks with traditional machine learning models. Right, which could be XGBoost, which makes them more accurate and also easier to understand. And overall, this combination of AI technologies. Right. Is really changing fraud detection. And again, one fraud model really may not really serve the purpose. So that is where we encourage customers to start building like different several models fit for the purpose and which makes systems more accurate, efficient and also better prepared for the future.

[Shruti Koparkar]

Awesome. That's a really, really great overview and you really help break down what each of these different types of models or algorithms are good for. And of course, all of them can be accelerated, or rather are accelerated using the GPU X rated EC2 instances as well as several different sort of software libraries available from Nvidia. Thank you so much, Sudhir for joining us, for talking to us about how customers are using accelerated EC2 instances in the financial services sector. Thank you.

[Sudhir Kaldendi]

Thank you for having me, Shruti. Thank you all.

[Shruti Koparkar]

Okay, so that's it for this episode, everyone. It was great chatting with Ray and Sudhir on how X rated computing is helping customers achieve great business outcomes across different industries. If you have any questions, you can find me as Shruti Koparkar on LinkedIn or X or you can send us feedback to [e-mail]. and until next time, keep on building.