#706: Automate LLM fine-tuning and selection with Amazon SageMaker Pipelines

Summary

AWS Podcast Episode #706: Automate LLM Fine-Tuning and Selection with Amazon SageMaker Pipelines

Release Date: February 3, 2025

---

Introduction

In Episode #706 of the AWS Podcast, hosted by Shruti Koparkar, Amazon Web Services delves into the evolving landscape of Large Language Model Operations (LLMOps) with a focus on Amazon SageMaker Pipelines. Joining Shruti are Piyush Kadam, Senior Product Manager for Amazon SageMaker, and Lauren Mullinax, Senior AI/ML Specialist Solutions Architect. This episode is tailored for data scientists, ML engineers, MLOps professionals, and anyone interested in the operational aspects of large language models (LLMs).

---

Understanding LLMOps

Shruti opens the discussion by contrasting traditional MLOps with the emerging field of LLMOps. Piyush Kadam provides a foundational understanding:

“LLMOps basically encompasses the frameworks and best practices that let you take these off-the-shelf foundation models and validate them, test them out, and then deploy them at scale so that you are generating responsible and cost-effective responses for your applications.”
— Piyush Kadam [04:59]

He traces the evolution from DevOps to MLOps, highlighting how foundation models have transformed the initial stages of model development. Unlike traditional ML models that require training from scratch with proprietary data, foundation models can be sourced from third-party providers or open-source repositories. This shift necessitates new practices in customization, evaluation, and deployment, which LLMOps seeks to address.

---

Operationalizing LLMs: Challenges and Solutions

Shruti emphasizes that, while the core principles of MLOps—such as experimentation, repeatability, reliability, and scalability—remain relevant, LLMOps introduces unique considerations:

“Do you need to version your prompts and things like that? So that's helpful to level set that fundamentally it's sort of the idea is the same of, you know, trying to manage this complex life cycle, but now focused on these large language models or foundation models.”
— Shruti Koparkar [06:06]

Piyush elaborates on two primary differentiators for LLMs:

1. Customization Process: Unlike ML models where data characteristics and training processes are transparent, foundation models operate as "black boxes." Customization begins with prompt engineering and may extend to fine-tuning with proprietary datasets to meet specific application needs. This requires rigorous evaluation to ensure models adhere to responsible AI guidelines, especially in sensitive industries like finance and healthcare.

2. Deployment Parity: While deployment processes for LLMs share similarities with traditional ML models, the emphasis on responsible AI and model evaluation introduces additional layers of complexity.

---

SageMaker Pipelines for LLMOps

Lauren Mullinax introduces Amazon SageMaker Pipelines, highlighting its role as a scalable, serverless solution designed to manage end-to-end ML and LLM workflows. Key features include:

• Visual Pipelines Designer: An intuitive drag-and-drop interface that simplifies pipeline creation without extensive coding.
• Python SDK: For advanced users who prefer scripting and customization.
• Scalability: Capable of handling tens of thousands of concurrent ML workflows, ensuring seamless integration with other AWS services.

“So pipelines one benefit is that it can scale to run tens of thousands of concurrent ML workflows in your production environment.”
— Lauren Mullinax [10:24]

Lauren underscores the importance of SageMaker Pipelines in automating repetitive tasks, facilitating experiment management, and maintaining scalability—all crucial for efficient LLMOps.

---

Specific Capabilities of SageMaker Pipelines for LLMOps

The discussion delves deeper into how SageMaker Pipelines caters specifically to LLMOps needs:

• Fine-Tuning Steps: Purpose-built steps for fine-tuning foundation models, including selection of managed instances and data processing.
• Distributed Training: Supports launching distributed training jobs across multiple GPUs, essential for handling models with billions of parameters.
• Experiment Management and Versioning: Integrated with SageMaker Clarify and the open-sourced FMEVAL library for advanced model evaluation metrics. Facilitates collaboration among teams by tracking different fine-tuning experiments and enabling model rollbacks.

“With fine-tuning, which I think is a very unique and important part of LLM ops, we have not seen that with MLOps and traditional ML models in the past.”
— Lauren Mullinax [13:37]

Lauren also highlights how SageMaker Pipelines integrates with services like Data Wrangler and GitHub, enhancing data processing and experiment tracking capabilities.

---

Cost Optimization in LLMOps Using SageMaker Pipelines

Cost management is a critical concern in LLMOps due to the high expenses associated with GPU usage. Piyush introduces two key features within SageMaker Pipelines that aid in cost optimization:

1. Step Caching: Automatically detects unchanged pipeline steps and reuses previous results, preventing redundant operations.
2. Selective Execution: Allows users to explicitly choose which pipeline steps to rerun, offering granular control over resource usage.

“We have a customer who has even 100 steps... If I start off with a pipeline completely created from this visual Designer... I just want to test out how my pipeline would behave if I update step number five.”
— Piyush Kadam [22:02]

He shares a real-world example where a customer saves costs by reusing results from unchanged steps, demonstrating the practical benefits of these features.

---

Version Control and Experiment Tracking

Effective version control and experiment tracking are paramount for managing complex LLM workflows. Lauren explains how SageMaker Pipelines facilitates this through:

• Model Registry: Tracks model lineage and supports approval workflows for moving models from development to production.
• Integration with MLflow and GitHub: Enhances visualization, tracking, and monitoring of experiment runs.
• Hyperparameter Logging: Records hyperparameters and metrics for each training run, enabling detailed comparisons and informed decision-making.

“You can have a model registry, having an approval workflow within that actual model registry, so that can help support your model approval process for different stages.”
— Lauren Mullinax [27:04]

These capabilities ensure that teams can efficiently manage multiple experiments, collaborate effectively, and maintain robust version control over their models.

---

Conclusion

The episode concludes with a recap of the transformative impact of SageMaker Pipelines on LLMOps. Shruti emphasizes the scalability and operational efficiency gained through automated pipelines, enabling organizations to manage large-scale LLM projects with ease.

“Until next time, keep on building.”
— Shruti Koparkar [30:08]

Listeners are encouraged to leverage SageMaker Pipelines to streamline their LLMOps workflows, optimize costs, and maintain high standards of model performance and compliance.

---

Key Takeaways:

• LLMOps builds upon MLOps, addressing the unique challenges of deploying and managing large language models.
• Amazon SageMaker Pipelines offers scalable, repeatable solutions with features tailored for LLMOps, including fine-tuning, distributed training, and comprehensive experiment management.
• Cost Optimization is achievable through step caching and selective execution, reducing unnecessary resource expenditure.
• Version Control and Experiment Tracking are seamlessly integrated, facilitating collaboration and ensuring model reliability and compliance.

For developers and IT professionals aiming to harness the power of large language models, this episode provides valuable insights into leveraging AWS tools to automate and optimize LLM workflows effectively.

Transcript

[Piyush Kadam]

This is episode 706 of the AWS.

[Shruti Koparkar]

Podcast, released on February 3rd, 2025. Hello, everyone. Welcome to the official AWS podcast. This is Shruti Koparkar and I will be your host for today's episode where we will discuss LLMOPS with Amazon SageMaker. So LLMs, or large language models are powering innovative applications across industries from, from, say, creating really creative content to providing insightful analysis. But building these powerful models and more importantly, deploying them and managing them in production is a complex challenge. And llmops is the solution. So this podcast is for data scientists, ML engineers, MLOps engineers, or anyone really who is interested in operationalizing large language models. So tune in as we explore the best practices, tools and strategies for successful LLM ops. Now joining me today are Piyush and Lauren. Welcome both, and why don't you introduce yourselves? So, piyush, maybe you go first.

[Piyush Kadam]

Hey folks, I'm Piyush Kadam. I'm a senior Product manager with Amazon SageMaker. I'm a longtime Amazonian and excited to talk more about this topic.

[Lauren Mullinax]

Hey everyone, I'm Lauren Mullinax. I'm a senior AI ML specialist solutions architect. Happy to be here, Shruti, and excited to talk more about llmops.

[Shruti Koparkar]

Excellent. Okay, so before we dive into sort of the details of LLMOPS and all the cool functionality we have in Amazon SageMaker, maybe let's just kick off with the basics. What is LLMOPS like? We are familiar with the MLOps terminal. What is LLMOPS and why is it so important as this AI landscape evolves?

[Piyush Kadam]

Yeah, for sure. I think the rapid innovation in AI is kind of generating a lot of acronyms and terminology. So it would be good to demystify what LLM ops is. It really started about probably 20, 30 years back with DevOps, where you needed a set of tools and best practices to manage your software. You have written a cool piece of software, but how do you go and distribute it to thousands or millions of your end users? That's when you got via DevOps, these innovations like CI CD pipelines, code repositories, automated rollbacks, A B, testing frameworks, et cetera. And then about 10, 15 years ago, when AI kind of stepped outside of academia into the industry, you needed to kind of go and morph that DevOps framework to fit for ML models. That's when you got the world of MLOps. Right. So it's still a software artifact, but slightly different. You needed to kind of have new features around model training, monitoring these models, deploying Them, they're slightly different in terms of how you would handle those compared to a traditional piece of software. You also got things like experimentation, experimentation, specifically during model creation time. That's where things like mlflow came out. And now of course, we are in the era of foundation models and large language models. In the last couple of years now these MLOps tools have had to evolve to kind of work with the new paradigms that foundation models have introduced. Back in the time of traditional ML, you had to train your models from scratch with your own data and then make sure those models are performing well and then go deploy them. But now foundation models have kind of cut short that initial journey. You can just take foundation models off the shelf from third party providers or from open source repositories, but you still need to make sure that they work well for your application, they pass your evaluation criteria. And so the MLOps tools are now morphed into LLM Ops tools. And there are of course new paradigms too that need to be accounted for. For example, prompt engineering, this completely new in the world of LLMs. And now these MLOps tools now have to be able to experiment with different prompts, which was not a thing in MLOps. So LLOps basically encompasses the frameworks and best practices that let you take these off the shelf foundation models and validate them, test them out and then deploy them at scale so that you are generating responsible and cost effective responses for your applications.

[Shruti Koparkar]

Right, right. So still sort of fundamentally trying to do the same thing as say mlops in terms of, you know, like the. There used to be this mlops maturity model where you would establish your initial experimentation environment, but then over time, like make it repeatable, make it reliable, make it scalable, and all of those fundamentals still apply to LLM Ops. But the type of things you might need to care about, like versioning, for example, that you said it might look a little bit different because now you also need to think about prompts and you know, do you need to version your prompts and things like that? So that's helpful to level set that Fundamentally it's sort of the idea is the same of, you know, trying to manage this complex life cycle, but now focused on these large language models or foundation models. You touched upon this a little bit piyush in terms of sort of what are the key consideration when architecting these LLM ops workflows versus MLOps workflows? You touched upon some of the differences, but can we maybe double click on that and call out a few others.

[Piyush Kadam]

At a high level two ways in which LLMs differ from ML models and specifically around when it comes to operationalizing them. The first is the whole process of customizing. Earlier you could just take your own raw data set, apply some algorithm and create your net new model. You knew exactly what data that model was trained on. You knew what the data characteristics look like and what trade offs you took when you generated that model. Now with foundation models you're kind of operating with a black box. You know these model providers have done their best to give you a model that does it all, but at the end of the day you're going to have to apply it for a specific application. So that's where the customization journey, you don't have the original data sets. So you have to maybe first start with just tweaking the model with specific prompts and see how your application behaves. If that's not good enough, you have the option of going a bit further and fine tuning your models. Fine tuning is meeting kind of training to some extent, but not a full blown training run for your models. So you do take some private data sets and tweak certain layers of the models to fit your application well and then you again evaluate them. So evaluation ends up becoming a big piece of this because for every minor tweak in prompts or fine tuning data set that you do, you need to make sure that the model is at that black box is basically outputting the right things. For example, you want to check whether it's outputting biased content, is it generating any toxic responses. Especially if you start using these models in financial applications and healthcare, you really have to take care of these responsible AI guidelines. In fact, a lot of the countries now are coming up with RAI guidelines and regulations. So that's where for foundation models you have to go a certain extra mile to make sure that you are choosing the right off the shelf model to start with and then you're evaluating it to the fullest extent to make sure you comply by the regulations in the industry. And then I think the second part of it is deployment doesn't really change as much.

[Shruti Koparkar]

So you talked about model evaluation. I think I Remember learning about SageMaker clarify. Is that one of our sort of features, features that are or capabilities that we've built for this model evaluation and does that integrate with sort of the rest of our LLM OPS capabilities?

[Piyush Kadam]

Yeah, that's a good point. So we launched SageMaker Clarify, I believe two or three years ago. And again it started with evaluation for traditional ML models for tabular data. And we had things like explainability, bias detection, et cetera as part of that feature. And then last year at re invent in November 2023, we actually open sourced evaluation library called FMEVAL. It's available to download on our GitHub repo and it basically augmented clarify with bunch of new metrics and evals that were coming out of research. So for example, Stanford had published as part of the helm organization quite a good amount of research and we basically took those eval metrics and put them in that FMEVAL library. And that's resonating well with the customers because we just open source it. It's out there for the community to go morph it and change it as the AI landscape evolves.

[Shruti Koparkar]

Excellent, excellent. Okay, so now that we've kind of talked about the differences between LLMOPS and MLOPS, let's maybe talk about the capability within SageMaker which I think is called Amazon SageMake pipelines. Talk about what it is, what are some of the main benefits when it comes to SageMaker pipelines? And so maybe Lauren, you can, you know, I know that you work a lot with customers and you've seen them use this. Can you talk a little bit about sort of what it is and how our customers benefiting from it?

[Lauren Mullinax]

Sure. So yeah, I work with a lot of customers who've been using Amazon SageMaker for quite some time which is really, you know, purpose built for everything. You need end to end to build, monitor, execute and evaluate your ML workflows. And then SageMaker Pipelines, we really wanted to have a repeatable solution that was scalable, so it's serverless and it's really purpose built for your ML Ops and llmops workflows. So we've just created a new capability where there's an intuitive drag and drop UI that is called the new Visual Pipelines Designer. And so you could utilize that, which makes it a bit more streamlined to be able to have these integrated steps for LLM Ops or you could use the Python SDK. So pipelines one benefit is that it can scale to run tens of thousands of concurrent ML workflows in your production environment. So as if you're all familiar with, you know, ML Ops or DevOps and LLMOPS, it's not a one time process for actually, you know, building these workflows. You need to be able to iterate as you know, Peyush just talked about with evaluation, as you're getting, you know, more and more data, especially with LLM, you usually have lots of, you know, unstructured data and you need to be able to process that. And once you're training your model or either fine tuning your model, then you're going to have to evaluate your model and determine if there's any drift, what the evaluation metrics are, if that model is really meeting your needs. So that's where pipelines can come in and really help you to automate this whole process. As I mentioned, it's integrated with the rest of SageMaker and other AWS services, so you're able to automate a lot of these repetitive ML workflow tasks as you process data and trainer model. You know what I think is really important is that there is what we call SageMaker experiments that's built in. So that really helps you with managing your different trial runs, evaluating metrics right there and be able to determine, you know, what is the best run that you have as far as when you're training your model and determining what you know is the best model that you should be the best model that you should be using. So I'd say scalability and ability to reduce these workflows is definitely some of the main capabilities of pipelines.

[Shruti Koparkar]

Awesome, awesome. That's good to hear what some of the main benefits and capabilities are. Can you maybe next walk us through how SageMaker Pipelines helps specifically with LLMOps? And again you touched upon some of this like for example experiment management and versioning. But does it also have sort of capabilities for distributed training and deploying this model, parallelism techniques for distributed training and then experimentation thereof on the training side or data ingestion and processing? Can you just walk us through what those are specific to LLMOps?

[Lauren Mullinax]

Yeah, definitely. With SageMaker pipelines I think what's really unique is that we have these specific purpose built steps. So especially for fine tuning, which I think is a very unique and important part of LLM ops, we have not seen that with MLOPs and traditional ML models in the past. So with that you're able to choose the purpose built managed instance you would like to use for training and then also determine what data you would like to reuse or process. We've had integration with a service called Data Wrangler for a while as well as other services for large scale data processing. Here at aws you could also do feature engineering as well. I'd say as far as distributed training, you're able to launch distributed training jobs across multiple GPUs, which is extremely important for fine tuning these large language models efficiently. Since the large language models, they consist of billions of parameters so it's really important to be able to quantize or compress the model and be able to optimize memory that you use, you know, either during during fine tuning or training an LLM. So we offer a lot of different, you know, purpose built instances and high powered GPU instances there. With touching on experiment management again, I think it's really important to be able to track each run of the pipeline. So that's tracked automatically with pipelines and allows you to compare your different fine tuning experiments, which is what I see a lot with different customers. They're not just running one fine tuning job, they have multiple teams collaborating and working together. So with the ability to collaborate and having this built in versioning, you could take a look at previous experiments, be able to use features across different models and be able to roll back to earlier models as well. So that reproducibility aspect and scalability without any additional cost too we should also mention is really important for your optimization of llmops. Yeah, lastly I'd say like model versioning is extremely important too. So with that experiment tracking, once you've determined what model actually meets your needs based on using SageMaker clarify FMEMAL as Peyush mentioned for use cases, you can then have that model actually register to model registry. So that allows you to do lineage tracking and then approve the model for actual deployment. So that can really help with determining what works best to production and really helping to automate the deployment process.

[Shruti Koparkar]

Awesome. We talked about model evaluation with respect to Clarify, I also was reminded of there's obviously cases where having a human in the loop is important and I know that we have SageMaker ground truth for situations like that, so I would imagine that this integrates with that very well as well.

[Lauren Mullinax]

Yeah, definitely. If you would like to have your own automated workforce, that's definitely an option. Human the loop is definitely extremely important with, you know, evaluation just because this landscape is definitely ever changing. I think it's still very nascent. So having the ability to have a human still verify either you know, your prompts or determine, you know what, take a look at those actual metrics using by evaluating different LLMs or foundation models is really important to determine the ability to move on and you know, deploy your LLM in production.

[Shruti Koparkar]

Great. I remember one of you now I forget which one of you, but you did mention that recently we launched some visual designer capabilities. So not necessarily just the SDK but for folks who prefer a visual environment and more of a drag drop type of an environment, we've launched some capabilities There. Piyush, can you talk to us about what are some of the new components there and what are we trying to enable there?

[Piyush Kadam]

Yeah, for sure. So for the last couple of years, Pipelines has had a really strong and advanced SDK experience where data scientists and mles can create these end to end ML workflows. Using code Python was the most popular SDK that we've had out there. But increasingly we started to see, especially with foundation models, is that customers, the new set of users that were coming in, even application developers sometimes they wanted to create these ML workflows and for them it was kind of a steep learning curve to start, adopt the SDK, write code and then iterate on it. So back in August 2024 we launched a new kind of pipelines Visual Designer. The Pipelines Visual Designer basically canvas within SageMaker Studio that lets you craft these end to end ML pipelines for that entire machine learning journey. The great part is that you don't have to know a lot about the internals of that pipeline's stack or even touch the code. You have a bunch of steps, step types that we call that you can select and literally just go and drag and drop them into the canvas and connect them as like a directed acyclic graph. Dags are quite popular when it comes to workflow automation. So this pipeline, as Lauren already mentioned, that's a complete journey that starts somewhere in the data world and ends with your model deployed on an inference endpoint. So the set of steps that we offer in that Visual Designer, start with the data prep. So there's a data processing step type. We have a step type that lets you connect to an EMR instance and pull that data. Then as you progress in that journey, after Data Prep is done, we have model customization step. So there's a model training step for building models from scratch. There's a step to fine tune foundation models and that step is actually well integrated with our model hub, which is Jumpstart. So you can literally just take that fine tuning step, drop it in there and from a dropdown pick any of the hundreds of foundation models that SageMaker offers today. In the next phase comes experimentation and evaluation. So we have evaluation step as well in there. Although it's a kind of a low code, no code experience for users, we realize that there are advanced builders who want a mix of both worlds. So let's say I start off with a pipeline completely created from this visual Designer. But then I think, hey, this experience kind of lacks some customization. I need to put in My own Python script, or maybe I have a Python notebook that's lying around. I just need to plug that in into this one part of that pipeline. So we actually let you all do that. So in the Visual Designer, we have a custom code step type where you can pull in any Python script, any notebook, and create a chain of scripts within that Visual Designer itself. Then finally, of course, you have a deployment step that lets you deploy your customized model to a SageMaker inference endpoint. So it gives you the breadth of capabilities starting from data world to inference. But at the same time it lets you go little deeper into the advanced features by allowing you to link your Python scripts and notebooks together as well.

[Shruti Koparkar]

Yeah, that's awesome. I love that flexibility. So many features start off as, oh, we build this for folks who are not as familiar. And so it's drag and drop. But then advanced users, like, well, for like 75% of my tasks I can just use this because it's pretty good. But then for the 25%, I still want to be able to customize and write my own code. And we have that. Exactly. So that's awesome. You know, one of the things that, as you were talking about some of these customizations and this drag and drop flow, I was also kind of imagining these levers in my mind. And one of the primary things that customers are trying to optimize beyond performance and beyond sort of the accuracy of the models, and all of that is cost. Because we're operating at such a large scale, the costs are really high. And of course there's many ways to optimize costs, of course, like the instances that you use or the specific compute that you use. But from a LLM Ops standpoint, what are some of the levers within SageMaker pipelines that customers can use to optimize costs for these large scale projects?

[Piyush Kadam]

Yeah, that's a great question. I think this question is really important given how expensive GPUs are getting and how hard it is to get access to them. The whole point of pipelines is that once you go through the motions of building your model, deploying it, the next person that comes along in your team, or even you, you know, few weeks down the line, you don't have to go through that journey again. You just build that pipeline once, codified once, and then on demand with a single click, run it again. Or if you want to hook it up to your other systems, other workflows, you can automate that execution. But I'll share a recent example from an actual customer. So this Particular customer was trying to even understand which foundation model do I go and deploy. So it's at the exploration time how they use pipelines was they had one step that actually fine tuned a llama 3.2 model and there was a parallel, there was another step that fine tuned CLAUDE model and those kind of steps merged together into a evaluation step or a comparison step where they would compare. Okay, how did my model fare? Did model A perform better than model B? Or do I need to tweak the parameters again? And then the next step after that comparison was actually registering a model into the model registry even before deploying it. There are two features that I want to really talk about here in pipelines that help you save costs. So then once that model comparison pipeline is built, the really cool thing that you can do is the next time you want to run that pipeline, you can reuse some of the results from your previous execution. So let's say I have fine tuned today, I fine tuned the Llama model and the CLAUDE model and I run the pipeline today and tomorrow I just want to fine tune my Llama model. I want to let the CLAUDE model be. When I re execute that pipeline with the updated data set for Llama model, it's actually going to reuse the results for the CLAUDE model as is. It's not going to rerun that nothing has changed. So it automatically detects what artifacts have actually changed and only then goes and executes that particular step. This is what we call step caching, where if the parameters or config hasn't changed for a particular step, we just reuse the previous successful results for that step. That saves a ton of time. In this case, you have saved money on fine tuning that model again. The second cool feature is something we call selective execution. So let's assume we have a pipeline with 10 different steps that you can think of quite complex dags that our customers come up with. In fact, we have a customer who has even 100 steps, 102 or something like that. Now let's say the first part of steps is all about training, fine tuning and evaluation. And somewhere in the middle you are running some custom scripts or you're just triggering things off to other pipelines and then comes deployment. If I'm iterating on my evaluation scripts, I don't want to and I want to plug that updated version of the script inside the pipeline. I don't want to rerun my training jobs again in that pipeline. So I want to be able to say, hey, There are these 10 steps in my pipeline, I'm just updating the code in let's say step four to six. The rest doesn't change. So I just want to test out how my pipeline would behave if I update step number five. With selective execution, you can just go and say just execute by steps 4, 5, 6 and the rest remains the same. This is slightly different from step caching. Step caching automatically detects what artifacts have changed and reuses the results. Here you're explicitly saying I'm okay with whatever results are there in the other steps. These are the only steps that I want to rerun because maybe I don't want to spend money on the GPUs again rerunning that training job. So yeah, in documentation if you go look up, there'll be two things that will stand out. One is step caching and the other is selective execution.

[Shruti Koparkar]

Yeah, I mean as you mentioned, they are sort of related. One being slightly more automated and one gives you even more control. Right? Like maybe there are steps that would normally be re executed because things have changed but you don't care about them. Like you, you know, for whatever reasons you're okay with the older results. That flexibility is really good to have. You know this complexity that you just talked about in terms of building a pipeline with 10 or even hundreds of steps just made me think of sort of how do you track all this, right? Like the version control, tracking all the experiments that you're doing and then figuring out sort of like which one of those you actually want to use. And Laurent, can you maybe talk to us a little bit, double click on the version control and experiment tracking capabilities in SageMaker that support these LLM projects.

[Lauren Mullinax]

As you mentioned, we're seeing a lot of customers build these really hyper personalized applications utilizing LLMs. And so it's great that there are, you know, a lot of options and that these LLMs, you know, are pre trained on large amounts of data but you know, it's impossible for them to be trained on every type of data set out there that's like specific for every industry. And you also, you know, don't want them to be trained on all the data possibly out there as well for, for obvious reasons. So when I work with customers, they've told me, hey, I like that there's this LLM, but it's just not giving me the results I would like for certain use cases I worked with like for. In life sciences for medical terminology or it doesn't understand the customer's data. So whether entering a prompt, it's not getting the results they'd like to use with that for customers to be able to compare different fine tuning experiments as they're, you know, training the model and providing their own data. That's where, you know, the built in versioning as we've been talking about, where your data scientists and NML engineers can help to revisit, you know, previous experiments and then be able to roll back to earlier models utilizing like a model registry or you know, version control. We also have, you know, integration with services like, you know, GitHub. So that way you're able to test, you know, different, you know, multiple hyper parameters that could be best for these, these different use cases. Also you can use pipelines in conjunction with a lot of other services. We did announce an integration within SageMaker for MLflow which really helps you to visualize and track those experiment runs and also to monitor them too. So you can really determine in real time what your LLM you're using or you know, multiple LLMs in, in most cases is actually providing the, you know, results and accuracy that you're looking for for tracking these LLM projects. So this could be really effective then. So having a model registry, having an approval workflow within that actual model registry, so that can help support your model approval process for different stages. So if you're just doing some testing in a dev environment and moving to production, also all your hyperparameters are logged in experiments. So that's important for each training run. Being able to really record and compare those, those metrics across like different fine tuning jobs. Yeah, Paige, you know, mentioned a lot as far as like step caching and being able to do pipeline versioning too. So that, that really helps. So you have everything end to end where you could really see the, you know, different steps for, for data preparation and training and deployment to determine what's really, you know, working best or where, you know, need to do some more testing or you know, being able to edit your code. So this really helps for really expediting these, you know, large scale LLM workflows. Especially, you know, as the space is moving very quickly.

[Shruti Koparkar]

Yeah, thank you, thank you so much to both of you. Thank you piyush and thank you Lauren for joining us and sharing all of these insights on llmops. Everything from, from sort of setting up the initial, as I said, to making it repeatable, making it reliable and then scalable across so many of the LLM pipelines. Right. Like you said, you can execute thousands of times in parallel like each of these pipelines. And that's something else to be able to operationalize all of this at that scale. So thank you once again for joining. And to everyone who tuned in, thanks for listening. And until next time, keep on building.