Software Engineering Daily: Uber’s On-Call Copilot with Parth Chothani and Eduards Sidorovics

Podcast: Software Engineering Daily
Host: Sean Falconer
Guests: Parth Chothani (Staff Software Engineer, Uber AI Genai Team) and Eduards Sidorovics (Senior Software Engineer, Uber AI Platform Team)
Release Date: April 8, 2025

Introduction

In this episode, Sean Falconer interviews Parth Chothani and Eduards Sidorovics about GENIE, Uber's AI-powered on-call copilot designed to streamline on-call operations and enhance the efficiency of engineering teams across the company.

The Genesis of GENIE

Motivation and Challenges
Parth explains that Uber's extensive platform teams and the reliance on tools like Slack for support created significant inefficiencies. Engineers frequently faced delays in obtaining help, leading to frustration and reduced productivity.

“There was a lot of pain that we as engineers faced when asking for help from other support teams or other platform teams.”
— Parth Chothani [03:42]

Solution: An Automated Copilot
To address these challenges, Uber developed GENIE to provide real-time responses to queries by leveraging internal knowledge sources, thereby improving incident resolution and team collaboration.

“We wanted to have an automated solution which can look at all the internal knowledge sources and be able to answer questions that engineers across the company can take help from and really improve their efficiency.”
— Parth Chothani [03:42]

Architecture & User Interaction

User Onboarding and Interaction
Eduard describes the user experience where engineers onboard GENIE by specifying their internal wikis or helpdesk channels. The system then scrapes, embeds, and stores this data, enabling GENIE to respond to queries via a Slack bot.

“You just specify their sources and then boom, that's it.”
— Parth Chothani [05:33]

Backend Pipeline
The backend leverages big data technologies like Spark for parallel processing and generating embeddings using models such as OpenAI’s ADA. The data is then stored in a vector store, initially using Uber’s homegrown solution, Sia, and moving towards OpenSearch for better open-source compatibility.

“We use Spark to take a lot of internal sources, generate embeddings on the fly, and ingest data into Vector store at scale.”
— Parth Chothani [06:20]

Technical Details: Pipeline, Embedding, Vector Store

Embedding Models
GENIE employs both in-house and third-party embedding models, with a preference for OpenAI's ADA embeddings for their effectiveness.

“We preferred the ADA embedding models from OpenAI to begin with and those have worked reasonably okay...”
— Parth Chothani [12:34]

Vector Store Evolution
Initially utilizing a homegrown vector store, Uber transitioned to OpenSearch to enhance compatibility and scalability.

“We are trying to move towards other better Vector Store solutions like OpenSearch...”
— Parth Chothani [13:09]

Feedback and Evaluation

Continuous Feedback Loop
Eduard highlights the importance of user feedback, where responses from GENIE can be rated using emojis, helping the team gauge accuracy and areas for improvement.

“When GENIE replies, there's like a pop up of you can reply with the emoji saying okay, is it good...”
— Eduard Sidorovics [16:50]

Evaluating Documentation Gaps
When GENIE provides inaccurate answers, it indicates potential gaps in documentation, prompting updates and improvements.

“If the answer is not good, it means that either RAC components were not good or actually the documentation was not there.”
— Eduard Sidorovics [17:14]

Security Considerations

Protecting Sensitive Information
Security was a top priority. Uber implemented Genai Gateway to filter out Personally Identifiable Information (PII) before any data interacts with the model, ensuring data privacy and compliance.

“We really wanted, as Edwards mentioned, like the redaction PII data should be redacted before it gets sent out.”
— Parth Chothani [26:28]

Challenges Faced

Addressing Hallucinations
One of the initial challenges was ensuring GENIE provided accurate information, as early versions sometimes generated incorrect responses.

“Hallucination was a start where we were like, you know, just spitting out things that were pretty much wrong sometimes.”
— Parth Chothani [27:01]

User Experience Design
Creating a frictionless and intuitive UI without cumbersome approval processes was another significant hurdle.

Evaluation Methodologies
Developing in-house methods to measure productivity gains and answer accuracy required innovative thinking, as there were no industry standards available.

Managing Non-Determinism
The stochastic nature of LLMs introduced non-determinism, necessitating a shift in engineering mindset to handle unpredictable outputs.

“The non determinism definitely is one of the things that makes this whole product building so challenging.”
— Parth Chothani [34:23]

Impact and Metrics

Productivity Gains
GENIE has been deployed across over 150 channels, answering more than 70,000 questions with a 48% helpfulness rate. Parth estimates that it has saved Uber approximately 13,000 engineering hours since its inception.

“We estimated roughly when we did math around like 13k engineering hours so far we have saved across the company.”
— Parth Chothani [37:23]

Future Directions

Enhancing Accuracy and Features
The team is focused on developing GENIE V2 to meet increasing user expectations by improving accuracy and integrating advanced features like user intent detection.

“We are definitely very much thinking about, like, taking this and making it a V2 version where we can have a very high level of accuracy.”
— Parth Chothani [39:54]

Adapting to Evolving Technologies
With the rapid evolution of AI models, GENIE’s architecture is designed to be flexible, allowing for quick integration of new advancements to maintain and enhance performance.

Key Takeaways

• User-Centric Design: Simplifying onboarding and interaction is crucial for widespread adoption.
• Continuous Feedback: Implementing robust feedback loops ensures ongoing improvement and accountability.
• Security First: Protecting sensitive data through comprehensive security measures is essential.
• Adaptability: Building flexible systems allows adaptation to evolving technologies and user needs.
• Impact Measurement: Creative and rigorous methods to measure productivity gains demonstrate the tangible benefits of AI tools like GENIE.

Parth concludes with an encouraging note for developers venturing into generative AI applications:

“People have to just be open to the fact that whatever we build is… open to experimentation is a healthy mindset for GENIE.”
— Parth Chothani [40:37]

This episode provides a comprehensive overview of how Uber leveraged AI to create GENIE, addressing internal inefficiencies, enhancing productivity, and navigating the complexities of AI integration in a large-scale organization. It offers valuable insights for engineers and organizations looking to implement similar AI-driven solutions.