Utah's tax chatbot pilot exposed the non-deterministic problem every enterprise faces: initial LLM accuracy hit 65-70% when judged by expert panels, with another 20-25% partially correct. After months of iteration, three of four vendors delivered strong enough results for Utah to make a vendor selection and begin production deployment. Christian Napier, Director of AI for Utah's Division of Technology Services, explains why the gap between proof of concept and production is where AI budgets and timelines collapse.

His team deployed Gemini across state agencies with over 9,000 active users collectively saving nearly 12,000 hours per week. Meanwhile, agency-specific knowledge chatbots struggle with optional adoption, competing against decades of human expertise.

The bigger constraint isn't technical. Vendor quotes for the same citizen-facing solution dropped from eight figures to five during negotiations as pricing models shifted. When procurement cycles run 18 months and foundation models deprecate quarterly, traditional budgeting breaks.

Topics discussed:

Expert panel evaluation methodology for testing LLM accuracy in regulated tax advice scenarios
Low-code AI platforms reaching capability limits on complex use cases requiring pro-code solutions
Avoiding $5 million in potential annual licensing costs through Google Workspace AI integration timing
Tracking self-reported productivity gains of 12,000 hours weekly across 9,000 active users
AI Factory process requiring privacy impact assessments and security reviews before any pilots
Vendor pricing dropping from eight-figure to five-figure quotes as commercial models evolved
Forcing adoption through infrastructure replacement when legacy HR platform went read-only
Separating automation opportunities from optional tools competing with existing workflows
Digital identity requirements for future agent-to-government transactions and authorization
Regulatory relief exploration for AI applications in licensed professions like mental health