Thanh Vu, Sara Keretna, Richi Nayak and Thirunavukarasu Balasubramaniam, Queensland University of Technology, Australia

This study investigates the practical deployment of AI-based Text-to-SQL (T2S) models on a real-world telecommunication dataset, aiming to enhance employee productivity. Our experiment addresses the unique challenges in telecommunication datasets not explored in previous works using annotated datasets. Leveraging advanced retrieval augmented gen-erative (RAG) models like Vanna AI and Llamaindex, we benchmark their performance on synthetic datasets such as SPIDER and BIRD with different LLM backbones and sub-sequently compare the best-performing model to human performance on our proprietary dataset. We propose the Productivity Gain Index (PGI) to quantify the dual aspects of pro-ductivity improvement-time efficiency and accuracy' by comparing AI performance with human analysts across various SQL tasks. Results indicate significant productivity gains, with AI-based tools demonstrating superior query processing and accuracy performance. This prominent gap signals the potential of AI-based tool applications in the actual com-pany domain for improved productivity.

Text-to-SQL, Large Language Models, Productivity Gain Index, Retrieval-Augmented Generation, Artificial Intelligence Evaluation