LLM-based Models for Detecting Emerging Topics in Service Feedback

Public sector organizations, such as tax administrations, face the challenge of analyzing large volumes of multilingual feedback to ensure fair and equitable service delivery. Traditional manual review processes are not scalable, and static indicators often fail to capture nuanced service quality issues. This research aims to create a scalable, data-driven methodology that uses Large Language Models (LLMs) to identify emerging topics in feedback, helping organizations detect potential service disparities across diverse demographic groups.

The researchers propose a hybrid framework that integrates fine-tuned, quantized LLMs with human-in-the-loop oversight. The process involves:

• Fine-tuning: Adapting LLMs to the specific domain language of tax administration to improve contextual understanding.
• Quantization: Reducing the computational demands of the models to ensure they are practical for real-world deployment.
• Human-in-the-loop: Incorporating expert tax officers to validate and refine AI-generated insights, which helps mitigate the risk of LLM fabrication (confabulation).
• Trend Detection: Using the model to classify feedback into predefined Service Quality Elements (SQEs) and monitoring their prevalence over time to identify emerging issues.

The proposed methodology was evaluated through similarity analysis and expert surveys. The results indicate that the fine-tuned LLM approach aligns more closely with the assessments of experienced tax officers compared to baseline models. By automating the categorization of feedback, the system allows for faster, more reliable identification of service gaps. This work provides a practical path for public sector organizations to leverage AI for evidence-based decision-making, ultimately fostering greater public trust and ensuring more equitable service delivery.