A Systematic Evaluation of Traditional Privacy Policy Analysis Tools Against LLMs

This paper investigates whether modern, off-the-shelf Large Language Models (LLMs) can effectively replace specialized, domain-specific tools for privacy policy analysis. Historically, researchers have developed custom Natural Language Processing (NLP) pipelines to handle tasks like contradiction detection, regulatory compliance, and policy summarization. With the rise of powerful LLMs, the authors seek to determine if these general-purpose models can replicate the performance of these specialized tools using only prompt engineering, thereby reducing the need for labor-intensive, domain-specific tool development.

The researchers conducted a systematic evaluation comparing two state-of-the-art LLMs (GPT-5.2 and Gemini-2.5-Pro) against six representative privacy analysis tools. The study focused on three primary functionalities—contradiction detection, regulatory compliance, and policy aggregation—and two intermediate tasks: Semantic Role Labeling (SRL) and manual privacy policy labeling. The team curated a dataset of ten privacy policies from popular mobile applications, converting them into plain text. They utilized two prompting strategies: a simple prompt (mimicking a non-expert user) and a detailed prompt (mimicking an expert user) to assess performance without any fine-tuning or specialized training.

The results indicate that LLMs are highly capable of performing privacy analysis tasks that previously required specialized software. In almost every category, the LLMs matched or outperformed the traditional tools. For example, the models identified more intra-policy contradictions and provided more comprehensive results in policy aggregation tasks. Furthermore, in manual labeling tasks, the LLMs achieved high precision and recall (e.g., 91.4% precision for third-party sharing entities), demonstrating that they can effectively handle complex, foundational NLP tasks in the privacy domain. The authors conclude that while LLMs offer significant flexibility, a hybrid approach—combining the performance of LLMs with the scalability and cost-efficiency of traditional tools—may be the most practical path forward.