AI-PAVE-Br: Leveraging Large Language Models for Enhanced Product Attribute Value Extraction through a Golden Set Approach

Research Context and Objective

In the rapidly expanding Brazilian e-commerce sector, managing unstructured product data is a critical challenge. Product descriptions often lack standardization, making it difficult to extract key attributes like brand, model, or technical specifications. Traditional methods, such as rule-based systems or statistical Named Entity Recognition (NER), are often brittle and struggle to generalize across the diverse, dynamic linguistic landscape of Portuguese-language product catalogs. This paper introduces AI-PAVE-Br, a specialized system designed to leverage Large Language Models (LLMs) to perform high-accuracy Product Attribute Value Extraction (PAVE).

The Golden Set Benchmark

A major contribution of this work is the creation and release of the Golden Set, a manually annotated, high-quality dataset serving as a ground truth for PAVE in Portuguese. The researchers curated this set by sampling 20 diverse product categories—ranging from electronics like televisions and notebooks to household items like stoves and wardrobes. By using a statistically rigorous sampling methodology based on Cochran’s formula, the team ensured the dataset provides a reliable, reproducible benchmark for evaluating AI performance in the Brazilian market.

Methodology: LLMs and Prompt Engineering

Rather than relying on computationally expensive fine-tuning, the authors utilize Google’s Gemini 1.5 Flash model, employing targeted prompt engineering to adapt the model to the specific requirements of e-commerce data. The prompt structure includes clear instructions, context from product titles and descriptions, and few-shot examples to guide the model. This approach allows the LLM to act as an intelligent parser, effectively handling the contextual dependencies and novel attributes that frequently defeat traditional NER models.

Findings and Implications

Experimental results demonstrate that AI-PAVE-Br achieves a mean F1-score of 74.68, a substantial improvement over the 59.79 F1-score achieved by the traditional baseline. While the LLM-based approach shows superior semantic understanding, the authors highlight that challenges remain, particularly regarding the normalization of values (e.g., different ways to express voltage or dimensions) and the ambiguity of multi-valued attributes. This research provides a scalable, effective solution for non-English e-commerce markets and offers the community a valuable resource for future NLP research in structured information extraction.