AutoSpecNER: A Fine-Grained Named Entity Recognition Dataset for Vehicle Specification Extraction

As the automotive industry increasingly relies on online advertisements, extracting structured technical specifications from unstructured text has become critical. However, existing Named Entity Recognition (NER) resources are often too coarse-grained for technical automotive data. This paper addresses the need for a specialized dataset and benchmarking framework to extract 15 specific vehicle attributes—such as battery capacity, engine specifications, and trim levels—from both human-written and AI-generated advertisements.

The authors developed AutoSpecNER, a dataset containing 659 vehicle advertisements annotated with over 10,000 entities. The dataset is unique because it includes both user-generated content (which contains informal language and typos) and AI-generated content (which may contain hallucinations). The researchers established a rigorous 15-label schema and validated it through an inter-annotator agreement process, achieving a 91.5% micro-F1 score. They then benchmarked three distinct extraction methods: a hybrid rule-based system (using taxonomies and regex), fine-tuned transformer encoders (BERT, RoBERTa, ModernBERT, and DeBERTa), and several large language models (LLMs) using few-shot prompting and self-verification.

The study reveals that fine-tuned transformer encoders are the most effective approach for this domain. DeBERTa achieved the highest performance with a 90% micro-F1 score, consistently outperforming the rule-based baseline (43%) and the best-performing LLM (77.8%). The results suggest that while LLMs are powerful, smaller, specialized encoder models remain more practical and accurate for high-precision technical extraction tasks in low-resource settings.

This research provides a foundational resource for the automotive industry to automate the verification of vehicle listings. By enabling the extraction of fine-grained specifications, AutoSpecNER helps detect hallucinations in AI-generated advertisements and improves the accuracy of vehicle comparison tools, ultimately protecting consumers from misleading information.