ArtAnno: Annotating Implicit Semantics in Artworks through LLM Agent-Driven Bidirectional Human-AI Augmentation

Annotating artworks requires deep contextual knowledge of cultural symbols and metaphors, which is often invisible at the pixel level. Traditional annotation tools are either purely manual or offer one-way assistance, forcing experts to perform repetitive, time-consuming research. The authors propose a Bidirectional Human-AI Augmentation (BiHAA) framework, which treats the annotation process as a closed loop where AI assists the human, and the human's expertise is simultaneously distilled into the AI's knowledge base. To implement this, they developed ArtAnno, a multi-agent system that features a Proactive Agentic Support Module for real-time suggestions and an Interaction-Driven Evolution Module that captures user workflows and domain knowledge for future reuse.

ArtAnno operationalizes the BiHAA framework through two primary modules. The Proactive Agentic Support Module uses LLM agents to perform semantic mining, offering label suggestions and multimodal justifications for depicted objects. The Interaction-Driven Evolution Module monitors user annotation trajectories and corrections. When a user performs a task, the system extracts the underlying heuristics and domain-specific insights, storing them in a hierarchical Skill Library and a Knowledge Base. This ensures that as the user annotates more artworks, the system becomes increasingly capable of providing accurate, context-aware assistance, effectively reducing the cognitive load for subsequent tasks.

This research addresses the 'last mile' of computational art research—the gap between identifying physical objects and understanding their cultural significance. By moving away from static, one-way AI tools toward a dynamic, self-evolving system, ArtAnno demonstrates how human-AI collaboration can be more than just a productivity boost; it can serve as a mechanism for knowledge accumulation. This approach is particularly valuable for humanities research, where domain expertise is scarce and the cost of manual information verification is high.