Field Order Should Not Matter: Permutation-Invariant Embedding Model Fine-Tuning for Structured Metadata Retrieval

Retrieval over structured metadata—where records are schemas with multiple fields—is typically handled by serializing fields into a single string. This serialization forces an arbitrary field order. While often treated as an implementation detail, this order becomes a structural feature for transformer-based encoders. Standard fine-tuning allows models to memorize the absolute position of information rather than its semantic label. Consequently, if a portal redesign or data ingestion pipeline changes the field order, the retriever's performance degrades sharply. In this study, standard fine-tuned encoders lost 7.4 nDCG@10 points when the index was rebuilt under a different field order.

The authors propose a simple, architecture-agnostic fix: permutation-invariant fine-tuning (PI-FT). By serializing records under freshly sampled field orders at every training step and applying random field dropout, the model is forced to bind meaning to field labels rather than absolute positions. This approach requires only a two-line change in the data loader. The method effectively projects the model onto an invariant subspace, reducing the order-change penalty from 7.4 points to 0.2 points without sacrificing performance on the original index.

To evaluate this, the authors created DevDataBench, a large-scale, LLM-generated benchmark covering nearly 10,000 statistical indicators across 15 languages. By generating queries synthetically, the authors achieved full coverage of every indicator and facet, solving the cold-start problem inherent in usage logs. A 118M-parameter CPU-deployable encoder trained with PI-FT outperformed frontier API-based embeddings (0.707 vs. 0.556 nDCG@10), with the largest gains observed in low-resource languages. The framework also incorporates guided contrastive training to handle the high density of near-duplicates in statistical catalogs.