Wontopos Tablet 2: Measuring Multilingual and Multimodal Memory Retrieval Without Lexical Matching

This paper evaluates the performance of Tablet 2, a production long-term memory engine, across two primary axes: standard text-based retrieval benchmarks and the cross-lingual retrieval of captionless photographs. The author investigates whether a system can achieve high retrieval accuracy without relying on traditional lexical matching (like BM25) or keyword-based scoring, which often fail in multilingual or multimodal contexts.

The study measures Tablet 2 on two established text benchmarks (LongMemEval-S and BEAM-1M) and a custom multimodal setup using the Crossmodal-3600 dataset. A key feature of the methodology is the use of a lexical control (BM25) and open-source dense baselines to compare performance. The author emphasizes that the retrieval path is intentionally designed to be language-agnostic, avoiding any language-specific token overlap or keyword matching. The paper also provides detailed ablations on the reader model, re-ask budget, and context expansion to demonstrate how these variables significantly influence final scores.

Tablet 2 achieves strong performance on text benchmarks (95.7% on LongMemEval-S and 67.5% on BEAM-1M). In the multimodal domain, the engine successfully retrieves captionless photographs across 14 languages, significantly outperforming BM25, which structurally fails when no text is present. The author finds that while dense retrieval is more language-independent than lexical methods, it is not inherently flat; performance still varies based on the training data of the underlying text encoders. The paper also reports negative results, specifically that retrieval quality degrades sharply for low-resource languages and that adding English captions to images can paradoxically lower retrieval performance for non-English queries.

This paper highlights the importance of transparency in retrieval evaluation. By demonstrating that common settings like re-ask budgets and reader models can shift scores by nearly 9 points, the author argues that current benchmark leaderboards should be treated as loose placements rather than strict rankings. Furthermore, it provides a rigorous framework for testing multilingual retrieval systems by using captionless images, effectively forcing systems to rely on semantic understanding rather than lexical shortcuts.