When Readability and Source Retention Diverge: An Evaluability Gap in AI Translation

Machine translation is frequently used as a utility where users enter text and receive output. However, modern large language models often generate highly fluent text that can mask underlying alterations in meaning. When users submit personal text to translation interfaces, they must not only evaluate whether the output is readable and useful, but also decide whether to entrust their private information to the system. This study investigates the evaluability gap in AI translation—the phenomenon where users see the source text alongside the output, yet their holistic quality judgments fail to reflect differences in source retention.

The authors conducted a focal 2 by 2 experimental comparison (N = 306) using a simulated translation interface called TransLingo. The experiment manipulated two factors: the source-text condition (simple generated narratives versus complex literary-philosophical prose) and the output rendering (LLM-generated readability-oriented outputs versus researcher-revised fidelity-oriented outputs). Following exposure to the stimuli, participants completed measures assessing perceived translation quality, perceived system intelligence, agency-oriented anthropomorphism, task-performance trust, and stated willingness to disclose personal information across six domains.

A descriptive stimulus audit confirmed that fidelity-oriented outputs retained more of the original source text across both text conditions. However, factorial analyses revealed a significant interaction between rendering and source-text condition for perceived quality. Participants rated fidelity-oriented outputs significantly higher than readability-oriented outputs for simple narratives, but no reliable rendering differences emerged for complex prose. This same source-dependent pattern extended to perceived intelligence, agency-oriented anthropomorphism, and task-performance trust. A structural equation model demonstrated that task-performance trust served as a proximal correlate of stated disclosure willingness across all six tested information domains, while output rendering had minimal direct impact on disclosure willingness.