Funing Yang, Carolyn Jane Anderson
6 min
As computational methods for analyzing literature become more sophisticated, researchers need better ways to evaluate how well these systems understand fictional characters. This paper investigates whether current computational pipelines—which extract features like events, quotes, and assertions—can effectively capture the multi-faceted nature of character similarity. The authors ask: how well do these automated representations align with human-defined notions of character similarity, ranging from structural roles to expert literary analysis?
The authors introduce AustenAlike, a new benchmark suite based on Jane Austen’s six major novels. They define character similarity through three distinct lenses:
The authors evaluate two popular literary analysis pipelines, BookNLP and FanfictionNLP, by building character representations from extracted features (events, quotes, modifiers, and assertions). They compare these computational vectors against the three benchmarks and also test GPT-4’s ability to rank character similarity.
The study reveals a clear hierarchy in how well computational models understand characters. Feature-based representations, particularly those using events and assertions, perform well at identifying broad social and narrative similarities. For example, models can often group heroines together or distinguish characters by gender. However, all computational methods—including GPT-4—struggle with the expert-defined benchmark. The "expert" similarities are often subtle and thematic, requiring a depth of interpretation that current feature-extraction pipelines do not reach. Even the best-performing models fail to consistently place expert-identified "most similar" characters in their top ten lists.
This work highlights a significant gap between the "distant reading" capabilities of current NLP pipelines and the nuanced understanding of literary critics. By providing a structured, multi-faceted benchmark, the authors offer a roadmap for future research to move beyond basic demographic or structural classification toward more sophisticated, interpretive models of literary character.
Several systems have been developed to extract information about characters to aid computational analysis of English literature. We propose character similarity grouping as a holistic evaluation task for these pipelines. We present AustenAlike, a benchmark suite of character similarities in Jane Austen's novels. Our benchmark draws on three notions of character similarity: a structurally defined notion of similarity; a socially defined notion of similarity; and an expert defined set extracted from literary criticism. We use AustenAlike to evaluate character features extracted using two pipelines, BookNLP and FanfictionNLP. We build character representations from four kinds of features and compare them to the three AustenAlike benchmarks and to GPT-4 similarity rankings. We find that though computational representations capture some broad similarities based on shared social and narrative roles, the expert pairings in our third benchmark are challenging for all systems, highlighting the subtler aspects of similarity noted by human readers.
Alex: Though I'd push back slightly — in Austen especially, the gap between what a character says and what they mean is often the whole point. Is there any signal in the quotes that the events miss?
Sam: That's a fair challenge, and it's actually one of the paper's open questions. The quote-based representations do capture something — they're not zero signal — but they correlate more strongly with social similarity than interpretive similarity. Which suggests they're picking up on register, class markers, speech patterns, rather than narrative function. Whether there's recoverable interpretive signal in dialogue that better encoding could extract, the paper doesn't resolve.
Alex: What's the ceiling on the best-performing models? You said medium correlations with expert rankings — how should we read that?
Sam: Medium correlation is doing real work there. Even with NV-Embed, which is the stronger of the two encoders tested, the alignment with expert pairings is moderate at best. The gap between what the model ranks as similar and what the scholars rank as similar remains substantial. And this holds even when you give the model richer feature sets. So it's not primarily an encoder capacity problem — it's a feature representation problem. The models are working with the right kind of information in principle, but the features BookNLP extracts don't carry enough semantic structure to bridge that gap.
Alex: Which points to a deeper issue about what BookNLP is actually extracting.
Sam: Exactly. BookNLP is doing coreference resolution, entity tagging, event detection — it's a pipeline optimized for information extraction, not for capturing the kind of relational and thematic structure that literary analysis depends on. AustenAlike makes that limitation measurable rather than just asserted.
Alex: Is there a generalization concern here? Austen is a specific corpus with a specific scholarly apparatus around it. Does the benchmark design travel?
Sam: That's the honest limitation to name. The Persuasions corpus is unusually rich — decades of close reading by a specialized scholarly community — and Austen's novels are short enough and canonical enough that expert consensus is achievable. Replicating this benchmark design for, say, Victorian sensation fiction or contemporary genre novels would require a comparable scholarly infrastructure that mostly doesn't exist. So AustenAlike is a proof of concept for the methodology as much as a general-purpose evaluation tool.
Alex: So the contribution is twofold: a concrete benchmark for this specific domain, and a framework for thinking about what "character similarity" actually means computationally.
Sam: That's a fair summary. The multi-axial decomposition — structural, social, interpretive — is the transferable idea. It gives you a vocabulary for diagnosing where a model fails, not just that it fails. A system that scores well on social similarity but poorly on interpretive similarity is failing in a different and more interesting way than one that fails across the board. That diagnostic granularity is what the paper is really offering.
Alex: And the implication for the field is that closing this gap probably requires moving beyond feature extraction pipelines toward representations that encode narrative structure more directly.
Sam: That's where the paper points. Whether that means better event representations, discourse-level modeling, or something closer to how large language models process narrative context — the paper doesn't adjudicate. But it gives you a benchmark to test those approaches against, which is the necessary first step.
Alex: A well-scoped contribution, then — a measurable problem statement where previously there was mostly assertion. Thanks for walking through it. Thanks for listening to ResearchPod.