ResearchPod Summary
As computational methods for analyzing literature become more sophisticated, researchers need better ways to evaluate how well these systems understand fictional characters. This paper investigates whether current computational pipelines—which extract features like events, quotes, and assertions—can effectively capture the multi-faceted nature of character similarity. The authors ask: how well do these automated representations align with human-defined notions of character similarity, ranging from structural roles to expert literary analysis?
The authors introduce AustenAlike, a new benchmark suite based on Jane Austen’s six major novels. They define character similarity through three distinct lenses:
The authors evaluate two popular literary analysis pipelines, BookNLP and FanfictionNLP, by building character representations from extracted features (events, quotes, modifiers, and assertions). They compare these computational vectors against the three benchmarks and also test GPT-4’s ability to rank character similarity.
The study reveals a clear hierarchy in how well computational models understand characters. Feature-based representations, particularly those using events and assertions, perform well at identifying broad social and narrative similarities. For example, models can often group heroines together or distinguish characters by gender. However, all computational methods—including GPT-4—struggle with the expert-defined benchmark. The "expert" similarities are often subtle and thematic, requiring a depth of interpretation that current feature-extraction pipelines do not reach. Even the best-performing models fail to consistently place expert-identified "most similar" characters in their top ten lists.
Alex: Welcome to another episode of ResearchPod. Today we're looking at a paper that sits at the intersection of computational literary studies and NLP — specifically, whether our best character representation pipelines actually capture what a literary critic means when they say two characters are similar. Sam, what's the core problem?
Sam: The puzzle is a familiar one in NLP: surface features versus semantic depth. Existing pipelines — BookNLP being the canonical example — can cluster characters by demographic markers like gender, class, or marital status without any trouble. What they can't do is recognize the thematic parallels a trained reader sees. Two characters might share every social attribute and play completely different narrative roles, and the model won't notice.
Alex: So the paper is essentially asking whether these systems are doing literary analysis or just parsing metadata.
Sam: That's exactly the framing. And to make that question empirical, the authors built a benchmark called AustenAlike. The key design choice is what they call a "wisdom-of-the-expert-crowd" approach — they mined 43 volumes of Persuasions, the journal of the Jane Austen Society of North America, to extract character comparisons made by domain experts. That gives them a ground truth that isn't just researcher intuition — it's aggregated scholarly consensus about which characters across Austen's novels are meaningfully analogous.
Alex: That's a non-trivial annotation strategy. How does it cash out as a benchmark?
Sam: The benchmark operationalizes similarity along three axes. Structural similarity captures narrative role — is this character a protagonist, an antagonist, a foil? Social similarity tracks the demographic and relational markers BookNLP is good at: wealth, gender, family position. And interpretive similarity is the expert-derived axis — the pairings that show up repeatedly in the scholarly literature. The idea is that a model doing genuine literary analysis should rank high on interpretive similarity even when structural and social similarity diverge.
Alex: And what representations are they actually feeding into this comparison?
Sam: They extract two types of character features from the text: events — the actions and plot beats a character is involved in — and quotes, the dialogue they speak. These get encoded through T5 and NV-Embed to produce vector representations, and then they measure how well those vectors reproduce the expert rankings across each axis.
This work highlights a significant gap between the "distant reading" capabilities of current NLP pipelines and the nuanced understanding of literary critics. By providing a structured, multi-faceted benchmark, the authors offer a roadmap for future research to move beyond basic demographic or structural classification toward more sophisticated, interpretive models of literary character.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Alex: So the ablation is essentially: events versus quotes, and which axis each feature type tracks best.
Sam: Right. And the load-bearing finding is that event-based representations consistently outperform quote-based ones when it comes to interpretive similarity. The intuition holds up: what a character does across the arc of a novel is more diagnostic of their narrative function than what they say in any given scene. Dialogue is noisy — characters lie, perform, speak to social expectation. Events are harder to fake in that sense.
Alex: Though I'd push back slightly — in Austen especially, the gap between what a character says and what they mean is often the whole point. Is there any signal in the quotes that the events miss?
Sam: That's a fair challenge, and it's actually one of the paper's open questions. The quote-based representations do capture something — they're not zero signal — but they correlate more strongly with social similarity than interpretive similarity. Which suggests they're picking up on register, class markers, speech patterns, rather than narrative function. Whether there's recoverable interpretive signal in dialogue that better encoding could extract, the paper doesn't resolve.
Alex: What's the ceiling on the best-performing models? You said medium correlations with expert rankings — how should we read that?
Sam: Medium correlation is doing real work there. Even with NV-Embed, which is the stronger of the two encoders tested, the alignment with expert pairings is moderate at best. The gap between what the model ranks as similar and what the scholars rank as similar remains substantial. And this holds even when you give the model richer feature sets. So it's not primarily an encoder capacity problem — it's a feature representation problem. The models are working with the right kind of information in principle, but the features BookNLP extracts don't carry enough semantic structure to bridge that gap.
Alex: Which points to a deeper issue about what BookNLP is actually extracting.
Sam: Exactly. BookNLP is doing coreference resolution, entity tagging, event detection — it's a pipeline optimized for information extraction, not for capturing the kind of relational and thematic structure that literary analysis depends on. AustenAlike makes that limitation measurable rather than just asserted.
Alex: Is there a generalization concern here? Austen is a specific corpus with a specific scholarly apparatus around it. Does the benchmark design travel?
Sam: That's the honest limitation to name. The Persuasions corpus is unusually rich — decades of close reading by a specialized scholarly community — and Austen's novels are short enough and canonical enough that expert consensus is achievable. Replicating this benchmark design for, say, Victorian sensation fiction or contemporary genre novels would require a comparable scholarly infrastructure that mostly doesn't exist. So AustenAlike is a proof of concept for the methodology as much as a general-purpose evaluation tool.
Alex: So the contribution is twofold: a concrete benchmark for this specific domain, and a framework for thinking about what "character similarity" actually means computationally.
Sam: That's a fair summary. The multi-axial decomposition — structural, social, interpretive — is the transferable idea. It gives you a vocabulary for diagnosing where a model fails, not just that it fails. A system that scores well on social similarity but poorly on interpretive similarity is failing in a different and more interesting way than one that fails across the board. That diagnostic granularity is what the paper is really offering.
Alex: And the implication for the field is that closing this gap probably requires moving beyond feature extraction pipelines toward representations that encode narrative structure more directly.
Sam: That's where the paper points. Whether that means better event representations, discourse-level modeling, or something closer to how large language models process narrative context — the paper doesn't adjudicate. But it gives you a benchmark to test those approaches against, which is the necessary first step.
Alex: A well-scoped contribution, then — a measurable problem statement where previously there was mostly assertion. Thanks for walking through it. Thanks for listening to ResearchPod.