Evaluating Computational Representations of Character: An Austen Character Similarity Benchmark | Funing Yang, Carolyn Jane Anderson | ResearchPod

Transcript: Evaluating Computational Representations of Character: An Austen Character Similarity Benchmark

Alex: Welcome to another episode of ResearchPod. Today we're looking at a paper that sits at the intersection of computational literary studies and NLP — specifically, whether our best character representation pipelines actually capture what a literary critic means when they say two characters are similar. Sam, what's the core problem?

Sam: The puzzle is a familiar one in NLP: surface features versus semantic depth. Existing pipelines — BookNLP being the canonical example — can cluster characters by demographic markers like gender, class, or marital status without any trouble. What they can't do is recognize the thematic parallels a trained reader sees. Two characters might share every social attribute and play completely different narrative roles, and the model won't notice.

Alex: So the paper is essentially asking whether these systems are doing literary analysis or just parsing metadata.

Sam: That's exactly the framing. And to make that question empirical, the authors built a benchmark called AustenAlike. The key design choice is what they call a "wisdom-of-the-expert-crowd" approach — they mined 43 volumes of Persuasions, the journal of the Jane Austen Society of North America, to extract character comparisons made by domain experts. That gives them a ground truth that isn't just researcher intuition — it's aggregated scholarly consensus about which characters across Austen's novels are meaningfully analogous.

Alex: That's a non-trivial annotation strategy. How does it cash out as a benchmark?

Sam: The benchmark operationalizes similarity along three axes. Structural similarity captures narrative role — is this character a protagonist, an antagonist, a foil? Social similarity tracks the demographic and relational markers BookNLP is good at: wealth, gender, family position. And interpretive similarity is the expert-derived axis — the pairings that show up repeatedly in the scholarly literature. The idea is that a model doing genuine literary analysis should rank high on interpretive similarity even when structural and social similarity diverge.

Alex: And what representations are they actually feeding into this comparison?

Sam: They extract two types of character features from the text: events — the actions and plot beats a character is involved in — and quotes, the dialogue they speak. These get encoded through T5 and NV-Embed to produce vector representations, and then they measure how well those vectors reproduce the expert rankings across each axis.

Alex: So the ablation is essentially: events versus quotes, and which axis each feature type tracks best.

Sam: Right. And the load-bearing finding is that event-based representations consistently outperform quote-based ones when it comes to interpretive similarity. The intuition holds up: what a character does across the arc of a novel is more diagnostic of their narrative function than what they say in any given scene. Dialogue is noisy — characters lie, perform, speak to social expectation. Events are harder to fake in that sense.

Alex: Though I'd push back slightly — in Austen especially, the gap between what a character says and what they mean is often the whole point. Is there any signal in the quotes that the events miss?

Sam: That's a fair challenge, and it's actually one of the paper's open questions. The quote-based representations do capture something — they're not zero signal — but they correlate more strongly with social similarity than interpretive similarity. Which suggests they're picking up on register, class markers, speech patterns, rather than narrative function. Whether there's recoverable interpretive signal in dialogue that better encoding could extract, the paper doesn't resolve.

Alex: What's the ceiling on the best-performing models? You said medium correlations with expert rankings — how should we read that?

Sam: Medium correlation is doing real work there. Even with NV-Embed, which is the stronger of the two encoders tested, the alignment with expert pairings is moderate at best. The gap between what the model ranks as similar and what the scholars rank as similar remains substantial. And this holds even when you give the model richer feature sets. So it's not primarily an encoder capacity problem — it's a feature representation problem. The models are working with the right kind of information in principle, but the features BookNLP extracts don't carry enough semantic structure to bridge that gap.

Alex: Which points to a deeper issue about what BookNLP is actually extracting.

Sam: Exactly. BookNLP is doing coreference resolution, entity tagging, event detection — it's a pipeline optimized for information extraction, not for capturing the kind of relational and thematic structure that literary analysis depends on. AustenAlike makes that limitation measurable rather than just asserted.

Alex: Is there a generalization concern here? Austen is a specific corpus with a specific scholarly apparatus around it. Does the benchmark design travel?

Sam: That's the honest limitation to name. The Persuasions corpus is unusually rich — decades of close reading by a specialized scholarly community — and Austen's novels are short enough and canonical enough that expert consensus is achievable. Replicating this benchmark design for, say, Victorian sensation fiction or contemporary genre novels would require a comparable scholarly infrastructure that mostly doesn't exist. So AustenAlike is a proof of concept for the methodology as much as a general-purpose evaluation tool.

Alex: So the contribution is twofold: a concrete benchmark for this specific domain, and a framework for thinking about what "character similarity" actually means computationally.

Sam: That's a fair summary. The multi-axial decomposition — structural, social, interpretive — is the transferable idea. It gives you a vocabulary for diagnosing where a model fails, not just that it fails. A system that scores well on social similarity but poorly on interpretive similarity is failing in a different and more interesting way than one that fails across the board. That diagnostic granularity is what the paper is really offering.

Alex: And the implication for the field is that closing this gap probably requires moving beyond feature extraction pipelines toward representations that encode narrative structure more directly.

Sam: That's where the paper points. Whether that means better event representations, discourse-level modeling, or something closer to how large language models process narrative context — the paper doesn't adjudicate. But it gives you a benchmark to test those approaches against, which is the necessary first step.

Alex: A well-scoped contribution, then — a measurable problem statement where previously there was mostly assertion. Thanks for walking through it. Thanks for listening to ResearchPod.

Evaluating Computational Representations of Character: An Austen Character Similarity Benchmark

Funing Yang, Carolyn Jane Anderson

Aug 28, 2024·6:17·arxiv·Original Paper

AustenAlikeComputational Literary AnalysisCharacter SimilarityLiterary PipelinesCharacter RepresentationNLP Pipelines

Listen

6 min

Briefing

Research Question

As computational methods for analyzing literature become more sophisticated, researchers need better ways to evaluate how well these systems understand fictional characters. This paper investigates whether current computational pipelines—which extract features like events, quotes, and assertions—can effectively capture the multi-faceted nature of character similarity. The authors ask: how well do these automated representations align with human-defined notions of character similarity, ranging from structural roles to expert literary analysis?

Approach

The authors introduce AustenAlike, a new benchmark suite based on Jane Austen’s six major novels. They define character similarity through three distinct lenses:

Narrative Roles: Grouping characters by their function in the plot (e.g., Heroine, Deceiver, Rival).
Social Characteristics: Grouping characters by demographic features like rank, wealth, gender, age, and marital status.
Expert Pairings: A "wisdom-of-the-experts" dataset derived from 43 volumes of the journal Persuasions, capturing nuanced comparisons made by literary scholars.

The authors evaluate two popular literary analysis pipelines, BookNLP and FanfictionNLP, by building character representations from extracted features (events, quotes, modifiers, and assertions). They compare these computational vectors against the three benchmarks and also test GPT-4’s ability to rank character similarity.

Main Findings

The study reveals a clear hierarchy in how well computational models understand characters. Feature-based representations, particularly those using events and assertions, perform well at identifying broad social and narrative similarities. For example, models can often group heroines together or distinguish characters by gender. However, all computational methods—including GPT-4—struggle with the expert-defined benchmark. The "expert" similarities are often subtle and thematic, requiring a depth of interpretation that current feature-extraction pipelines do not reach. Even the best-performing models fail to consistently place expert-identified "most similar" characters in their top ten lists.

Why It Matters

This work highlights a significant gap between the "distant reading" capabilities of current NLP pipelines and the nuanced understanding of literary critics. By providing a structured, multi-faceted benchmark, the authors offer a roadmap for future research to move beyond basic demographic or structural classification toward more sophisticated, interpretive models of literary character.

Key Terms and Definitions

AustenAlike — A benchmark suite consisting of three distinct sets of character similarity groupings (narrative, social, and expert) derived from Jane Austen's novels.
BookNLP — A computational pipeline designed to extract information from literary texts, including character mentions, events, and quotes.
FanfictionNLP — A specialized text processing pipeline tailored for fanfiction, used here to extract character assertions and quotes.
Narrative Role — A classification of a character based on their structural function within a novel's plot, such as Heroine, Deceiver, or Rival.
Persuasions — A peer-reviewed journal dedicated to the study of Jane Austen, used by the authors to extract expert-identified character similarities.

Abstract

Several systems have been developed to extract information about characters to aid computational analysis of English literature. We propose character similarity grouping as a holistic evaluation task for these pipelines. We present AustenAlike, a benchmark suite of character similarities in Jane Austen's novels. Our benchmark draws on three notions of character similarity: a structurally defined notion of similarity; a socially defined notion of similarity; and an expert defined set extracted from literary criticism. We use AustenAlike to evaluate character features extracted using two pipelines, BookNLP and FanfictionNLP. We build character representations from four kinds of features and compare them to the three AustenAlike benchmarks and to GPT-4 similarity rankings. We find that though computational representations capture some broad similarities based on shared social and narrative roles, the expert pairings in our third benchmark are challenging for all systems, highlighting the subtler aspects of similarity noted by human readers.

Evaluating Computational Representations of Character: An Austen Character Similarity Benchmark: Research Paper Podcast