ResearchPod Summary
PlanarBench investigates whether Large Language Models (LLMs) possess genuine spatial reasoning capabilities. Unlike prior benchmarks that focus on symbolic graph algorithms (e.g., shortest paths or connectivity), this study requires models to produce a valid, crossing-free, straight-line ASCII drawing of a planar graph given only an edge list. This task forces the model to perform holistic constraint satisfaction, as every edge placement must account for the geometric constraints imposed by all other edges.
The author evaluated 91 LLMs on a corpus of 199 non-isomorphic connected planar graphs ranging from 2 to 7 vertices. The evaluation uses a mechanical validation pipeline that checks for node presence, edge connectivity, and the absence of edge crossings. To account for the limitations of ASCII art, the study employs a tiered scoring system: strict validation for perfect straight-line drawings, and relaxed validation for drawings that are topologically correct but use non-straight paths or require coordinate-based verification.
The study reveals that edge count is the dominant predictor of task difficulty, with a Pearson correlation of r = -0.85, significantly outperforming vertex count (r = -0.47). This suggests that the primary bottleneck for LLMs is the combinatorial complexity of routing edges without intersections, rather than the number of nodes. Furthermore, the benchmark identifies a performance cliff: models with fewer than 30 billion parameters struggle significantly, and solve rates for graphs with 11 or more edges drop below 2% across all models. The results also show that standard leaderboard rankings are poor predictors of spatial reasoning performance, and that extended thinking tokens can substantially improve results for certain model architectures.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a new study that tests how well large language models—or LLMs—can actually "see" and map out spatial relationships.
Sam: That's right. The paper introduces something called "PlanarBench," which forces models to draw a specific type of diagram using only text characters. The central claim is that these models struggle when the complexity of the connections, rather than just the number of points, increases.
Alex: So this paper is basically asking whether these models can move beyond memorizing facts and actually perform real-time spatial planning?
Sam: Exactly. Most current tests for AI models use symbolic tasks, like finding the shortest path between two points. Those can often be solved by memorizing a standard algorithm. PlanarBench is different because it requires the model to build a layout from scratch—and the hard constraint is that no lines can cross each other. That's a much more demanding kind of problem.
Alex: That's like asking someone to draw a complex subway map on a napkin where none of the tracks are allowed to cross.
Sam: That's a good way to put it. Imagine you have a set of stations and you need to connect them with lines. Every time you add a new track, you have to make sure it doesn't intersect with any you've already drawn. The model has to place its "stations"—the nodes—and then route the "tracks"—the edges—without creating a tangle. And it has to do all of that using only keyboard characters.
Alex: Which brings up the ASCII art angle. Why use that format specifically?
Sam: It's a deliberate choice. ASCII art—using standard keyboard characters like dashes and pipes to create images—is something these models have encountered millions of times in their training data. More importantly, it makes the output automatically checkable. You don't need a human to look at the drawing and decide if it's correct. The code just scans the grid and asks: are all the nodes present? Are the lines in the right places? And are there any illegal crossings?
Alex: So it's a clean, objective test. No ambiguity about whether the answer is right or wrong.
Sam: Precisely. The researchers call it "mechanically verifiable." That matters a lot for a benchmark, because it means you can test 91 different models and get consistent, comparable results.
PlanarBench provides a domain-neutral probe for constraint-satisfaction capacity that is resistant to memorization due to the permutability of node labels. By shifting the focus from symbolic computation to geometric layout, it exposes a different facet of model intelligence that is not captured by existing graph-theoretic benchmarks. The open-ended nature of the graph corpus allows for continuous scaling, providing a long-term metric for evaluating the spatial reasoning progress of future models.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Alex: And what did those results actually show?
Sam: A very clear pattern. The biggest predictor of failure wasn't the number of nodes—the stations—but the number of edges, meaning the connections between them. As you add more connections, the number of valid ways to draw the graph without any crossings shrinks rapidly, and the models hit a wall.
Alex: So the bottleneck isn't how many things you're connecting, but how many connections you're trying to manage at once?
Sam: Exactly. And the drop-off is sharp. Once you get to around 11 or 12 edges, success rates fall below two percent. The paper describes it as a "capability cliff"—a point where the models essentially stop being able to solve the puzzle at all.
Alex: That's a significant drop. Does this mean these models simply don't have a mental map of two-dimensional space?
Sam: The paper suggests that current models lack a deep, holistic spatial reasoning ability. When they succeed, it's often because they've seen similar patterns before. But when you give them a graph they haven't memorized, they struggle to navigate the geometric constraints. It's not just about knowing where things go—it's about understanding how each placement limits your future moves. Like in chess, where every piece you put down changes what's possible next.
Alex: You mentioned that some models have an "extended thinking" feature. Does that help?
Sam: It does, for some architectures. The idea is that the model generates hidden reasoning steps before committing to a final answer. The study shows that for certain models, this extra processing time leads to a meaningful jump in performance—as if the model is mentally sketching out the graph before putting it on the grid.
Alex: Does that translate to better performance on other AI benchmarks too?
Sam: Interestingly, no. The models that top the usual leaderboards aren't necessarily the best at this spatial task. That suggests the skills needed to ace a standard test—answering trivia, writing code—are quite different from the skills needed to solve a constraint-heavy geometric puzzle.
Alex: So we might be getting a skewed picture of how capable these models are if we only look at those standard tests.
Sam: That's the researchers' argument. Because PlanarBench is open-ended and the labels can be rearranged, the model can't rely on a shortcut or a memorized pattern. It has to do the spatial work from scratch. That makes it a more honest probe of reasoning.
Alex: What about the format itself, though? Is ASCII holding the models back in ways that aren't really about spatial reasoning?
Sam: That's the main methodological limitation the authors acknowledge. ASCII is a grid, so you're stuck with straight lines and specific diagonal characters. If you were drawing on a blank page, you could curve your lines to avoid a crossing. The grid removes that flexibility, which may make the task artificially harder than it would be in a more open environment.
Alex: So a model might actually have better spatial reasoning than its score suggests, but the grid forces it into mistakes it wouldn't otherwise make?
Sam: The authors are careful about that. They included a "relaxed" scoring system that gives partial credit if the nodes are in the right positions, even if the lines aren't perfectly routed. But even with that extra allowance, the cliff at around 11 edges holds. It's a robust finding across scoring methods.
Alex: So we've found a real, measurable limit in how these systems handle complex, interconnected constraints.
Sam: That's the key takeaway. By moving away from symbolic problems—where a memorized algorithm can carry you—and toward geometric ones, we get a much clearer picture of where the current generation of AI hits its limit. And because the number of valid graphs grows quickly with complexity, the benchmark can keep scaling as models improve. It's a useful measuring stick for the long term.
Alex: A clear-eyed look at the boundaries of machine reasoning. Thanks for walking us through it, Sam.
Sam: It was a pleasure. The data is quite clear, and it gives us a more grounded way to think about what these models actually understand—and what they don't.