Document parsing is now widely used in applications, such as large-scale document digitization, retrieval-augmented generation, and domain-specific pipelines in healthcare and education. Benchmarking these models is crucial for assessing their reliability and practical robustness. Existing benchmarks mostly target high-resource languages and provide limited coverage for low-resource settings, such as Turkish. Moreover, existing studies on Turkish document parsing lack a standardized benchmark that reflects real-world scenarios and document diversity. To address this gap, we introduce OCRTurk, a Turkish document parsing benchmark covering multiple layout elements and document categories at three difficulty levels. OCRTurk consists of 180 Turkish documents drawn from academic articles, theses, slide decks, and non-academic articles. We evaluate seven OCR models on OCRTurk using element-wise metrics. Across difficulty levels, PaddleOCR achieves the strongest overall results, leading most element-wise metrics except figures and attaining high Normalized Edit Distance scores in easy, medium, and hard subsets. We also observe performance variation by document type. Models perform well on non-academic documents, while slideshows become the most challenging.
Alex: Welcome to another episode of ResearchPod.
Sam: The paper, titled "OCRTurk: A Comprehensive OCR Benchmark for Turkish," introduces a new testing set for tools that pull text and other parts from scanned pages. These tools struggle with Turkish because most tests are built for English, missing Turkish's tricky letter shapes and flexible sentence structures.
Alex: So this is basically about creating a fair test for Turkish documents, since the usual ones from English don't capture the real challenges?
Sam: Yes, exactly. Imagine trying to digitize old Turkish theses or academic PDFs—English-based systems warp special characters like ç or ğ, and crumble on mixed layouts with tables or equations. The paper fills that gap with OCRTurk, the first benchmark using 180 real Turkish documents from articles, theses, slideshows, and non-academic texts, split into easy, medium, and hard levels based on layout complexity.
Alex: Right, so the core problem is that without a proper Turkish test set, we can't tell how well these systems handle everyday documents like slides full of figures or tables?
Sam: That's the key issue. Existing Turkish work focuses mostly on raw text from handwriting or fonts, or synthetic data from one book, but skips real-world variety like equations and figures. OCRTurk tests seven models on pulling out text, tables, math formulas, and images accurately.
Alex: And why does that matter for things like digitizing archives or feeding data to AI?
Sam: It matters because reliable parsing unlocks uses like searching huge document collections or powering AI summaries in education and healthcare. Without tests reflecting Turkish's word flexibility and diverse layouts, models seem better than they are on real archives. This benchmark, with pages turned into precise Markdown formats by hand-checkers, gives a true measure. Models shine on simple non-academic pages but falter where structures get dense.
Alex: So models do well on straightforward pages, but layouts with tables or equations expose the weaknesses. How did they ensure the test set itself was spot-on accurate?
Sam: They started with real PDFs from sources like academic archives, theses databases, and slideshow materials. Two annotators converted them to a standard format: plain text copied word-for-word, tables reshaped into a grid code that computers read easily, math equations rewritten in a math notation language, and images saved separately. Tools like ChatGPT helped speed up the formatting, but everything got checked by hand, character by character, against the originals to catch mistakes. They also cropped out repeating headers and footers first, since many models ignore those anyway—this keeps the test fair.
Alex: That manual step sounds thorough—like double-checking homework to avoid grading errors. With that solid ground truth, how exactly did they measure if models got the text, tables, and so on right?
Sam: First, they pulled out the key parts from the models' outputs and the true versions, then cleaned the text by fixing common glitches in Turkish letter marks—like turning a misplaced dot over a g into the proper shape—and stripping extra labels. For plain text, one measure counts the minimum changes needed to match the original, like fixing typos in a copied essay; they divide that by the longest version's length to get a score from zero to one, where closer to one is better. A second check focuses just on Turkish's special letters, like ç or ş, by seeing what fraction get mangled.
Alex: Okay, so text gets two checks: overall matching and Turkish-specific accuracy. What about tables—do they just compare the words, or the whole structure?
Sam: Tables need both. They use the word-match score we just covered for content. But for layout—like rows, columns, and how cells connect—they treat the table as a branching outline, similar to a family tree, and count the fewest swaps, adds, or deletes to make the model's tree match the real one; closer to one means better structure.
Alex: And across all that, PaddleOCR came out ahead overall... but you mentioned slideshows as a weak spot for everyone?
Sam: Yes, the paper shows PaddleOCR handling text and tables about twice as well as others on average, thanks to its strength on Turkish characters and layouts. But all models, including it, score much lower on figures from slideshows—likely because those mix dense visuals with flexible text blocks that English-tuned systems don't expect. This highlights a clear gap: real Turkish documents need benchmarks like this to spot those limits. The dataset and evaluation tools are now public, so others can build on it.
Alex: Right, so even with those checks, figures remain tricky across the board. How do the models stack up when you break it down by easy, medium, and hard documents?
Sam: They combined the text, table, and equation scores into one overall mark for each difficulty level—the closer to one, the fewer fixes needed to match the real content. All models hit near-perfect on easy pages, since those have simple layouts without much clutter. Hard pages with dense multi-line math, big tables, and subplot figures test them more, where PaddleOCR edges ahead.
Alex: Interesting—PaddleOCR holds up best on the tough ones. Does that pattern hold when looking at document types, like academic versus slideshows?
Sam: Yes, they averaged scores the same way across categories. Non-academic pages score highest, being straightforward without heavy math or grids. Slideshows lag due to overlapping visuals and irregular text blocks that confuse extraction. PaddleOCR tops or ties three out of four categories.
Alex: So non-academic is easiest, slideshows hardest—makes sense with the visual mess you mentioned. Overall, though, PaddleOCR's consistency stands out.
Sam: Precisely. It leads text, equations, tables broadly, and holds in difficulties and categories. This element view, plus public data, spots where Turkish docs need work—like slideshow figures—without assuming any model is flawless. The evidence points to targeted improvements ahead.
Alex: Yeah, that consistency across elements and categories gives PaddleOCR a real edge. But with those patterns in mind, what kinds of mistakes do the models keep making that pull down the scores?
Sam: The paper points to common slips across models—like mishandling Turkish special characters, which leads to missing titles or skipped lines. In equations, they swap variables or write math in plain text instead of proper format. Tables often get treated as loose text, with invented data in empty spots or phantom columns added. Figures in image-heavy columns fool systems into grabbing whole sections as one big picture.
Alex: So it's not just missing pieces, but inventing or misclassifying them too. Does the paper flag any bigger limits in the benchmark itself?
Sam: Yes, they note the set is limited to 180 pages because hand-annotating ground truth takes significant time—each page needs careful conversion and verification. It misses some edge cases like photographed documents or even wider types beyond the four categories. Still, this size captures meaningful variety in real Turkish docs.
Alex: Fair point on the scale—manual work isn't quick. Looking ahead, how might this push things forward practically?
Sam: The public dataset and scripts let others test new models directly, and plans for an online leaderboard will track progress systematically. That speeds improvements for digitizing Turkish archives, theses, or educational materials—making search and AI summaries reliable where English tools fail. For low-resource languages like Turkish, it means better handling of heritage docs.
Alex: That's a clear picture of where things stand and what's next. Thanks, Sam—this has been a thoughtful dive into OCRTurk and Turkish document challenges. Thanks for listening to ResearchPod.