Marios Koulakis, Constantin Seibold
5 min
Abstract
A significant gap exists between theory and practice in deep learning. Generalization and approximation error bounds are often derived for simplified models or are too loose to be informative. Many rely on the manifold hypothesis and on geometric regularity such as intrinsic dimension, curvature, and reach. Progress requires insight into data-manifold geometry and suitable benchmarks, yet existing options are polarized: analytic manifolds with known geometry but limited applicability, or real-world datasets where geometry is only coarsely estimable. We introduce a benchmarking framework for studying data geometry. We repurpose and extend dSprites and COIL-20 with additional transformation dimensions and dense, axis-aligned sampling, and pair them with finite-difference estimators that recover curvature, reach, and volume at near-ground-truth accuracy in a regime where general-purpose estimators are unreliable or difficult to deploy. The framework is intended as a controlled testbed, useful as a calibration environment for geometric estimators and a sandbox for probing theoretical assumptions. To illustrate its use, we present two application studies, namely assessing the scaling behavior of the bounds of Genovese et al. and Fefferman et al., and tracking the layer-wise geometry of a $β$-VAE, highlighting the behavior of current bounds and the value of controlled benchmarks for guiding and validating future theory. A reference implementation is available at https://github.com/koulakis/manifold-microscope.
Alex: And the framework itself acts as a calibration tool — like a ruler you know is accurate, so you can use it to check whether your other, less certain instruments are giving you the right readings?
Sam: That's a perfect way to put it. The authors use this known-geometry testbed to do two things. First, they check whether the theoretical error bounds from previous research actually hold up when you run real experiments. Second, they use it to track how a specific type of AI model — called a beta-VAE — reshapes the geometry of data as it passes through successive layers of its internal network.
Alex: So instead of just guessing why a model performs well or poorly, you can actually watch the geometric transformations happening layer by layer. What does that reveal?
Sam: It reveals whether the model is doing what the theory predicts. If the geometry changes in ways that match the theoretical expectations, that's evidence the theory is sound. If it doesn't, that's a signal the theory needs revision. Either outcome is useful — it's the difference between having a map you can trust and navigating blind.
Alex: That makes sense. Now, every tool has limits. What are the boundaries of this approach?
Sam: The framework currently works well in low-dimensional spaces — roughly where the intrinsic dimension of the data is around four or five. When the data becomes higher-dimensional or has irregular shapes that don't fit clean patterns, the approach struggles. There's also an issue the authors call rasterization artifacts. Think of trying to measure a smooth curve on a screen made of large, blocky pixels. If your measurement grid is too coarse relative to the underlying shape, you end up measuring the jagged edges of the pixels rather than the smooth curve beneath. That creates noise that obscures the true geometry.
Alex: It's like trying to draw a circle using square tiles — at some point, you're just seeing the tiles. So what does the future of this work look like?
Sam: The logical next step is extending the framework beyond flat, standard surfaces. Future versions could handle what are called non-Euclidean topologies — shapes that curve or twist in ways a standard grid can't easily capture, like the surface of a sphere or a torus. That would open the door to what the authors describe as geometric debugging for much larger, more powerful models — the kind used today for generating images or text.
Alex: So the trajectory is from a precise but limited calibration tool toward something that can handle the full complexity of modern AI. And the underlying value is transparency — being able to see, rather than assume, how a neural network is processing information.
Sam: That's exactly it. By turning abstract geometric theory into concrete, reproducible experiments, this framework gives researchers a way to verify their assumptions rather than simply hoping they hold. That kind of rigorous foundation is what the field needs as these models grow more powerful and more consequential.
Alex: A clearer view inside the black box. Thanks for walking me through it, Sam. And thanks to everyone listening — this has been ResearchPod.