Data Evolution by Wittgenstein’s Rule Following

How can we generate a new dataset that logically continues a sequence of previously observed datasets, even when those datasets have varying sample sizes, feature dimensions, and underlying distributions? The authors address this by moving away from standard synthetic data generation—which typically samples from a fixed distribution—toward a framework that treats data evolution as a process of continuing an implicit rule.

Inspired by Ludwig Wittgenstein’s philosophy, specifically his ideas on rule-following and family resemblance, the authors propose the Wittgenstein’s Rule Following (WRF) framework. Instead of trying to map points from one dataset to another, WRF represents each dataset using a vector of structural descriptors. These descriptors summarize geometric, distributional, and clustering properties (and label-based properties in supervised settings).

To generate the next dataset, the algorithm:

1. Predicts Targets: It calculates a 'rule-following' target by extrapolating the trajectory of historical descriptors and a 'family-resemblance' target by averaging them.
2. Generates Candidates: It creates candidate datasets from the historical data using mixture recombination.
3. Scores and Refines: Candidates are scored based on their alignment with the predicted targets and can be further optimized using differentiable surrogate descriptors to ensure they match the desired structural properties.

This work provides a novel, philomatically-motivated approach to nonstationary data problems. By focusing on structural evolution rather than pointwise transformation, WRF offers a flexible way to model data streams or evolving phenomena where the fundamental nature of the data changes over time. It bridges the gap between philosophical concepts of language-games and practical machine learning, providing a tool for scenarios where traditional data augmentation or generative models are insufficient.