LANGUAGE IDENTIFICATION VIA COMPOSITIONAL DATA ANALYSIS: A LINEAR-TIME CLASSIFIER BASED ON LOG-RATIO GEOMETRY

This paper addresses the limitations of traditional frequency-based language identification, which often applies Euclidean distance metrics to relative frequency data. Because relative frequencies are constrained to sum to unity (a compositional simplex), standard Euclidean geometry can lead to spurious correlations and subcompositional incoherence. The authors propose a pipeline that treats character and bigram distributions as compositional vectors, applying the centered log-ratio (CLR) transformation to map these vectors into a zero-sum subspace where Euclidean distances are mathematically equivalent to Aitchison distances.

The proposed algorithm extracts unigram and bigram counts from text, applies Laplace smoothing to handle sparsity, and transforms the resulting vectors using the CLR mapping. This transformation allows for a deterministic, linear-time classification process. To improve accuracy, the authors incorporate a length-dependent diacritic adjustment term that accounts for the varying importance of orthographic markers across different text lengths. The final classification is determined by minimizing the total weighted distance between the input text's transformed vector and pre-computed language reference profiles.

This approach offers a computationally efficient alternative to deep learning models for language identification. By providing a mathematically rigorous framework for handling compositional linguistic data, the method achieves robust accuracy in linear time (O(L)). This makes it particularly well-suited for resource-constrained environments, such as edge computing or high-throughput preprocessing pipelines, where interpretability and low latency are critical requirements.