Enhancing deep learning models for time series classification via knowledge distillation

This paper investigates the effectiveness of knowledge distillation (KD) as a model compression technique for time series classification (TSC). The authors aim to determine if transferring knowledge from large, complex teacher models to smaller, more efficient student models can maintain high classification performance while reducing computational and memory requirements. The study evaluates this across three distinct deep learning architectures: the Fully Convolutional Network (FCN), the Inception model, and the transformer-based ConvTran model.

The researchers evaluate their approach using the UCR Archive, a comprehensive benchmark repository for time series data. They systematically modify architectural components—such as convolutional filters, Inception modules, and attention heads—to create student models of varying sizes. The distillation process involves training the student model using a total loss function that combines standard cross-entropy loss (using hard labels) and a distillation loss (using Kullback–Leibler divergence on softened teacher predictions). This allows the student to mimic the teacher's output distribution, effectively capturing the teacher's generalization capabilities.

The study demonstrates that knowledge distillation is most effective for student models of intermediate complexity. Key results include:

• The distilled FCN student model achieved a 38-fold reduction in parameters.
• The distilled Inception student model maintained performance nearly identical to its teacher while using 42% fewer parameters.
• The ConvTran student model with two attention heads showed the most significant performance gains through the distillation process.

As deep learning models for time series analysis become increasingly complex, their deployment on resource-constrained devices (such as embedded sensors or real-time edge hardware) becomes difficult. This research provides a practical framework for compressing state-of-the-art TSC architectures without sacrificing accuracy, facilitating the use of advanced deep learning in real-world, resource-limited environments.