Best Preprocessing Techniques for Sentiment Analysis

Sentiment analysis on social media is notoriously difficult due to informal language, slang, and noise. While researchers often apply various preprocessing techniques—such as stemming, tokenisation, and spelling correction—to clean this data, there is little consensus on the optimal sequence for these steps. This paper systematically evaluates the impact of different preprocessing orders on sentiment classification performance using three Twitter datasets and four standard machine learning models: Naive Bayes, Support Vector Machines (SVM), Decision Trees, and K-means.

The researchers tested 15 different sequences of preprocessing steps to determine which order maximizes F1-score accuracy. They found that tokenisation is the most critical step, as it provides the foundation for all subsequent operations. Conversely, spelling correction was consistently the least impactful technique, often introducing as much noise as it removed. The study recommends a specific pipeline: tokenisation, then text cleaning (specifically text lowering), followed by stemming and stop-word removal. Notably, the authors found that stop-word removal should be performed before stemming to improve computational efficiency, and they emphasize that negating words like 'not' and 'no' must be preserved to maintain sentiment accuracy.

Preprocessing is a computationally expensive and time-consuming phase of the machine learning pipeline. By identifying a systematic, high-performing order for these techniques, practitioners can avoid the 'exploratory phase' of trial-and-error, saving significant time and resources. The results provide a clear, actionable roadmap for building more robust sentiment analysis models for social media data without needing to test thousands of arbitrary combinations.