Unveiling Public Opinion: A Study of Sentiment Analysis Using LSTM and Traditional Models

This study evaluates the effectiveness of various machine learning and deep learning algorithms for sentiment analysis on Twitter. As social media platforms become primary venues for public discourse, the ability to accurately categorize user-generated content into positive, negative, or neutral sentiments is essential for trend forecasting and public opinion interpretation. The author compares five distinct models—Logistic Regression, Random Forest, Naïve Bayes, Gradient Boosting, and LSTM—to determine which approach best handles the contextual and sequential nature of textual data.

The research utilizes a Kaggle Twitter dataset consisting of 27,482 entries. The data underwent rigorous preprocessing, including conversion to lowercase, removal of special characters and stopwords, tokenization, and lemmatization. To convert text into numerical input for the models, the study employed TF-IDF vectorization. The dataset was split into an 80% training set and a 20% testing set. Performance was evaluated using standard metrics, including accuracy, precision, recall, F1-score, and the micro-average ROC-AUC score.

The results demonstrate that the LSTM model provides superior performance compared to traditional machine learning techniques. While models like Random Forest showed high training accuracy (97.70%), they exhibited signs of overfitting with lower testing accuracy. The LSTM model achieved a balanced testing accuracy of 80.00% and a robust micro-average ROC-AUC score of 0.92. This indicates that the LSTM's architecture is better suited for capturing the complex, sequential dependencies inherent in social media text, allowing for more reliable sentiment classification.

Effective sentiment analysis enables businesses and policymakers to interpret public opinion in real-time. By identifying that deep learning architectures like LSTM outperform conventional statistical classifiers, this research provides a practical roadmap for developers and analysts looking to build more accurate, adaptive, and scalable systems for monitoring digital communication and market dynamics.