A Machine Learning-Based Sentiment Analysis Framework for Twitter Tweets Using Natural Language Processing

Abstract

With the escalating need to derive and evaluate subjective information from social media platforms such as Twitter, where millions of Twitter users update their tweets in the form of textual data every day, the application of Sentiment Analysis becomes even more relevant. This paper proposes an integrated system by integrating the applications of NLP and ML techniques to classify and derive the sentiment and psychological indicators of users instantaneously and with higher accuracy and explainability. For the instant derivation of sentiment polarity and psychological indicators with better accuracy, the proposed system uses TF-IDF feature extraction with Logistic Regression classification. In addition, it uses VADER sentiment analyzer that gives an instant rule-based classification of emotional tone of given text. The proposed system includes a number of preprocessing stages of data such as microtext normalization, tokenization, removal of stopwords and lemmatization. The proposed system is implemented and the results indicate that the proposed lightweight and interpretable approach obtains performance close to the state-of-the-art while maintaining the lowest computational cost. The proposed system is showcased via the development of a Streamlit-based web application to make the system user-friendly for researchers, practitioners as well as for the public. The derived system can instantaneously classify any text to positive, negative, neutral sentiments and also derive the psychological indicators of users related to mental health such as anxiety, sadness and also depression. Hence, the proposed system bridges the existing gap between research prototypes and practical applications with respect to social media sentiment analysis.