Reputation at Risk: Sentiment Analysis and Social Media Listening Tools under the Lens of Critical Multimodal Discourse Studies
DOI:
https://doi.org/10.7146/hjlcb.vi64.145480Keywords:
Multimodality, Sentiment Analysis, Critical Multimodal Discourse Studies, Artificial Intelligence, Online Crisis Management Communication, Social Media Listening ToolsAbstract
In the field of online crisis management communication, AI-based social media listening tools (i.e., tools designed to track and monitor online conversations about a topic or brand) play a pivotal role in opinion mining and reputation audits. Compared to manual analyses, AI enables a faster large-scale collection and classification of vast amounts of data from several online platforms, thus facilitating the task of detecting and monitoring the sentiment linked to a brand and/or product. Nonetheless, AI-based analyses are far from unbiased. This paper adopts a critical multimodal perspective to explore the challenges linked to sentiment analysis in digital multimodal aggregations. It presents an empirical study in which the results of sentiment analysis performed by the social media listening tool Meltwater and manual tagging are compared to evaluate the efficacy of the tool in assessing online reputation damage following a crisis event. The findings of this study suggest that, in addition to possible incorrect classifications of texts (e.g. lack of understanding of pragmatic features or texts in languages other than English), the AI-based tool’s misinterpretation of emotional cues also includes multimodal ensembles. This is due to social media listening tools’ dependence on unimodal (verbal-only) classifiers that fail to produce reliable outputs. Despite the predictive power of these tools, the findings ultimately indicate that the accuracy of sentiment analysis is still affected by a hard-to-die bias concerning the primacy of language over non-verbal communication – a trend which is in contrast with the multimodal nature of semiosis and the proliferation of complex multimodal artifacts online.
References
Al-Ghamdi, L. M. (2021). Towards adopting AI techniques for monitoring social media activities. Sustainable Engineering and Innovation, 3(1), 15-22.
Angel, S. O. et al. (2021). Systematic literature review of sentiment analysis in the Spanish language. Data Technol Appl, 55(4), 461-479. https://doi.org/10.1108/DTA-09-2020-0200.
Avery, E. J. (2017). Public information officers’ social media monitoring during the Zika virus crisis, a global health threat surrounded by public uncertainty. Public Relations Review, 43(3), 468-476.
Bateman, J. A. et al. (2017). Multimodality: Foundations, research and analysis: A problem-oriented introduction. De Gruyter Mouton.
Boianu, C., & Bîră, M. (2021). Using social media listening in crisis communication and management: New methods and practices for looking into crises. Sustainability, 13, 13015. https://doi.org/10.3390/su132313015.
Calvo, R., & D’Mello, S. (2010). Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing, 1(1), 18-37.
Cambria, E. et al. (2017). Affective Computing and Sentiment Analysis. In Cambria, E. et al. (eds) A practical guide to sentiment analysis (pp. 1-10). Springer.
Catenaccio, P. (2021). Discursive strategies of legitimation on the web: Stakeholder dialogue in the agri-biotech industry. Discourse, Context & Media, 43, 100535. https://doi.org/10.1016/j.dcm.2021.100535
Catenaccio, P. (2023). Large Language models and discourse analysis. An introduction. Paper presented at: International Symposium AI: Challenges and potentialities. A Discourse Analysis Perspective. 19th-20th October 2023. University of Bergamo, Italy.
Coombs, W. T. (2019). Crisis Communication. The best evidence from research. In Gephart, R. et al. (eds) The Routledge companion to risk, crisis and emergency management (pp. 51-66). Routledge.
Djonov, E., & Zhao, S. (2014). From Multimodal to Critical Multimodal Studies through Popular Discourse. In Djonov, E. and Zhao, S. (eds) Critical multimodal studies of popular discourse (pp. 1-16). Routledge.
Duarte, N. et al. (2017). Mixed messages? The limits of automated social media content analysis. Center for Democracy and Technology. https://cdt.org/insights/mixed-messages-the-limits-of-automated-social-media-content-analysis/ (last ac-cessed 14.07.2023).
Evans, V. (2017). The emoji code: The linguistics behind smiley faces and scaredy cats. Picador.
Eriksson, M. (2018). Lessons for crisis communication on social media: A systematic review of what research tells the practice. International Journal of Strategic Communication, 12, 526-551.
Fairclough N. (1992). Discourse and social change. Polity Press.
Fairclough, N., & Wodak, R. (1997). Critical discourse analysis. In T. A. van Dijk (ed), Discourse as social interaction (pp. 258-284). Vol. 2. Sage.
Foucault, M. (1972). The archaeology of knowledge and the discourse on language. Pantheon.
Halliday, M. A. K. (1978). Language as social semiotic: the social interpretation of language and meaning. Edward Ar-nold.
Hayes J. L. et al. (2021). Can social media listening platforms’ artificial intelligence be trusted? Examining the accuracy of Crimson Hexagon’s (Now Brandwatch Consumer Research’s) AI-driven analyses. Journal of Advertising, 50(1), 81-91.
Jain, P. K. et al. (2021). A systematic literature review on machine learning applications for consumer sentiment analysis using online reviews. Computer Science Review, 41, 100413.
Jewitt et al. (2016). Introducing Multimodality. Routledge.
Jewitt, C. (2016). Multimodal analysis. In Georgakopoulou, A. and Spilioti, T. (eds) The Routledge handbook of language and digital communication (pp. 69-84). Routledge.
Jewitt, C., & Kress, G. (2010). Multimodality, literacy and school English. In D. Wyse, R. Andrews and J. Hoffman (eds), The Routledge international handbook of English, language and literacy teaching (pp. 342-352). Routledge.
Joshi et al. (2017). Sentiment resources: lexicons and datasets. In E., Cambria et al. (eds), A practical guide to sentiment analysis (pp. 85-106). Springer.
Khatua A. et al. (2020). Predicting political sentiments of voters from Twitter in multi-party contexts. Applied Software Computer Journal, 97, 106743. https://doi.org/10.1016/j.asoc.2020.106743.
KhosraviNik, M. (2018). Social media techno-discursive design, affective communication and contemporary politics. Fudan Journal of the Humanities and Social Sciences, 11, 427-442. https://doi.org/10.1007/s40647-018-0226-y
KhosraviNik, M. (2020). Digital meaning-making across content and practice in social media critical discourse studies. Critical Discourse Studies. https://doi.org/10.1080/17405904.2020.1835683.
KhosraviNik, M., & Unger, J. (2016). Critical discourse studies and social media: power, resistance and critique in chang-ing media ecologies. In R. Wodak and M. Meyer (eds), Methods of Critical Discourse Studies (pp. 206-233). Sage.
Kotras, B. (2020). Opinions that matter: The hybridization of opinion and reputation measurement in social media listening software. Media Culture and Society, 42 (7-8), 1495-1511.
Kress, G., & van Leeuwen, T. (1996/2020). Reading images: The grammar of visual design. Routledge.
Kress, G. (2010). Multimodality. A social semiotic approach to contemporary communication. Routledge.
Lai, S. et al. (2023). Multimodal Sentiment Analysis: A survey. https://arxiv.org/abs/2305.07611 .
Lanfranchi, V. (2017). Machine learning and social media in crisis management: agility vs ethics. WiPe Paper – Ethics Legal and Social Issues Proceedings of the 14th ISCRAM Conference.
Lei, L., & Liu, D. (2021). Conducting Sentiment Analysis. Cambridge University Press.
Lemke, J. L. (2002). Travels in hypermodality. Visual Communication, 1(3), 299-325.
Levashina, J. et al. (2014). The structured employment interview: narrative and quantitative review of the research literature. Pers Psychol, 67(1), 241-293.
Liang, B. et al. (2022). Aspect-based sentiment analysis via affective knowledge enhanced graph convolutional networks. Knowledge-Based Systems, 235, 107643. https://doi.org/10.1016/j.knosys.2021.107643
Liu, B. (2012). Sentiment analysis and opinion mining. Vol. 16. Synthesis Lectures on Human Language Technologies. Morgan & Claypool. https://doi.org/10.2200/S00416ED1V01Y201204HLT016.
Machin, D. (2013). What is multimodal critical discourse studies? Critical Discourse Studies, 10(4), 347-355. https://doi.org/10.1080/17405904.2013.813770.
Majumder, N. et al. (2018). Multimodal sentiment analysis using hierarchical fusion with context modeling. Knowledge-Based Systems, 161, 124-133.
Martinec, R., &Salway, A. (2005). A system for image-text relations in new (and old) media. Visual Communication, 4, 337. https://doi.org/10.1177/1470357205055928.
McDonald, C.G., & Slawson, V. C. (2002). Reputation in an Internet auction market. Economic Inquiry, 40(4), 633-650.
Mohammad, S. M. (2017). Challenges in Sentiment Analysis. In Cambria, E. et al. (eds). A Practical guide to sentiment analysis (pp. 61-84). Springer.
Moschini, I., & Sindoni, M. G. (2022). Mediation and multimodal meaning making in digital environments. Routledge. https://doi.org/10.4324/9781003225423.
O'Halloran, K. L. et al. (2014). Multimodal pragmatics. In Schneider, K. and Barron, A. (eds) Pragmatics of discourse (pp. 239-268). De Gruyter Mouton.
Papacharissi, Z. (2015). Affective publics: Sentiment, technology, and politics. Oxford University Press.
Perakakis, E. et al. (2019). Social media monitoring: an innovative intelligent approach. Designs, 3(2), 1-12.
Picard, R. (2000). Affective computing. Boston: The MIT Press.
Polli C., Sindoni, M.G. (2024). Multimodal computation or interpretation? Automatic vs. critical understanding of text-image relations in racist memes. Discourse, Context & Media, 57, 100755. https://doi.org/10.1016/j.dcm.2024.100755
Santonocito C.S., Polli C., (2023). An experimental critical multimodal discourse study to the AI-driven sentiment analysis of online crisis communication. Lingue e Linguaggi, 59, 333-349. https://doi.org/10.1285/i22390359v59p333
Schreier, M. (2012). Qualitative content analysis in practice. Sage.
Schwaiger, J. et al. (2021). UR: SMART–A tool for analyzing social media content. Information Systems and e-Business Management, 19, 1275-1320.
Sindoni M. G. (2013). Spoken and written discourse in online interactions: A multimodal approach. Routledge.
Sindoni M. G. (2023). Multimodality and translanguaging in video interactions. Cambridge University Press.
Sunderland, J. (2004). Gendered discourses. Palgrave Macmillan.
Upadhyaya, A. et al. (2023). Towards sentiment and Temporal Aided Stance Detection of climate change tweets. Informa-tion Processing & Management, 60(4). https://docs.tweepy.org/en/stable/
Vijay, D. et al. (2018). Corpus creation and emotion prediction for Hindi-English code-mixed social media text. In: Pro-ceedings of the 2018 conference of the North American chapter of the Association for Computational Linguistics: stu-dent research workshop, pp 128-135.
Wankhade, M. et al. (2022). A survey on sentiment analysis methods, applications, and challenges. Artif Intell Rev, 55, 5731-5780. https://doi.org/10.1007/s10462-022-10144-1
Ziora, L. (2016). The sentiment analysis as a tool of business analytics in contemporary organizations. Studia Ekonomiczne, 281, 234-241.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Chiara Polli, Carmen Serena Santonocito
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
a. Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
b. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
c. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
