From Data to Emotion: A Multimodal Approach to Real-Time Emotion-Aware Marketing

Abstract

This study presents a system that aims to dynamically analyze users' emotional states and deliver real-time campaign recommendations. Recognizing that emotional cues manifest across different channels, the system adopts a multimodal approach that integrates biometric signals, social media content (both visual and textual), and application behavior data. By fusing these diverse data sources, the system enhances its ability to accurately infer users' emotional states. Data is collected from mobile applications, wearable devices, and third-party platforms. Apache Kafka serves as the message broker, while Apache Flink performs real-time event processing—either directly or after interpretation by the Artificial Intelligence Module. Campaigns are selected based on the inferred emotional state and pushed to the user as personalized recommendations. The high-level goal of this work is to develop a robust multimodal AI model capable of detecting users' emotional states from heterogeneous data streams. Unlike existing approaches, this system integrates multimodal signals in real time, and it is expected to achieve high accuracy in emotion recognition. This system has strong potential for deployment in smart services, user engagement platforms, and real-time decision-making environments.