Quantum Cognitively Motivated Decision Fusion Framework for Video Sentiment Analysis

Video sentiment analysis as a decision-making process is inherently complex, involving fusion of decisions from multiple modalities and the so-caused cognitive biases. Inspired by recent advances in the emerging field of quantum cognition, we show that the sentiment judgment from one modality could be incompatible with the judgment from another, i.e., the order matters and they cannot be jointly measured to produce a final judgment. Thus the cognitive process in video sentiment analysis exhibits ``quantum-like’’ biases that cannot be captured by classic probability theories. Accordingly, we propose a fundamentally new, quantum cognitively-motivated fusion framework for predicting sentiment judgments. In particular, we formulate utterances as quantum superposition states of positive and negative sentiment judgments, and uni-modal classifiers as mutually incompatible observables, on a complex-valued Hilbert space with positive-operator valued measures. Experiments on two benchmarking datasets illustrate that our framework significantly outperforms various existing decision level and a range of state-of-the-art content-level fusion approaches. The results also show that the concept of incompatibility allows effective handling of all combination patterns including those extreme cases that are wrongly predicted by all uni-modal classifiers. Download paper here