Multimodal analysis
Multimodal analysis is the interpretation of data that combines several forms of communication, such as speech, tone of voice, facial expression, gesture, images and on-screen activity, recognizing that meaning comes from how they work together.
How it works
Traditional qualitative analysis works from transcripts, which keep the words and lose everything else. Multimodal analysis restores the rest: a hesitation, a laugh, a frown at a price, a cursor hovering over a button for 10 seconds. Analysts transcribe or code each mode, then examine how they align, for example whether a positive statement comes with a positive tone.
The approach draws on social semiotics, interactional sociology and sensory anthropology, and qualitative analysis software can hold video and audio alongside transcripts for coding. In customer research, it compares what a participant says with their tone, expression and on-screen actions at the same moment.
A worked example
In their interview transcripts, 14 of 20 participants call a language-learning app's new pricing page "clear". On video, 9 of those 14 pause for more than 5 seconds on the plan table and scroll back to it twice. The team treats "clear" as a polite answer and tests a simpler table.
Related
Automate this in Pulse
A multimodal analyst can read what a participant said alongside Pulse's video insights of their visible reaction during the same answer.