Decode the Avatar Face: Affect Recovery in Dyadic VR

Abstract

Humans can naturally infer affect from a face, but in virtual reality (VR), facial cues are obscured by the headset and must be reconstructed on an avatar. In side-by-side collaboration, however, even this reconstructed face may fall outside the shared task view; a virtual mirror can keep it visible without requiring users to turn away from shared content. Yet it remains unclear how much of a user’s affect a partner can recover from this mirror-mediated face alone. To investigate this, a user study was conducted in which avatar faces were isolated in a live dyadic VR task. A Sender expressed an affective reaction to an image using only facial expressions, while a Receiver judged that reaction through a virtual mirror view of the Sender’s avatar face. The study used a 2 × 2 design varying interpersonal distance and mirror orientation behavior. Results show that Receivers decoded affect above chance. Quadrant-level accuracy reached 56.2%, while valence and arousal axes were each recovered at around 76%. Because the exact quadrant requires both axes to be correct, broad emotional direction was easier to read. Modeled through a Truth-and-Bias-inspired analysis, perceived affect was also closely associated with the Receiver’s own concurrent affect. No effect of distance or mirror behavior on performance was detected. These findings suggest that avatar-mediated affect judgments reflect both Sender-linked facial information and receiver-linked affective variation.