Embaudy

A real-time interactive installation that visualizes student emotions through body movement and facial expression.


Type

Interaction Design


Role

Designer

Team

Divya Dhavala, Afasana Shanta

Timeline

2 months

Tools

Touch Designer, Figma, Midjourney, Media Pipe, Google Forms, Premiere Pro

Outcome

Case Study Presentation

Problem: Post-pandemic, college campuses continue to grapple with invisible crises. According to the National College Health Assessment and Active Minds Survey, over 60% of students report ongoing anxiety or depression. Many struggle to identify what they’re feeling, let alone communicate it. Embaudy began as a response to that silence. What if the way we move could speak for us? What if an interactive system could help visualize those feelings without saying a word?

Solution: To address this gap, our team explored how computer vision, creative coding, and real-time feedback could create a more approachable entry point into mental health dialogue. Embaudy invites passersby to engage with a digital screen that mirrors their gestures and expressions through dynamic visuals. The result is a playful, ambient experience that helps students reflect on how they feel, without needing to explain it out loud.


Initial Brief & Exploration

Our team set out to create an interactive design project using creative coding, computer vision, and AI. With such a broad scope, our team:

Researched installations featuring motion-tracking, emotion-recognition, and immersive public experiences (e.g., marketing campaigns, art performances, and emotional-sensing devices).

Conducted an affinity mapping exercise to cluster insights, revealing three core themes:

  1. Body & Facial Tracking

  2. Interactive Public Installation

  3. Emotional/Mental Health

From there, we refined our project brief:

Grounding the problem space with secondary research

Data from sources like the National College Health Assessment indicated that 60% of college students met criteria for at least one mental health issue post-pandemic. Anxiety affected 59%, while 48% faced depression (Active Minds Survey). This confirmed a genuine need to address emotional awareness on campus.

User Research

Surveying 25 students on their emotional routines revealed shared patterns:

  • Difficulty Expressing Emotions: Students often used vague terms like “good,” “tired,” or “tense.”

  • Existing Methods: Many students reflect daily on their feelings using journaling, talking with friends, or creative outlets.

  • Desire for Emotional Awareness: Most believed understanding and communicating emotions could positively affect stress management and relationships.

  • Nonverbal Stress: Students noticed changes in body language, sleep, and appetite, reinforcing the importance of an interactive, visual approach.

Defining the project goals

  1. Offer an ambient, public-facing experience where students can reflect on how they feel

  2. Encourage open discussions about mental health without positioning Embaudy as a clinical solution.

Constraints

  • Limited to accessible tech: TouchDesigner, MediaPipe, and standard webcams

  • The installation needed to be intuitive and visually legible without guidance

  • Short timeline and public setting demanded a robust but minimal system

Ideation & Prototyping

We generated dozens of concepts through sketching and rapid prototyping. After testing out multi-screen and audio-based ideas, we focused on the clearest pathway: using facial expressions and gestures to trigger real-time visual feedback.

Technical Challenges

  1. Multilayered screen interactions were difficult to implement since there were no way to save user data from screen to screen. We pivoted and opted for a one screen layout

  2. Facial tracking was not accurate and would loose focus. We exaggerated some of the user input data from the hand movements, to increase the visualization feedback to the users. We also tried the Kinect Azure camera for better tracking however it was deprecated and didn’t work with our current devices

Visual Design & Emotional Mapping

To make the experience feel intuitive and emotionally resonant, we grounded our visuals in symbolic systems:

  • Color Mapping: Survey respondents often described emotions in color, so we researched emotional color theory from institutions like the University of Toronto to design a consistent palette

  • Elemental Motifs: Fire, water, air, and earth served as metaphoric anchors, making each visual state feel organic and expressive

  • Particle Systems: Using GPU-based flows in TouchDesigner, we created real-time effects that danced, swirled, or pulsed in response to gesture and expression

Initial Prototype

Our initial prototype features a large digital screen in a public area. Users approach, and the camera detects facial gestures and hand movements. We wanted Embaudy to feel like a mirror made of motion—a soft swirl of particles that responded to your body language like a mood ring, glowing and changing with even the slightest shift in posture or tone.

1. Real-Time Visual Change: Colors and particle motions adjust in response to user movement or expression.

2. Emotional Prompt: A subtle legend indicates general emotion categories (e.g., “calm,” “energetic,” “tense”) tied to color shifts.

3. Resource Link: A QR code directs users to mental health resources, encouraging deeper reflection.

Feedback & Iteration

We presented Embaudy to a panel of design faculty and peers. Feedback included:

  • Clarify Purpose: Emphasize that this is an art-driven interactive piece, not a clinical diagnostic tool.

  • Call to Action: Move the QR code front and center for easier access.

  • Make It Fun & Approachable: Lean on playful elements to reduce potential intimidation around mental health topics.

Outcome

Initial testers said the visuals made them feel unexpectedly “seen.” Reviewers noted the emotional clarity and accessibility of the experience, praising its potential for deployment across campus. By embracing creative technology as a mirror, we opened a low-pressure, visual entry point into emotional reflection.

Next Steps

  • Usability Testing: Gather on-campus data to see if visuals and interactions effectively spark dialogue.

  • Technical & Accessibility Upgrades: Experiment with improved sensors or alternative cameras; consider public installations in varied lighting.

  • Collaboration: Partner with industrial designers to refine hardware setup and explore resilience in unsupervised public spaces.

  • Refine Business Value: Identify campus partners (e.g., counseling centers, student organizations) who might sponsor or co-promote the installation, ensuring long-term viability.

Lessons Learned

Designing Embaudy taught us that emotional experiences in public space must be intuitive, expressive, and safe.

  • Subtle gestures need bold responses: We learned that small expressions often went unnoticed by the system. Amplifying visual feedback with vivid colors and motion helped make interactions feel clear and engaging.

  • Emotional design must feel welcoming: Framing the piece as an artistic experience rather than a clinical tool made it more approachable. Students were more open to engaging when the tone was playful and reflective rather than prescriptive.

  • Nonverbal systems can spark reflection: By translating movement and expression into visual language, we saw how students could explore their emotions without needing to put them into words.

  • Interdisciplinary collaboration was essential: Working across creative coding, visual design, and mental health storytelling helped us create a system that was both technically sound and emotionally resonant.