EmoSynth

EmoSynth: A combination of Emotion and Synthesizer, highlighting the device’s ability to synthesize emotional expressions.

Sensorium AI x Ability Project

Junru Chen Lang Qin Emily Lei Kassia Zheng Muqing Wang

We are Sensorium AI Group from the Multisensory class. We designed a wearable device that works with Sensorium Ex, Luke, Michael, and our professor, Lauren. To fulfill the design objectives of facilitating real-time artistic expression and modulation for performers with cerebral palsy and/or limited speech, our device allows the user to freely select the emotion and level of emotion they want to express. Additionally, there is a selection of high and low audio to represent diverse genders, along with a reset button to prevent accidental touches during movement on stage. Each selection on the device is denoted by an embedded LED hint to better inform the users.

wear on the arm

wear on the chair ( wheelchair)

tactile soft buttons and the touch bar

EmoSynth with its adjustable add-on piece

Timeline

  • 09/14/2023: Primary research ( Reference cases and Inspiration)

  • 09/21/2023: Ideas brainstorm and sketch

  • 09/28/2023: Finalized our project idea

  • 10/05/2023:Developed first quick prototype

  • 10/12/2023: User testing with the quick prototype

  • 10/19/2023: Implementation of user testing result

  • 10/26/2023: Finalize the sensors and buy materials

  • 11/02/2023: Prototyping with the sensors (input signal) 

  • 11/09/2023: Start to work on the visual and audio output

  • 11/16/2023: Finalize our soft material and the functions of each button

  • 11/30/2023: Connect flex sensors, lights and buttons with the Arduino

  • 12/07/2023: Finalized the fabrication part

  • 12/14/2023: Documentation and final testing/troubleshooting.

09/14/2023

Our goals for designing this project:

  • Hints for performers during the performance 

  • Making performers perform with their comfortable expression 

  • Translating performers’ expression into all different senses

  • Convey the emotions from non-speak performers accurately

  • Translate/interpret the delay in communication between performers

Research For Inspirations

09/21/2023

Background

Based on our target user, Thomas, and the general audience, we have proposed two ideas: sign language translation and alternative emotional expression.

“Translation of Sign Language“

Miro Board: https://miro.com/app/board/uXjVMjQE1-c=/

Mind Map

Sketch

Alternative Emotional Expression

Notes from the Critic

  • Invisible and visible design

  • Thomas’s frequent language/issues

  • Pre-set device? Technical issues.

  • The generated voice is more vivid than the machinery voice.

  • How to train AI mode? 

  • Product or part of art/opera?

  • Connect to Thomas

09/28/2023

Recapping Two Ideas

Translation of Sign Language

A wearable device that generates 'voice' based on a series of inputs such as pressure, stretching, squeezing, light, distance, and more.

Takeaway from mentoring

Alternative emotional expression

To convey emotions visually or auditorily, through expressions, body language, heart rate, breathing, and so on.


Constraint of Translation

American Sign Language relies on the inclusion of facial expressions. Conveying emotions through hand sign translations alone may not yield the most accurate or complete results.

Explore opportunities beyond the body.

Position, Movement, and Relationships on Stage 

Change the Target Audience

Originally, our plan was to create customized products for the protagonist, Thomas. However, we later discovered that his father had already developed a customized system called Talksuite for him during his childhood and had also founded a company. As a result, we need to revise our current target audience or objective.

Micro & Macro Body Movement

10/05/2023

Rapid Prototype

conductive tape

double sided foam tape

cover tape

Weekly Update

  1. A meeting with Jerron Hermon (the director of the opera) will be organized for the next couple of weeks. 

  2. Paola (the composer) is also gonna get us some notes/ideas on voice and where to start, maybe with a vocabulary of words where we focus on what kind of inputs and gestures can let us work with them expressively.


Questions For the Next Meeting

  1. What’s going to happen in this Opera? Such as movement and positioning…

  2. How many people are going to use this product?

  3. What’s the maximum of the movement going to be?

Critique Feedback

Objective:

Capture the micro and macro body movement-gestures

Forms blending in the costume or stage design, how?

What data/emotion/thought brain wave gonna output?

-not quite active

-clear data?

-how to process?

What is a chocker/necklace for?

-maybe not comfortable

-what data it detects

Headset detect movement

-take the parameter we need

Physical Form is important

-appearance: looks like

-performance needs

-maybe additional decor

10/12/2023

HYPOTHESIS

An educated guess on what you think the results of the study will be. 

If we can create a wearable device that is able to recognize sign languages and analyze the emotions of the user based on an AI model, then it may help them to effectively communicate with people who do not understand sign languages. In the case of the Sensorium AI, with a pre-set movement, content, and voice output, it may allow Thomas to express the context with the sign language as triggers and translate it into voices while he is performing on the stage.

RESEARCH GOALS

  1. Help individuals with speech impairments communicate with others and express their emotions more effectively. 

  2. Specifically focus on performers in opera with speech impairments to help them express emotions effectively and timely on stage.

METHODOLOGY

Number of participants: 5
Location: NYC
Age range: not limited
Gender: not limited
Disability Identity: speech impairments
Experience: prefer people with performance experience

PROCEDURE

Data collection: Bodystorming, feasibility testing  Find more methods here
Length: approximately 30 mins
Location: Unknown

 

TESTING SCRIPT

INFORMED CONSENT

Hi, my name is <XX>. Today We would like to invite you to participate in a research study designed to explore the potential benefits of wearable devices in aiding non-verbal performers in conveying content and emotions on stage. Before you decide whether or not to participate, it is important for you to understand the purpose of the study, your role, and your rights as a participant

Please let me know if you’d like a break, have any access requirements, or need any accommodations. There are no right or wrong answers and we can stop at any point. Do I have your permission to video and audio record? Quotes will be anonymized and all data will only be shared internally with my team, stored on a secure, password-protected cloud server, and will be deleted after the study is completed.

GENERAL QUESTIONS:

  1. What’s your first impression of this wearable device?

  2. Do you feel any barriers from this device?

  3. Will anything from this device affect the costumes and stage design?

  4. What is the safest way to have this device attached to you by considering your interactions with other performers on the stage?

Card Sorting

USER TASK FEEDBACK

Mary:

It will be Cool to wear on stage.

There might be some kind of conflict between recognizing movements and the freedom of performers on the stage.

To ensure the wearable device, I might need to move in a very specific way.

Maybe the speed of movement should be considered.

Cindy:

This looks very futuristic.

There might be some unintended activation.

Nicole:

The appearance of this device kinda aligns with the story background.

The device may make the performer concerned that their movements will damage the device so that they cannot perform as well as they used to.

I am concerned about the learning curve of this device. 

Are there any tips I need to learn or remember by wearing this device?

Josh:

Will sweating and makeup influence the circuit?

Potential risk to the body’s security since it’s an electronic device.

Fanny:

It’s the solo device for each individual? The construction of relationships is inevitable on the stage/in the opera, multi-player might be the concept that you can dive in.

Edward:

I like the design because it expands the expression of dancing and movement on the stage. As a dancer who performs at large theaters, we are concerned about how to affect audiences far away from the stage. I think the design would help us express and visualize our emotions more dramatically. I am looking forward to your next step! The one suggestion I would like to make is that be sure to pay attention to the comfort as well as the beauty of design. It’s quite hard to imagine wearing a winter glove all the time while dancing. Find some breathable material and try it on yourself for several hours before you decide to use the material.

10/19/2023

Comments on the form:

It will be Cool to wear on stage.

This looks very futuristic.

The appearance of this device kinda aligns with the story background.

Potential Constraints in the Performance

There might be some kind of conflict between recognizing movements and the freedom of performers on the stage.

To ensure the wearable device, I might need to move in a very specific way.

The device may make the performer concerned that their movements will damage the device so that they cannot perform as well as they used to.

Damage Concerns

There might be some unintended activation.

Will sweating and makeup influence the circuit?

Potential risk to the body's security since it’s an electronic device.

Learning Process

I am concerned about the learning curve of this device.

Are there any tips I need to learn or remember by wearing this device?

10/26/2023

DIY Touch Sensor Test

Conductive Fiber & Fabric

We initially selected this material intending to sew the fiber into the fabric. However, we encountered difficulties sewing it onto the gloves. After discussing this with Luke, we realized that he wanted the material to facilitate interaction not only with the hands but also with the wheelchair. As a result, we decided to redesign a device that could be attached in various locations, such as the wrist or the wheelchair.

Main Materials

11/07/2023

Transfer data into visuals and sound in TouchDesigner

MaxMapper Test

Add switches and buttons to the circuit

11/14/2023

Build Up The Circuit

11/19/2023

Defined Circuit

Defined Physical Form

11/30/2023

Prototype Process Update

12/07/2023

Finalizing The Fabrication & Fitting

12/14/2023