EmoSynth
EmoSynth: A combination of Emotion and Synthesizer, highlighting the device’s ability to synthesize emotional expressions.
Sensorium AI x Ability Project
Junru Chen Lang Qin Emily Lei Kassia Zheng Muqing Wang
We are Sensorium AI Group from the Multisensory class. We designed a wearable device that works with Sensorium Ex, Luke, Michael, and our professor, Lauren. To fulfill the design objectives of facilitating real-time artistic expression and modulation for performers with cerebral palsy and/or limited speech, our device allows the user to freely select the emotion and level of emotion they want to express. Additionally, there is a selection of high and low audio to represent diverse genders, along with a reset button to prevent accidental touches during movement on stage. Each selection on the device is denoted by an embedded LED hint to better inform the users.
wear on the arm
wear on the chair ( wheelchair)
tactile soft buttons and the touch bar
EmoSynth with its adjustable add-on piece
Timeline
09/14/2023: Primary research ( Reference cases and Inspiration)
09/21/2023: Ideas brainstorm and sketch
09/28/2023: Finalized our project idea
10/05/2023:Developed first quick prototype
10/12/2023: User testing with the quick prototype
10/19/2023: Implementation of user testing result
10/26/2023: Finalize the sensors and buy materials
11/02/2023: Prototyping with the sensors (input signal)
11/09/2023: Start to work on the visual and audio output
11/16/2023: Finalize our soft material and the functions of each button
11/30/2023: Connect flex sensors, lights and buttons with the Arduino
12/07/2023: Finalized the fabrication part
12/14/2023: Documentation and final testing/troubleshooting.
09/14/2023
Our goals for designing this project:
Hints for performers during the performance
Making performers perform with their comfortable expression
Translating performers’ expression into all different senses
Convey the emotions from non-speak performers accurately
Translate/interpret the delay in communication between performers
Research For Inspirations
Reference
Inspiration 1: http://xhslink.com/6NqCyu
Inspiration 2: https://www.media.mit.edu/projects/alterego/overview/
Inspiration 3: https://www.deafwest.org/
Other Possibilities:
Emotion AI: https://www.affectiva.com/about-affectiva/
EMOTIC Dataset- Context Analysis: https://s3.sunai.uoc.edu/emotic/index.html
Brainwave: https://prezi.com/p/fvxrqad30mjq/music-from-brainwaves/
09/21/2023
Background
Based on our target user, Thomas, and the general audience, we have proposed two ideas: sign language translation and alternative emotional expression.
“Translation of Sign Language“
Miro Board: https://miro.com/app/board/uXjVMjQE1-c=/
Mind Map
Sketch
Alternative Emotional Expression
Notes from the Critic
Invisible and visible design
Thomas’s frequent language/issues
Pre-set device? Technical issues.
The generated voice is more vivid than the machinery voice.
How to train AI mode?
Product or part of art/opera?
Connect to Thomas
09/28/2023
Recapping Two Ideas
Translation of Sign Language
A wearable device that generates 'voice' based on a series of inputs such as pressure, stretching, squeezing, light, distance, and more.
Takeaway from mentoring
Alternative emotional expression
To convey emotions visually or auditorily, through expressions, body language, heart rate, breathing, and so on.
Constraint of Translation
American Sign Language relies on the inclusion of facial expressions. Conveying emotions through hand sign translations alone may not yield the most accurate or complete results.
Explore opportunities beyond the body.
Position, Movement, and Relationships on Stage
Change the Target Audience
Originally, our plan was to create customized products for the protagonist, Thomas. However, we later discovered that his father had already developed a customized system called Talksuite for him during his childhood and had also founded a company. As a result, we need to revise our current target audience or objective.
Micro & Macro Body Movement
10/05/2023
Rapid Prototype
conductive tape
double sided foam tape
cover tape
Weekly Update
A meeting with Jerron Hermon (the director of the opera) will be organized for the next couple of weeks.
Paola (the composer) is also gonna get us some notes/ideas on voice and where to start, maybe with a vocabulary of words where we focus on what kind of inputs and gestures can let us work with them expressively.
Questions For the Next Meeting
What’s going to happen in this Opera? Such as movement and positioning…
How many people are going to use this product?
What’s the maximum of the movement going to be?
Critique Feedback
Objective:
Capture the micro and macro body movement-gestures
Forms blending in the costume or stage design, how?
What data/emotion/thought brain wave gonna output?
-not quite active
-clear data?
-how to process?
What is a chocker/necklace for?
-maybe not comfortable
-what data it detects
Headset detect movement
-take the parameter we need
Physical Form is important
-appearance: looks like
-performance needs
-maybe additional decor
10/12/2023
HYPOTHESIS
An educated guess on what you think the results of the study will be.
If we can create a wearable device that is able to recognize sign languages and analyze the emotions of the user based on an AI model, then it may help them to effectively communicate with people who do not understand sign languages. In the case of the Sensorium AI, with a pre-set movement, content, and voice output, it may allow Thomas to express the context with the sign language as triggers and translate it into voices while he is performing on the stage.
RESEARCH GOALS
Help individuals with speech impairments communicate with others and express their emotions more effectively.
Specifically focus on performers in opera with speech impairments to help them express emotions effectively and timely on stage.
METHODOLOGY
Number of participants: 5
Location: NYC
Age range: not limited
Gender: not limited
Disability Identity: speech impairments
Experience: prefer people with performance experience
PROCEDURE
Data collection: Bodystorming, feasibility testing Find more methods here
Length: approximately 30 mins
Location: Unknown
TESTING SCRIPT
INFORMED CONSENT
Hi, my name is <XX>. Today We would like to invite you to participate in a research study designed to explore the potential benefits of wearable devices in aiding non-verbal performers in conveying content and emotions on stage. Before you decide whether or not to participate, it is important for you to understand the purpose of the study, your role, and your rights as a participant
Please let me know if you’d like a break, have any access requirements, or need any accommodations. There are no right or wrong answers and we can stop at any point. Do I have your permission to video and audio record? Quotes will be anonymized and all data will only be shared internally with my team, stored on a secure, password-protected cloud server, and will be deleted after the study is completed.
GENERAL QUESTIONS:
What’s your first impression of this wearable device?
Do you feel any barriers from this device?
Will anything from this device affect the costumes and stage design?
What is the safest way to have this device attached to you by considering your interactions with other performers on the stage?
Card Sorting
USER TASK FEEDBACK
Mary:
It will be Cool to wear on stage.
There might be some kind of conflict between recognizing movements and the freedom of performers on the stage.
To ensure the wearable device, I might need to move in a very specific way.
Maybe the speed of movement should be considered.
Cindy:
This looks very futuristic.
There might be some unintended activation.
Nicole:
The appearance of this device kinda aligns with the story background.
The device may make the performer concerned that their movements will damage the device so that they cannot perform as well as they used to.
I am concerned about the learning curve of this device.
Are there any tips I need to learn or remember by wearing this device?
Josh:
Will sweating and makeup influence the circuit?
Potential risk to the body’s security since it’s an electronic device.
Fanny:
It’s the solo device for each individual? The construction of relationships is inevitable on the stage/in the opera, multi-player might be the concept that you can dive in.
Edward:
I like the design because it expands the expression of dancing and movement on the stage. As a dancer who performs at large theaters, we are concerned about how to affect audiences far away from the stage. I think the design would help us express and visualize our emotions more dramatically. I am looking forward to your next step! The one suggestion I would like to make is that be sure to pay attention to the comfort as well as the beauty of design. It’s quite hard to imagine wearing a winter glove all the time while dancing. Find some breathable material and try it on yourself for several hours before you decide to use the material.
10/19/2023
Comments on the form:
It will be Cool to wear on stage.
This looks very futuristic.
The appearance of this device kinda aligns with the story background.
Potential Constraints in the Performance
There might be some kind of conflict between recognizing movements and the freedom of performers on the stage.
To ensure the wearable device, I might need to move in a very specific way.
The device may make the performer concerned that their movements will damage the device so that they cannot perform as well as they used to.
Damage Concerns
There might be some unintended activation.
Will sweating and makeup influence the circuit?
Potential risk to the body's security since it’s an electronic device.
Learning Process
I am concerned about the learning curve of this device.
Are there any tips I need to learn or remember by wearing this device?
10/26/2023
Main Components in the circuit:
Main Component in the circuit
lock/unlock
Reset
Reference link for prototype: Physical computing project from another website
DIY Touch Sensor Test
Conductive Fiber & Fabric
We initially selected this material intending to sew the fiber into the fabric. However, we encountered difficulties sewing it onto the gloves. After discussing this with Luke, we realized that he wanted the material to facilitate interaction not only with the hands but also with the wheelchair. As a result, we decided to redesign a device that could be attached in various locations, such as the wrist or the wheelchair.
Main Materials
11/07/2023
Transfer data into visuals and sound in TouchDesigner
MaxMapper Test
Add switches and buttons to the circuit
11/14/2023
Build Up The Circuit
11/19/2023
Defined Circuit
Defined Physical Form
11/30/2023
Prototype Process Update
12/07/2023
Finalizing The Fabrication & Fitting