Read your audience

Case Study • Research & Design • Sweden 2021
My Role:

Research, Ideation, Prototyping, development, Testing, Writing

Year:

2021

Team:

Solo project

Time:

In progress

A project about an opportunity to make online presentations more intuitive with the help of Computer Vision and AI.

Like most people, I enjoy getting instant feedback when presenting to an audience. So, presenting to a screen, for the past year has been challenging as it is less engaging and harder to read the audience’s emotions.
It made me wonder if there is not a way of making presenting online more engaging and how I could read people’s reactions to my presentation even better while being stuck behind my screen.
As a result, I spent the last few weeks experimenting with how I could make emotions visible for myself so that I could feel the same way I did when demoing my work in front of real people.

Read on Medium

The problem

My initial thoughts were that because the audience was not visible to me I could not empathize with the audience as well as I would in a real-life setting. But then I realized, we need to communicate in a different way. The challenge, therefore, lies in how to get the audience to participate more during a demo.

Design Process

The design process captures the steps that led to the final solution.

Design process overview

Research

For the research face, I studied different concepts about what emotions are and how they can be interpreted by face recognition. More details can be found in the Medium article.

Ideation

An affinity diagram was created during the research phase to organize the findings. Using the affinity diagram, I generated ideas for each category and selected one that I found most appealing. My favorite was "making recommendations based on emotions". Therefore, I decided to explore this further.

Interaction on happiness

If the listeners indicate happiness for a brief period of time, the prototype reacts with an overlay of recommendations from which the user can pick to convey their sentiments to the presenter. For example clapping hands, a smile, and more.

Recommendations what a user can do while expressing happiness

Interaction on disapproval

As mentioned previously, disapproval is a combination of two basic emotions, and people experience it when they lack approval or believe that someone or something is terrible or wrong.
If during a presentation suddenly something becomes unclear, I realized it would be extremely beneficial to either ask a question there and then or make a note to discuss it later on. The function of being able to save a question and then pick it up at the end of the presentation has proven to be incredibly beneficial. In comparison, for example, in other video conferencing tools you have to scroll back in the group chat to find your original comment. The presenter’s speech is tracked with the help of AI any signs of disapproval are recognized and converted into a series of questions. These questions can then be forwarded to the presenter in the group chat, therefore once again enhancing the interaction between the audience and the presenter.

Recommendations on what to do if a user expresses disapproval.

Learnings

  • Using Computer vision to react to emotions is extremely beneficial for both the presenter and the audience.
    But the overall interaction needs to be subtle and not dominant in the presentation.
  • Subconscious interaction over emotions can enhance video calls and can give the presenter more confidence and make the audience feel included.
    Interaction based on emotions can be used to educate people about various functionalities.

Limitations

  • Emotional tracking could be used to urge users to participate more as the program initiates a dialog with the users through provided interactions.
  • Because facial expressions can be faked, emotional tracking on a single sensor has a medium/low accuracy, necessitating the usage of multiple sensors for improved accuracy.

Next steps

I believe it is very important for a designer to create inclusive technology. For this reason, I ask myself: “Does this technology work equally well for different genders and skin colors?”. Therefore this is something I am currently putting a lot of energy into.
As a foreigner, I have first-hand experience of how different cultures express emotions, therefore I will try to research precisely what are the differences and how to create an equally good experience through all cultures.