Design and Results of our first Study

Since 2016 we are performing research on analysing textual user feedback in order to enable product owners and quality manager to put current research approaches into practise.

Most methods at least partially rely on quantitative measures like ratings or votes. While this is working so some extend within app stores, where ratings are present, it is not possible anymore when analzying social media feedback.

In addition, emotion is one of the key aspects of user experience. Which is not captured by most research approaches.

The need for suitable metrics for social media analyses as well as the importance of emotion leads us to investigate emojis. In the beginning of 2016 we had the vision to make use of emojis to detect sentiment and emotions within user feedback.

This lead to two leading question:

„Are Emojis perceived homogenously among people?“

„How are emojis linked to real emotions?“

History and Development of Emojis

The question who designed the popular smiley face first is still not fully clarified. A graphic artist from Massachusetts designed 1963 a yellow smiley for a company to raise the employs motivation. The designer Harvey Ball earned 45$ for his work. Almost ten years later, a French journalist named Franklin Loufrani registered the trademark “Smiley” for commercial use. In 2016 the company had a revenue of 185 Mio. €. [1]

The emoji-trend in the digital world started in 1999 with Shigetaka Kuritas 176 pictograms. In 2017 the pictograms were exhibited in the Museum of Modern Art in Ney York City (picture below). The name of these 12 x 12 pixel pictograms is a compilation of the two Japanese words “e” which means “picture” and “moji” which means “character”.

https://www.moma.org/calendar/exhibitions/3639

However, this was just the start of the meteoric career of the emojis. In 2011 the Standard Emoji keyboard arrived to iOS 5. 2015 the Oxford Dictionary announced for the first time a pictograph ‘Word of the year’: “face with tears of joy”. Nowadays billions of emojis are used in almost every communication channel and around the world. E.g. 5.1 billion emojis are sent via Facebook messenger every day.

Performing our First Survey on the Perception of Emojis

As we want to analyze emotions in reviews by analyze the emojis in these reviews, we have to link emojis to emotions first. Thera are no empiric data of the perceived emotions of each emoji by various users. For this purpose, we performed a survey to map emojis to an emotion model. You can see an overview of the sequence of our survey in the following flow chart:

Analyse existing emotion models
Build an emotion model in the context of emojis in product reviews

Collect emojis from Unicode
Collect emojis from different keyboards

Cluster similar emojis into groups
Validate cluster and groups

Create questionnaires
Select particpants
Collect results

Emotion Model used for the Survey

To map emotions to the emojis in a consistent way we needed an emotion model. It shouldn’t be too detailed and difficult to understand for users from various domains. Therefor it was no option to use existing psychological models e.g. by Ekman or Plutchik. We decided to build our own model, inspired by the existing ones and other sources like facebook reactions or IBM Watson (Watson natural language).

The emotion model (Fig.??) has two dimensions. The first one is the dimension of sentiment with an ordinal scale between positive, neutral and negative. The second one is the dimension of emotions. Sentiments act as a major category and the emotions as a subcategory so we can combine sentiment analysis with more detailed emotion analysis.

Collect emojis for analysis

The next step was to collect all emojis we wanted to classify. We considered emojis as well as emoticons, which are textual representations of an emotion or a gesture like =) or ;-). To gather emojis that are relevant for our study, we looked at the Unicode Standard for emojis. The figure below shows an excerpt of the group “person raising hand” with different designs from various providers

As there are more than 3000 emojis available, it was necessary to reduce them. In the first step we excluded emojis representing objects or activities such as: , , , , and several hundred flags. However, there are some objects that may be used to express emotions like: , , , , which we did not excluded

After the reduction we had in total 612 emojis. Considering this list, it became obvious that many emojis had the same or a similar meaning. So we grouped in the second step similar emojis or different variations together. We then choose a representative for every group. This process led to 99 groups.

Study Design

In the survey we applied the emotion model to every representative. Participants were ask to assign each emoji a sentiment and an emotion from the model. The participants of the survey were different ages, both gender and various levels of familiarity with the usage of emojis. We gave the questionnaire to people we know like friends, coworkers and friends of coworkers. In a pre-study with 11 participants we saw that 18 emojis (Figure below) were perceived in terms of sentiment as well as emotion in the same way by all 11 participants. To shorten the survey we excluded these 18 emojis from the survey. 96 participants completed the shorter questionnaire.

Results

We analyzed responses from 107 participants with a social generation distribution and level of familiarity you can see in de figure below. Most participants had western European background.

Considering the distribution of emoji categorizations into sentiments the participants agreed with 71 votes (67% of all participants) on each other. The median value was 73. This shows that most of the emojis were classified quite clearly with respect to the sentiment.

In our analysis, we selected the sentiment for an emoji if at least half of the responses gave the same result. With respect to this requirement we could sort 591 out of 612 emojis into a sentiment. Furthermore we were able to categorize 485 emojis with an agreement of 70% or more. You can see an overview of the classification of the different representatives in the following table:

Emojis classified according to sentiment

Another finding was that most emojis, which we can categorize according to the sentiment, also had a clear meaning in terms of emotion. Emojis were categorized into an emotion if the agreement was at least 50% as well. In this way we could classify 512 emojis from the 591 that we classified into a sentiment. You can see an overview of the classification of the different representatives in the following table:

Emojis classified according to emotion

We also performed an analysis regarding how the different groups (familiarity, gender, social generation) rated the emojis. In general, we did not see huge differences between those participants. In the spider charts below you can see the distributions of the different groups:

Distribution of sentiment

Emotions by familiarity

Distribution of sentiment by gender

Distribution of emotions by gender

Distribution of sentiment by generation

Distribution of emotions by generation

The data obtained from our survey shows that most emojis that were part of our study could be mapped very well to a sentiment and an emotion. In conclusion we can answer both of our leading questions from above for 512 of the 612 collected emojis: These emojis are perceived homogenously among people and we could link them to real emotions form our emotion model.

Implications of our first study

For companies developing mobile apps, there are increasing requirements such as short time to market or high quality. On the other hand users of such app do have more and more influence on the success of apps, as they can easily provide feedback nowadays via different channels (such as the Apple App Store or social media) or delete the app. Up to now, analyzing textual user feedback had the challenge that an automation is only possible in a non-satisfiable way due to the limitations of natural language understanding.

The results of our survey enables us to analyze larger sets of texts in terms of emotions in a quick way. Furthermore, this can basically be done for any textual data source as emojis are widely spread. So the results provide the opportunity to analyze the users’ perspective on the product when the traditional five-star rating scale is not available (e.g., this is typical for social media content).

In another study we measured and explored the correlation between emojis and star ratings for mobile apps based on around 180,000 user reviews. We found a strong correlation between star ratings and emojis. The finding indicates that product reviews can be compared to other types of user feedback as long as the feedback contains emojis.

Composition of emotional sentiment in star ratings

Du willst uns helfen diese Studie fort zu führen?

Dann Teile unsere Umfrage doch mit deinen Freunden!

[Sassy_Social_Share]

Wir nutzen Emojis von www.emojione.com