AUTOMATICALLY COMPUTING EMOTIONS AROUSED FROM IMAGES THROUGH SHAPE MODELING

• US 20170109603A1
• Filed: 12/30/2016
• Published: 04/20/2017
• Est. Priority Date: 08/16/2012
• Status: Active Grant

• Current Assignee: Penn State Research Foundation
• Original Assignee: Penn State Research Foundation
• Inventors: Adams, Jr., Reginald B., Wang, James Z., Lu, Xin, Suryanarayan, Poonam, Li, Jia, Newman, Michelle
• Granted Patent: US 9,904,869 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 11/206 Drawing of charts or graphs
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 40/174 Facial expression recognition

Abstract

Shape features in natural images influence emotions aroused in human beings. An in-depth statistical analysis helps to understand the relationship between shapes and emotions. Through experimental results on the International Affective Picture System (IAPS) dataset, evidence is presented as to the significance of roundness-angularity and simplicity-complexity on predicting emotional content in images. Shape features are combined with other state-of-the-art features to show a gain in prediction and classification accuracy. Emotions are modeled from a dimensional perspective in order to predict valence and arousal ratings, which have advantages over modeling the traditional discrete emotional categories. Images are distinguished vis-a-vis strong emotional content from emotionally neutral images with high accuracy. All of the methods and steps disclosed herein are implemented on a programmed digital computer, which may be a stand-alone machine or integrated into another piece of equipment such as a digital still or video camera including, in all embodiments, portable devices such as smart phones.

Claims

1. A system for analyzing the emotional content of an image, comprising the steps of:

a programmed computer having an input for receiving at least one digital image, wherein the computer is programmed to perform the following functions automatically;

(a) extract visual information from the image;

(b) model shape features in the image, and

(c) compute the emotional content of the image based upon the extracted visual features and modeled shape features; and

an output device in communication with the programmed computer enabling a user to receive the computed emotional content of the image.

2. The system of claim 1, wherein:

the programmed computer also receives non-visual information about the image; and

the computer is further operative to automatically output information regarding the emotional content of the image based upon the modeled shape features and one or both of the extracted visual features and non-visual information.

3. The system of claim 2, wherein:

the programmed computer forms part of a camera; and

the non-visual information includes camera setting parameters.

4. The method of claim 2, further including:

an input for receiving textual information about the image; and

wherein the non-visual information includes the textual information about the image.

5. The system of claim 1, wherein the input is configured to receive a still digital image or a plurality of sequential digital images representing frames in a video or movie.

6. The system of claim 1, wherein the computer is further programmed to automatically perform the following functions:

use the modeled shape features to determine valence or arousal coordinates based upon a dimensional space model of emotions with valence and arousal coordinates; and

compute the emotional content of the image based upon the valence or arousal coordinates of the image in the space model.

7. The system of claim 1, wherein the computer is further programmed to automatically use the shape features to distinguished images with strong emotional content from emotionally neutral images.

8. The system of claim 1, wherein the computer is further programmed to automatically perform the following functions:

model the visual properties of roundness, angularity and simplicity based upon on one or more shapes extracted from the image; and

compute the emotional content based upon the roundness, angularity or simplicity.

9. The system of claim 1, wherein the computer is further programmed to automatically perform the following functions:

identify contours in the image;

represent the contours as lines and curves;

perform statistical analyses on the lines and curves to model the visual properties of roundness, angularity, and simplicity; and

compute the emotional content based upon the roundness, angularity and simplicity.

10. The system of claim 9, wherein the computer is programmed to determine if the lines or curves are locally meaningful.

11. The system of claim 9, wherein the computer is programmed to determine if the contours are strong or continuous.