AUTOMATED IMAGE CAPTURE BASED ON EMOTION DETECTION

• US 20200134296A1
• Filed: 10/25/2018
• Published: 04/30/2020
• Est. Priority Date: 10/25/2018
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Inc.
• Inventors: Rajvanshi, Tushar, Gupta, Sourabh, Bedi, Ajay
• Granted Patent: US 10,755,087 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: H04M 1/724 User interfaces specially adapted for cordless or mobile telephones
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/633 for displaying additional information relating to control or operation of the camera
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image

Abstract

Methods and systems are provided for performing automated capture of images based on emotion detection. In embodiments, a selection of an emotion class from among a set of emotion classes presented, via a graphical user interface, is received. The emotion class can indicate an emotion exhibited by a subject desired to be captured in an image. A set of images corresponding with a video is analyzed to identify at least one image in which a subject exhibits the emotion associated with the selected emotion class. The set of images can be analyzed using at least one neural network that classifies images in association with emotion exhibited in the images. Thereafter, the image can be presented in association with the selected emotion class.

Claims

1. A computer-implemented method for automated image capture based on emotion detection, the method comprising:

receiving a selection of an emotion class from among a set of emotion classes presented via a graphical user interface, the emotion class indicating an emotion exhibited by a subject desired to be captured in an image;

analyzing a set of images corresponding with a video to identify at least one image in which a subject exhibits the emotion associated with the selected emotion class, wherein the set of images are analyzed using at least one neural network that classifies images in association with emotion exhibited in the images; and

providing the at least one image in association with the selected emotion class.

2. The computer-implemented method of claim 1, wherein the emotion class comprises a type of emotion.

3. The computer-implemented method of claim 1, wherein the emotion class comprises a type of emotion, an emotion level that indicates an extent of a particular type of emotion, or a combination thereof.

4. The computer-implemented method of claim 1, wherein the emotion class is selected by a user via the graphical user interface.

5. The computer-implemented method of claim 1, wherein the set of images are analyzed in real time as the video is streamed.

6. The computer-implemented method of claim 1, wherein the at least one neural network comprises a neural network that analyzes facial expression and body expression exhibited within images.

7. The computer-implemented method of claim 1, wherein the at least one image is provided presentation in association with the selected emotion class.

8. The computer-implemented method of claim 1, wherein the at least one neural network outputs an emotion class and corresponding emotion score for each image of the set images.

9. The computer-implemented method of claim 8, wherein the at least one image in which a subject exhibits the emotion is identified using the emotion classes and corresponding emotion scores output via the at least one neural network.

10. The computer-implemented method of claim 1, wherein the at least one neural network comprises a facial neural network that analyzes the set of images based on facial expression and a body neural network that analyzes the set of images based on facial and body expressions.

11. One or more computer-readable media having a plurality of executable instructions embodied thereon, which, when executed by one or more processors, cause the one or more processors to perform a method for automated image capture based on emotion detection, the method comprising:

obtaining a set of images for which to identify at least one image in which a subject exhibits a particular emotion class;

classifying each image of the set of images into an emotion class based on the emotion exhibited in the corresponding image using at least one neural network, the at least one neural network classifying each image based on facial expression and body expression exhibited in the corresponding image; and

selecting one or more images classified into the particular emotion class for presentation, wherein the one or more images classified into the particular emotion class are selected based on emotion scores output by the at least one neural network in accordance with the classification.

12. The media of claim 11, wherein the particular emotion class is selected via a graphical user interface from among a set of emotion classes.

13. The media of claim 12, wherein the particular emotion class comprises a type of emotion, an emotion level that indicates an extent of a particular type of emotion, or a combination thereof.

14. The media of claim 11, wherein the at least one neural network comprises a first neural network that classifies images based on facial expression and a second neural network that classifies images based on facial expression and body expression.

15. The media of claim 14, wherein classification comprises:

using the first neural network to identify a first emotion class and corresponding first emotion score for each image, and

when a first emotion score falls below a threshold for a particular image, using the second neural network to identify a second emotion class and corresponding second emotion score for the particular image.

16. The media of claim 15, wherein when the first neural network and second neural network are used to generate the first emotion score and the second emotion score, generating a combined emotion score using the first emotion score and the second emotion score, the combined emotion score being used in selecting the one or more images classified into the particular emotion class for presentation.

17. A computing system comprising:

one or more processors; and

one or more non-transitory computer-readable storage media, coupled with the one or more processors, having instructions stored thereon, which, when executed by the one or more processors, cause the computing system to provide;

means for training a neural network system, wherein the neural network system includes a first neural network that classifies images as exhibiting an emotion based on facial expression and body expression; and

means for using the neural network system to identify an image within a set of images in which a subject exhibits a particular emotion class.

18. The system of claim 17, wherein the neural network system includes a second neural network that classifies images as exhibiting an emotion based only on facial expressions.

19. The system of claim 18, wherein the neural network system is used to identify the image in which the subject exhibits the particular emotion class by:

inputting the image into the second neural network to obtain a first emotion class and a corresponding first emotion score;

determining that the first emotion score falls below a threshold value;

based on the first emotion score falling below the threshold value, inputting the image into the second neural network to obtain a second emotion class and a corresponding second emotion score; and

generating a combined emotion score using the first emotion score and the second emotion score, the combined emotion score being used to identify the image as exhibiting the particular emotion class.

20. The system of claim 17 further comprising means for causing presentation of the image in association with the particular emotion class.