Image display apparatus capable of recommending contents according to emotional information

• US 8,704,760 B2
• Filed: 12/03/2010
• Issued: 04/22/2014
• Est. Priority Date: 12/04/2009
• Status: Active Grant

• Current Assignee: LG Electronics, Inc. (LG Corporation)
• Original Assignee: LG Electronics, Inc. (LG Corporation)
• Inventors: Kang, Mingoo, Kang, Haengjoon, Hwang, Sunjung, Seo, Youngjae, Park, Jongsoon, Park, Jinyung, Kim, Jongchul, Park, Junho
• Primary Examiner(s): Sitta, Grant
• Application Number:
  US12/959,557
• Publication Number:
  US 20110134026A1
• Time in Patent Office: 1,236 Days
• Field of Search: 725/46, 345/156
• US Class Current: 345/156
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: H04N 21/4826 using recommendation lists, e.g. of programs or channels sorted out according to their score

Abstract

In accordance with an aspect of the present invention, a method for controlling an image display apparatus includes obtaining, by the image display apparatus, emotion information associated with a user of the image display apparatus, determining a contents to be recommended for the user among a plurality of different contents, based on the obtained emotion information associated with the user, and selectively or automatically reproducing one of the recommended contents at the image display apparatus.

Claims

1. A method for controlling an image display apparatus, the method comprising:

obtaining voice information via a voice recorder of the image display apparatus;

obtaining physical information via a body signal sensor of the image display apparatus;

obtaining, via an image capturing unit of the image display apparatus, image information by capturing via a depth camera included in the image capturing unit, a particular user'"'"'s face, extracting coordinates of the captured user'"'"'s face, and capturing via an RGB camera included in the image capturing unit, facial expressions based on the extracted coordinates, the image information being extracted from the captured facial expressions;

respectively storing, in a database of the image display apparatus, the image information, the voice information, and the physical information for a plurality of users including the particular user;

storing, in the database, emotional information for the plurality of users, respectively, the emotional information including content categories preferred by each of the plurality of users;

determining an emotional state of the particular user using the image information extracted from the captured facial expressions, the voice information, and the physical information of the particular user, the emotional information associated with the particular user including the determined emotional state;

determining a content to be recommended for the particular user from among a plurality of different contents, based on both the determined emotional state and the preferred content categories associated with the particular user; and

selectively or automatically displaying, on a display screen of the image display apparatus, the recommended content that is based on both the determined emotional state and the preferred content categories associated with the particular user.

2. The method of claim 1, wherein the displaying step selectively displays, on the display screen, multiple recommended contents based on a user input.

3. The method of claim 1, wherein the determining the emotional state includes:

assigning different weights to the obtained image information, voice information, and physical information depending on attributes of the particular user or a preset condition; and

combining the different weights assigned to the obtained image information, voice information, and physical information.

4. The method of claim 1, further comprising:

displaying, on the display screen, a list of multiple recommended contents.

5. The method of claim 1, further comprising:

displaying, on the display screen, thumbnails of the plurality of different contents simultaneously with a content-related image.

17. The method of claim 3, wherein the weights assigned to the obtained voice information are less than the weights assigned to the obtained image information.

6. A method for controlling an image display apparatus, the method comprising:

obtaining voice information via a voice recorder of the image display apparatus;

obtaining physical information via a body signal sensor of the image display apparatus;

obtaining, via an image capturing unit of the image display apparatus, image information by capturing via a depth camera included in the image capturing unit, a particular user'"'"'s face, extracting coordinates of the captured user'"'"'s face, and capturing via an RGB camera included in the image capturing unit, facial expressions based on the extracted coordinates, the image information being extracted from the captured facial expressions;

respectively storing, in a database of the image display apparatus, the image information, the voice information, and the physical information for a plurality of users including the particular user;

storing, in the database, emotional information for the plurality of users, respectively, the emotional information including content categories preferred by each of the plurality of users;

determining an emotional state of the particular user using the image information extracted from the captured facial expressions, the voice information, and the physical information of the particular user, the emotional information associated with the particular user including the emotional state;

displaying, on a display screen of the image display apparatus of the particular user, one or more instances of emotional information for one or more users at one or more other image display apparatuses;

receiving, at the one or more other image display apparatuses, a first recommendation of the particular user for a recommended content among a plurality of different contents based on both the determined emotional state and the preferred content categories associated with the particular user; and

transmitting, to another apparatus, the first recommendation.

7. The method of claim 6, further comprising:

transmitting, to another user among the one or more users at the one or more other image display apparatuses, at least one instance of current emotional information of the particular user;

receiving, at the image display apparatus of the particular user, a second recommendation of the another user for another recommended content among the plurality of different contents based on the transmitted at least one instance of current emotional information of the particular user; and

displaying, on the display screen, the second recommendation.

8. The method of claim 6, further comprising:

displaying, on the display screen of the image display apparatus of the particular user, a thumbnail list of the plurality of different contents.

9. The method of claim 6, further comprising:

transmitting, to a content provider, at least one instance of current emotional information of the particular user;

receiving, at the image display apparatus from the content provider, a third recommendation for a third content among the plurality of different contents based on the transmitted at least one instance of current emotional information of the particular user; and

displaying, on the display screen, the third recommendation.

10. A method for controlling an image display apparatus, the method comprising:

obtaining voice information via a voice recorder of the image display apparatus;

obtaining physical information via a body signal sensor of the image display apparatus;

obtaining, via an image capturing unit of the image display apparatus, image information by capturing via a depth camera included in the image capturing unit, a particular user'"'"'s face, extracting coordinates of the captured user'"'"'s face, and capturing via an RGB camera included in the image capturing unit, facial expressions based on the extracted coordinates, the image information being extracted from the captured facial expressions;

respectively storing, in a database of the image display apparatus, the image information, the voice information, and the physical information for a plurality of users including the particular user;

storing, in the database, emotional information for the plurality of users, respectively, the emotional information including content categories preferred by each of the plurality of users;

determining an emotional state of the particular user using the image information extracted from the captured facial expressions, the voice information, and the physical information of the particular user, the emotional information associated with the particular user including the emotional state;

determining at least one recommended broadcast program for the particular user among a plurality of different broadcast programs, based on the determined emotional state and the preferred content categories associated with the particular user;

displaying, on a display screen of the image display apparatus, the plurality of different broadcast programs differentiating the at least one recommended broadcast program that is based on the determined emotional state and the preferred content categories associated with the particular user; and

selectively or automatically displaying, on the display screen, the at least one recommended broadcast program that is based on the determined emotional state and the preferred content categories associated with the particular user.

11. The method of claim 10, wherein each of the plurality of different broadcast programs includes at least one of an image object representing a specific emotion for each of the plurality of broadcast programs and a text object indicating at least one of the specific emotion, an attention level, and an emotional intensity level.

12. The method of claim 11, wherein the plurality of different broadcast programs are displayed using an electronic program guide (EPG), and the at least one recommended broadcast program is highlighted or focused on in a different size, brightness or color in comparison to other broadcast programs.

13. The method of claim 11, wherein the plurality of different broadcast programs are displayed using a thumbnail list, and the thumbnail list includes thumbnail images respectively corresponding to the plurality of different programs on the display screen of the image display apparatus or on a subdisplay, and

wherein the at least one recommended broadcast program is highlighted or focused on in a different size, brightness or color in comparison to other broadcast programs.

14. The method of claim 13, wherein the subdisplay is distinguished from the display screen of the image display apparatus physically or in hardware, communicates with the display screen wirelessly or by cable, and displays at least one of a subtitle, a channel list, and additional information of the thumbnail list, and

wherein the display screen of the image display apparatus displays a reproduced image of the at least one recommended broadcast program.

15. The method of claim 13, wherein the thumbnail images correspond to a content part that elicits a strongest emotional response from the particular user.

16. The method of claim 10, further comprising:

determining an emotional state of the plurality of users, and

assigning different weights to each of the plurality of users according to the content categories preferred by each of the plurality of users.

18. An image display apparatus comprising:

a display screen;

a voice recorder configured to obtain voice information;

a body signal sensor configured to obtain physical information;

an image capturing unit configured to obtain image information by capturing via a depth camera included in the image capturing unit, a particular user'"'"'s face, extract coordinates of the captured user'"'"'s face, and capture via an RGB camera included in the image capturing unit, facial expressions based on the extracted coordinates, the image information being extracted from the captured facial expressions;

a memory configured to respectively store the image information, the voice information, and the physical information for a plurality of users including the particular user, the memory being configured to store emotional information for the plurality of users, respectively, the emotional information including content categories preferred by each of the plurality of users; and

a controller configured to;

determine an emotional state of the particular user using the image information extracted from the captured facial expressions, the voice information, and the physical information of the particular user, the emotional information associated with the particular user including the determined emotional state,

determine a content to be recommended for the particular user from among a plurality of different contents, based on both the determined emotional state and the preferred content categories associated with the particular user, and

selectively or automatically display, on the display screen, the recommended content that is based on both the determined emotional state and the preferred content categories associated with the particular user.