Video indexing based on viewers' behavior and emotion feedback

• US 6,585,521 B1
• Filed: 12/21/2001
• Issued: 07/01/2003
• Est. Priority Date: 12/21/2001
• Status: Expired due to Fees

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Obrador, Pere
• Primary Examiner(s): Rovnak, John Edmund
• Application Number:
  US10/023,810
• Publication Number:
  US 20030118974A1
• Time in Patent Office: 557 Days
• Field of Search: 434/236, 434/237, 434/238, 345/721, 369/24.01, 369/27.01, 369/98, 725/10, 725/12
• US Class Current: 434/236
• CPC Class Codes: G09B 23/28 for medicine

Abstract

A method and a corresponding apparatus for video indexing based on viewers'"'"' behavior and emotion feedback to automatically index a multimedia sequence using viewers'"'"' casual feedback while watching the multimedia sequence. The method and apparatus use a system, such as an emotion detection system that includes one or more multimedia acquisition devices, such as video cameras, to detect viewers'"'"' behavior and emotion feedback, and to produce as video indices various categories of behavior and emotion segments. Each video camera may include one or more image sensors and/or audio sensors. The viewers or other users, i.e., user, may later selectively view the multimedia sequence by browsing through the video indices of various categories of emotion. The video indices are similar to chapters within digital video disc (DVD) movies that enable a user to jump directly to a particular chapter without having to fast forward.

Claims

1. A method for video indexing based on viewers'"'"' behavior and emotion feedback, comprising:

monitoring one or more viewers during a multimedia sequence play back;

detecting the one or more viewers'"'"' behavior and emotion feedback;

categorizing the one or more viewers'"'"' behavior and emotion feedback as a plurality of categories of behavior and emotion segments; and

indexing the multimedia sequence using the plurality of categories of behavior and emotion segments,

wherein a user can selectively view the multimedia sequence using the plurality of categories of behavior and emotion segments as video indices.

2. The method of claim 1, wherein the detecting step includes detecting facial expression of the one or more viewers.

3. The method of claim 1, wherein the detecting step includes detecting physical movement of the one or more viewers.

4. The method of claim 1, wherein the detecting step includes detecting audio signal in the one or more viewers'"'"' voice.

5. The method of claim 4, further comprising inserting text annotations corresponding to the audio signal.

6. The method of claim 1, wherein the categorizing step includes categorizing the one or more viewers'"'"' behavior and emotion feedback as one or more of “

amused”

, “

angry”

, “

astonished”

, “

curious”

, “

patient”

, “

happy”

, “

proud”

, “

scared”

, “

worried”

, and other emotion feedback.

7. The method of claim 1, wherein the indexing step includes scene change and shot change indexing.

8. The method of claim 1, wherein the indexing step includes manual indexing using high resolution still images.

9. The method of claim 1, further comprising sending the multimedia sequence and the plurality of categories of behavior and emotion segments to other users through a network, whereby the other users can selectively view the multimedia sequence using the plurality of categories of behavior and emotion segments as the video indices.

10. The method of claim 1, further comprising using a video display device to selectively view the multimedia sequence using the plurality of categories of behavior and emotion segments as the video indices.

11. The method of claim 10, further comprising enabling the user to click on an icon from the video display device to browse through the multimedia sequence using the plurality of categories of behavior and emotion segments as the video indices.

12. An apparatus for video indexing based on viewers'"'"' behavior and emotion feedback, comprising:

one or more multimedia acquisition devices capable of monitoring one or more viewers during a multimedia sequence play back, wherein each multimedia acquisition device includes one or more sensors for detecting the one or more viewers'"'"' behavior and emotion feedback; and

a computer capable of categorizing the one or more viewers'"'"' behavior and emotion feedback as a plurality of categories of behavior and emotion segments, and indexing the multimedia sequence using the plurality of categories of behavior and emotion segments,

wherein a user can selectively view the multimedia sequence using the plurality of categories of behavior and emotion segments as video indices.

13. The apparatus of claim 12, wherein the one or more sensors include image sensors.

14. The apparatus of claim 12, wherein the one or more sensors include audio sensors.

15. The apparatus of claim 12, further comprising a network capable of sending the multimedia sequence and the plurality of categories of behavior and emotion segments to other users, whereby the other users can selectively view the multimedia sequence using the plurality of categories of behavior and emotion segments as the video indices.

16. The apparatus of claim 12, wherein the multimedia acquisition device is a video camera.

17. A computer readable medium providing instructions for video indexing based on viewers'"'"' behavior and emotion feedback, the instructions comprising:

monitoring one or more viewers during a multimedia sequence play back;

detecting the one or more viewers'"'"' behavior and emotion feedback;

categorizing the one or more viewers'"'"' behavior and emotion feedback as a plurality of categories of behavior and emotion segments; and

indexing the multimedia sequence using the plurality of categories of behavior and emotion segments,

wherein a user can selectively view the multimedia sequence using the plurality of categories of behavior and emotion segments as video indices.

18. The computer readable medium of claim 17, wherein the instructions for detecting include instructions for detecting facial expression of the one or more viewers.

19. The computer readable medium of claim 17, wherein the instructions for detecting include instructions for detecting physical movement of the one or more viewers.

20. The computer readable medium of claim 17, wherein the instructions for detecting include instructions for detecting audio signal in the one or more viewers'"'"' voice.