Method and apparatus for predicting events in video conferencing and other applications

• US 6,894,714 B2
• Filed: 12/05/2000
• Issued: 05/17/2005
• Est. Priority Date: 12/05/2000
• Status: Expired due to Fees

• Current Assignee: Pendragon Wireless LLC (Pendrell Corporation)
• Original Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Inventors: Gutta, Srinivas, Strubbe, Hugo, Colmenarez, Antonio
• Primary Examiner(s): Eng, George
• Application Number:
  US09/730,204
• Publication Number:
  US 20020101505A1
• Time in Patent Office: 1,624 Days
• Field of Search: 348 1401- 1403, 348/14.05, 348 1407- 141, 348/14.12, 348/14.16, 348/169, 348/211.99, 34821111-21114, 381/92, 381/110, 382/103, 382/291, 340/539.13
• US Class Current: 348/14.07
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Methods and apparatus are disclosed for predicting events using acoustic and visual cues. The present invention processes audio and video information to identify one or more (i) acoustic cues, such as intonation patterns, pitch and loudness, (ii) visual cues, such as gaze, facial pose, body postures, hand gestures and facial expressions, or (iii) a combination of the foregoing, that are typically associated with an event, such as behavior exhibited by a video conference participant before he or she speaks. In this manner, the present invention allows the video processing system to predict events, such as the identity of the next speaker. The predictive speaker identifier operates in a learning mode to learn the characteristic profile of each participant in terms of the concept that the participant “will speak” or “will not speak” under the presence or absence of one or more predefined visual or acoustic cues. The predictive speaker identifier operates in a predictive mode to compare the learned characteristics embodied in the characteristic profile to the audio and video information and thereby predict the next speaker.

Claims

1. A method for tracking a speaker in a video processing system, said video processing system processing audio and video information, the method comprising the steps of:

estimating an emotional state of a first speaker currently speaking from acoustic and prosodic features to predict when the first speaker is about to end speaking;

processing both said audio and video information to identify one of a plurality of cues defining behavior characteristics that suggest that a second person is about to speak;

maintaining a profile for at least one person that establishes a threshold for at least one of said plurality of cues; and

obtaining an image of said second person associated with said identified cue.

2. The method of claim 1, wherein at least one camera is focused in accordance with a one of pan, tilt and zoom values associated with said second person associated with said identified cue.

3. The method of claim 1, wherein said plurality of cues includes at least one visual cue identifying behavior that is typically exhibited by a person before said person speaks, and includes at least one acoustic cue identifying behavior that is typically exhibited by a person before said person speaks.

4. The method of claim 3, wherein said visual cue includes detecting the eyes of a person looking at said first speaker.

5. The method of claim 3, wherein said visual cue includes detecting the raising of a hand or finger by a person.

6. The method of claim 3, wherein said visual cue includes detecting a facial pose of a person in the direction of the first speaker.

7. The method of claim 3, wherein said visual cue includes detecting a nodding of the head by a person.

8. The method of claim 3, wherein said visual cue includes detecting a smile by a person.

9. The method of claim 3, wherein said visual cue includes detecting a person leaning forward.

10. The method of claim 3, wherein said acoustic cue includes non-verbal speech suggesting a person is about to speak.

11. The method of claim 1, wherein an event is associated with said first speaker is about to end speaking.

12. The method of claim 1, further comprising the step of obtaining an image of said second person associated with said identified cue.

13. The method of claim 1, further comprising the step of obtaining a pan-view with a camera when one of said cues indicates a person is about to end speaking.

14. A system for tracking a speaker in a video processing system, said video processing system processing audio and video information, comprising:

a memory for storing computer readable code; and

a processor operatively coupled to said memory, said processor configured to;

process acoustic and prosodic features of said audio information to predict when a first person is about to end speaking;

process both said audio and video information to identify one of a plurality of cues defining behavior characteristics that suggest that a second person is about to speak;

maintain a profile for at least one person that establishes a threshold for at least one of said plurality of cues; and

obtain an image of said second person associated with said identified cue.

15. The system of claim 14, further comprising at least one camera focused in accordance with a one of pan, tilt and zoom values associated with said second person associated with said identified cue.

16. The system of claim 14, wherein said plurality of cues includes at least one visual cue identifying behavior that is typically exhibited by a person before said person speaks, and includes at least one acoustic cue identifying behavior that is typically exhibited by a person before said person speaks.

17. An article of manufacture for tracking a speaker in a video processing system, said video processing system processing audio and video information, comprising:

a computer readable medium having a computer readable code means embodied thereon capable of execution by a processor, said computer readable code means comprising;

a step to process both of said audio and video information to identify one of a plurality of cues defining behavior characteristics that suggest that a first person is about to speak;

a step to process both of said audio and video information to identify another of the plurality of cues defining behavior characteristics that suggest that a second person is about to end speaking;

a step to maintain a profile for at least one person that establishes a threshold for at least one of said plurality of cues; and

a step to obtain an image of said first person based on said identified cues.

18. The article of manufacture of claim 17, further comprising at least one camera focused in accordance with a one of pan, tilt and zoom values associated with said second person associated with said identified cue.

19. The article of manufacture of claim 17, wherein said plurality of cues includes at least one visual cue identifying behavior that is typically exhibited by a person before said person speaks, and includes at least one acoustic cue identifying behavior that is typically exhibited by a person before said person speaks.