System and process for adding high frame-rate current speaker data to a low frame-rate video using audio watermarking techniques

• US 7,355,623 B2
• Filed: 04/30/2004
• Issued: 04/08/2008
• Est. Priority Date: 04/30/2004
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Cutler, Ross
• Primary Examiner(s): Ramakrishnaiah; Melur
• Application Number:
  US10/837,973
• Publication Number:
  US 20050243168A1
• Time in Patent Office: 1,439 Days
• Field of Search: 348 1401- 1409, 348/14.1, 348/14.11, 348/14.12, 348/14.13, 370/260, 370/261, 709/204, 345/629
• US Class Current: 348/14.12
• CPC Class Codes: G10L 19/018 Audio watermarking, i.e. embedding inaudible data in the audio signal
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

A system and process for highlighting the current speaker on an on-going basis in each frame of a low frame-rate video of an event having multiple people in attendance, such as a video teleconference, is presented. In general, this is accomplished by periodically identifying an attendee that is currently speaking at a rate substantially faster than the video frame rate, and for each frame of the video updating the frame to highlight the attendee currently speaking. More particularly, an A/V source provides video and audio data streams to the client computing device, with current speaker data embedded into the audio stream via audio watermarking techniques. The client device extracts the current speaker data from the audio stream, and then renders and displays the video while using the current speaker data to periodically update the frame being displayed to highlight the current speaker.

Claims

1. A computer-implemented process for facilitating the identification of a current speaker in each frame of a low frame-rate video, comprising using a computer to perform the following process actions:

obtaining audio and video of an event having multiple people in attendance;

transmitting the video of the event at a prescribed frame rate to a client computing device;

tracking the movements of the attendees and recording their positions when each video frame is transmitted and their subsequent positions between the transmission of the video frames;

periodically identifying which of the attendees is currently speaking at a rate significantly faster than the prescribed video frame rate;

periodically generating an indicator which comprises the location of the attendee who is currently speaking as depicted in the last-transmitted video frame regardless of their current position;

embedding the indicators as they are generated into the obtained audio of the event using an audio watermarking technique; and

continuously transmitting the audio of the event with the embedded indicators to the client computing device for use in highlighting a region in the last-transmitted video frame depicting the attendee at the location specified in the last-embedded indicator.

2. The process of claim 1, further comprising the process actions of:

each time an indicator is generated which comprises the location of the attendee who is currently speaking, generating a separate indicator for each attendee depicted in the last-transmitted video frame who is not currently speaking which comprises the location of the non-speaking attendee as depicted in the video frame regardless of the attendee'"'"'s current position; and

embedding each of said indicators associated with non-speaking attendees into the obtained audio of the event using said audio watermarking technique for use in un-highlighting a region in the last-transmitted video frame depicting the non-speaking attendee at the location specified in the indicator whenever that region is highlighted based on a previously embedded indicator specifying the attendee under consideration was then speaking.

3. The process of claim 2, wherein the process action of periodically generating an indicator which comprises the location of the attendee who is currently speaking, comprises the action of generating said indicator at a prescribed interval.

4. The process of claim 2, wherein the process action of periodically generating an indicator which comprises the location of the attendee who is currently speaking, comprises the actions of:

generating an indicator immediately after the generation of each video frame; and

thereafter,

generating indicators only when either an attendee who was not speaking at the time the last indicator was generated begins speaking or an attendee that was speaking at the time the last indicator was generated stops speaking.

5. The process of claim 1, wherein the video and audio are transmitted directly to the client computing device.

6. The process of claim 1, wherein the video and audio are transmitted to the client computing device via a computer network.

7. The process of claim 2, wherein each indicator takes the form of a tuple comprising a location parameter which specifies the location in the last-transmitted video frame where an attendee associated with the tuple is depicted.

8. The process of claim 7, wherein each tuple further comprises a speaker status parameter which specifies if the attendee associated with the tuple is currently speaking or not.

9. A system for facilitating the identification of a current speaker in each frame of a low frame-rate video, comprising:

a general purpose computing device;

at least one video camera;

at least one microphone; and

a computer program comprising program modules executable by the computing device, comprising,

a video stream creation module which generates a data stream of video frames at a prescribed frame rate,

a current speaker detection module which,

periodically identifies the current speaker among the persons depicted in each video frame of the video stream at a rate substantially faster than the video frame rate, and

tracks the movements of the persons depicted in each video frame between the generation of said frames so as to equate their current location with their original location when the video frame was generated;

a current speaker data module which generates indicators, each of which specifies,

the location of a person depicted in a video frame associated with the indicator, and

whether the person whose location is specified is currently speaking or not,

an audio data stream creation module which generates a continuous stream of audio data, and

an audio watermark encoder module which embeds said indicators as they are generated into the audio data stream using an audio watermarking technique,

a transmission module which transmits the video stream and indicator-embedded audio stream to a client computing device or transfers the streams to storage for later transmission to the client computing device.

10. The system of claim 9, wherein the computer program further comprises a video encoder module which encodes the video data stream prior to transmission or storage.

11. The system of claim 9, wherein the computer program further comprises an audio encoder module which encodes the audio data stream prior to transmission or storage.

12. The system of claim 9, wherein the computer program further comprises a sound source localization module which uses the audio data stream to identify the location of persons depicted in the video frames of the video data stream.

13. The system of claim 12, wherein the current speaker detection module comprises sub-modules for using the video data stream and person location information generated by the sound source localization module to periodically identify the current speaker among the persons depicted in each video frame of the video stream at a rate substantially faster than the video frame rate, and track the movements of the persons depicted in each video frame between the generation of said frames so as to equate their current location with their original location when the video frame was generated.

14. The system of claim 9, wherein the current speaker data module comprises a tuple generator sub-module which generates said indicators in the form of tuples, each of which comprises,

a location parameter identifying the location of a person depicted in a video frame associated with the tuple as depicted in that frame, and

a speaker status parameter identifying whether the person whose location is specified by the location parameter of the tuple is currently speaking or not.

15. A computer-implemented process for highlighting the current speaker in each frame of a low frame-rate video of an event having multiple people in attendance, comprising using a computer to perform the following process actions:

obtaining the low frame-rate video of the event;

obtaining a continuous audio stream of the event which has embedded therein via a watermarking technique periodically generated indicators, each of which comprises the location of the attendee who is currently speaking in a last-obtained video frame, wherein said indicators are generated at a rate significantly faster than the video frame rate;

synchronizing the audio and video streams; and

extracting each indicator from the audio stream as it is obtained and highlighting a region in last-obtained video frame based on the location of the current speaker specified in the indicator under consideration, wherein said highlighting visually distinguishes a current speaker from all other attendees depicted in the last-obtained video frame.

16. The process of claim 15, wherein the process action of highlighting a region in the last-obtained video frame based on the location of the current speaker specified in an indicator under consideration, for each indicator extracted from the audio stream in turn, comprises the actions of:

identifying a region in the video frame, which the indicator under consideration is associated with, that has a prescribed size and shape and which has a prescribed geometric relationship to the specified location of the current speaker;

modifying the appearance of all or a part of the region in a prescribed manner so as to visually distinguish a current speaker from all other attendees depicted in the video frame.

17. A system for highlighting the current speaker in each frame of a low frame-rate video of an event having multiple people in attendance, comprising:

a general purpose computing device;

a computer program comprising program modules executable by the computing device, comprising,

a video input module which obtains the low frame-rate video of the event,

an audio input module which obtains a continuous audio stream of the event that has embedded therein via a watermarking technique periodically generated indicators, each of which comprises the location of an attendee depicted in the last-obtained video frame and indicates if that attendee is currently speaking or not, wherein said indicators are generated at a rate significantly faster than the video frame rate,

a synchronizer module which synchronizes the audio and video streams,

an audio watermark detector module which extracts each indicator from the audio stream as it is obtained,

a speaker highlighting module which highlights a region in the last-obtained video frame that is associated with an attendee that a last-extracted indicator specifies is currently speaking, based on the location of the attendee specified in that indicator, wherein said highlighting visually distinguishes a current speaker from all other attendees depicted in the last-obtained video frame that are not currently speaking.

18. The system of claim 17, wherein the highlighting module comprises sub-modules which for each indicator extracted:

determines from the indicator under consideration if the attendee associated with that indicator is currently speaking or not;

whenever it is determined the attendee associated with the indicator under consideration is speaking, determines if a region of the last-obtained video frame which is associated with the location of the current speaker specified in the indicator under consideration is highlighted or not; and

whenever it is determined the region of the last-obtained video frame which is associated with the location of the current speaker specified in the indicator under consideration is not highlighted, highlights that region in a prescribed manner.

19. The system of claim 18, wherein the highlighting module further comprises sub-modules which for each indicator extracted that it is determined the attendee associated with the indicator is not speaking:

determines if a region of the last-obtained video frame which is associated with the location of the non-speaking attendee specified in the indicator under consideration is highlighted or not; and

whenever it is determined the region of the last-obtained video frame which is associated with the location of the non-speaking attendee is highlighted, un-highlights that region.

20. The system of claim 17, wherein the low frame-rate video is encoded, and wherein the computer program further comprises a module for decoding the video prior to the highlighting module processing it.

21. The system of claim 17, wherein the audio is encoded, and wherein the computer program further comprises a module for decoding the audio prior to the audio watermark detector module processing it.