System and process for adding high frame-rate current speaker data to a low frame-rate video using delta frames

• US 7,355,622 B2
• Filed: 04/30/2004
• Issued: 04/08/2008
• Est. Priority Date: 04/30/2004
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Hovhannisyan, Armen, Cutler, Ross, Clarke, Simon
• Primary Examiner(s): Ramakrishnaiah; Melur
• Application Number:
  US10/837,244
• Publication Number:
  US 20050243167A1
• Time in Patent Office: 1,439 Days
• Field of Search: 348 1401- 1409, 348/14.1, 348/14.11, 348/14.12, 348/14.13, 370/260, 370/261, 709/204
• US Class Current: 348/14.12
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/148 Interfacing a video terminal to a particular transmission medium, e.g. ISDN

Abstract

A system and process for highlighting the current speaker on an on-going basis in each frame of a low frame-rate video of an event having multiple people in attendance is presented. In general, this is accomplished by periodically identifying an attendee that is currently speaking at a rate substantially faster than the video frame rate, and updating each frame of the video to highlight the current speaker. More particularly, an A/V source provides a video stream to a client computing device that includes delta frames interspersed between the frames of the low frame-rate video. The full video frames act as keyframes, and the delta frames provide the changes needed to modify the last displayed version of the last keyframe to highlight just the region associated with the location of a current speaker. This allows the client device to operate as a standard A/V rendering and display unit.

Claims

1. A computer-implemented process for highlighting a current speaker in each frame of a low frame-rate video at a rate significantly faster than the video frame rate, comprising using a computer to perform the following process actions:

obtaining audio and video of an event having multiple people in attendance;

tracking the movements of the attendees and recording their positions when each frame of the video is obtained and their subsequent positions until the next video frame is obtained;

periodically identifying which of the attendees is currently speaking at a rate significantly faster than the prescribed video frame rate;

generating a data stream of video frames from the obtained video of the event comprising,

keyframes generated at a prescribed frame rate, and

delta frames, one or more of which are generated between the generation of each pair of consecutive keyframes, wherein each delta frame comprises just those changes needed to the last-generated keyframe as modified if all previously generated delta frames, if any, applicable to that keyframe were applied thereto, which highlight a region in that keyframe associated with the location of a current speaker as depicted in the last-generated keyframe in a way that visually distinguishes that attendee from all other currently non-speaking attendees also depicted in the last-generated keyframe; and

generating an audio data stream from the obtained audio of the event.

2. The process of claim 1, wherein the process action of generating delta frames that when applied to the last-generated keyframe as it would appear if all previously generated delta frames, if any, applicable to that keyframe were applied thereto highlights a region in that keyframe associated with the location of a current speaker as depicted in the last-generated keyframe, comprises the action of whenever the last-generated keyframe as it would appear if all previously generated delta frames, if any, applicable to that keyframe where applied thereto highlights an attendee who is no longer speaking, includes changes in the delta frame which un-highlight that attendee.

3. The process of claim 2, wherein the process action generating delta frames, comprises the action of generating each delta frame at a prescribed interval.

4. The process of claim 2, wherein at the time of its generation a region in each keyframe associated with the location of a current speaker is highlighted so as to visually distinguish that attendee from all other currently non-speaking attendees also depicted in the keyframe.

5. The process of claim 4, wherein the process action generating delta frames, comprises the action of generating each delta frame only when either an attendee who was not speaking at the time the last delta frame was generated begins speaking or an attendee that was speaking at the time the last delta frame was generated stops speaking.

6. The process of claim 2, wherein at the time of their generation, keyframes are not highlighted to visually distinguish a currently speaking attendee from all other currently non-speaking attendees also depicted in the keyframe, and wherein the process action generating delta frames, comprises the actions of:

generating a delta frame immediately after the generation of each keyframe; and

thereafter,

generating each delta frame only when either an attendee who was not speaking at the time the last delta frame was generated begins speaking or an attendee that was speaking at the time the last delta frame was generated stops speaking.

7. The process of claim 1, wherein the video and audio data streams are transmitted directly to a client computing device.

8. The process of claim 1, wherein the video and audio data streams are transmitted to a client computing device via a computer network.

9. The process of claim 1, wherein the video and audio data streams are stored for future transmission to a client computing device.

10. The process of claim 1, wherein the process action of generating delta frames comprising changes needed to the last-generated keyframe as modified if all previously generated delta frames, if any, applicable to that keyframe were applied thereto, which highlight a region in that keyframe associated with the location of a current speaker as depicted in the last-generated keyframe, comprises the actions of:

identifying a region in the keyframe that has a prescribed size and shape and which has a prescribed geometric relationship to the specified location of the current speaker;

modifying the appearance of all or a part of the region in a prescribed manner so as to visually distinguish a current speaker from all other attendees depicted in the keyframe.

11. A system for highlighting a current speaker in each frame of a low frame-rate video at a rate significantly faster than the video frame rate, comprising:

a general purpose computing device;

at least one video camera;

at least one microphone; and

a computer program comprising program modules executable by the computing device, comprising,

a video stream creation module which generates a stream of keyframes at a prescribed frame rate using a video signal output from each video camera,

an audio stream creation module which generates a continuous stream of audio data using an audio signal output from each microphone;

a current speaker detection module which,

periodically identifies the current speaker among the persons depicted in each keyframe of the video stream at a rate substantially faster than the keyframe generation rate, and

tracks the movements of the persons depicted in each keyframe between the generation of said keyframes so as to equate their current location with their original location when the keyframe was generated;

a delta frame generation module which generates one or more delta frames between the generation of each pair of consecutive keyframes, wherein each delta frame comprises just those changes needed to the last-generated keyframe as it would appear if all previously generated delta frames, if any, applicable to that keyframe were applied thereto, which highlight a region in that keyframe associated with the location of a current speaker as depicted in the last-generated keyframe in a way that visually distinguishes that attendee from all other currently non-speaking attendees also depicted in the last-generated keyframe.

12. The system of claim 11, wherein the computer program further comprises a video encoder module which encodes the video data stream for transfer to a client computing device or to storage for later transfer to the client computing device.

13. The system of claim 11, wherein the computer program further comprises an audio encoder module which encodes the audio data stream for transfer to a client computing device or to storage for later transfer to the client computing device.

14. The system of claim 11, wherein the computer program further comprises a sound source localization module which uses the audio data stream to identify the location of persons depicted in the video frames of the video data stream.

15. The system of claim 14, wherein the current speaker detection module comprises sub-modules for using the video data stream and person location information generated by the sound source localization module to periodically identify the current speaker among the persons depicted in each video frame of the video stream at a rate substantially faster than the video frame rate, and track the movements of the persons depicted in each video frame between the generation of said frames so as to equate their current location with their original location when the video frame was generated.

16. The system of claim 11, wherein the delta frame generation module comprises sub-modules which for each delta frame generated and for each attendee depicted in the keyframe associated with the delta frame being generated:

determines from the current speaker detection module if the attendee is currentiy speaking or not;

whenever it is determined the attendee is currently speaking, determines if a region of the keyframe which is associated with the location of the attendee is highlighted or not;

whenever it is determined the region of the keyframe which is associated with the location of the attendee is not highlighted, highlights that region in a prescribed manner;

whenever it is determined the attendee is not currently speaking, determines if a region of the keyframe which is associated with the location of the non-speaking attendee is highlighted or not; and

whenever it is determined the region of the keyframe which is associated with the location of the non-speaking attendee is highlighted, un-highlights that region.

17. A computer-implemented process for highlighting the current speaker in each frame of a low frame-rate video of an event having multiple people in attendance, comprising using a computer to perform the following process actions:

obtaining the low frame-rate video of the event which comprises,

keyframes generated at a prescribed frame rate, and

delta frames, one or more of which are generated between the generation of each pair of consecutive keyframes, wherein each delta frame comprises just those changes needed to the last-generated keyframe as modified if all previously generated delta frames, if any, applicable to that keyframe were applied thereto, which highlight a region in that keyframe associated with the location of a current speaker as depicted in the last-generated keyframe in a way that visually distinguishes that attendee from all other currently non-speaking attendees also depicted in the last-generated keyframe, wherein generating the one or more delta frames comprised periodically identifying which of the attendees is currently speaking at a rate siginificantly faster than the prescribed frame rate;

obtaining a continuous audio stream of the event;

synchronizing the audio and video streams; and

rendering and displaying the video while playing the audio.

18. A system for highlighting the current speaker in each frame of a low frame-rate video stream of an event having multiple people in attendance, comprising:

a general purpose computing device;

a computer program comprising program modules executable by the computing device, comprising,

a video input module which obtains the low frame-rate video stream, said video stream comprising,

keyframes generated at a prescribed frame rate, and

delta frames, one or more of which are generated between the generation of each pair of consecutive keyframes, wherein each delta frame comprises just those changes needed to the last-generated keyframe as modified if all previously generated delta frames, if any, applicable to that keyframe were applied thereto, which highlight a region in that keyframe associated with the location of a current speaker as depicted in the last-generated keyframe in a way that visually distinguishes that attendee from all other currently non-speaking attendees also depicted in the last-generated keyframe, wherein generating the one or more delta frames comprised periodically identifying which of the attendees is currently speaking at a rate significantly faster than the prescribed frame rate,

an audio input module which obtains a continuous audio stream of the event,

a synchronizer module which synchronizing the audio and video streams, and

a rendering and display module which renders and displays the video while playing the audio.

19. The system of claim 18, wherein the low frame-rate video stream is encoded, and wherein the computer program further comprises a module for decoding the video prior to the synchronizer module synchronizing the audio and video streams.

20. The system of claim 18, wherein the audio stream is encoded, and wherein the computer program further comprises a module for decoding the audio prior to the synchronizer module synchronizing the audio and video streams.