System and method for very low frame rate video streaming for face-to-face video conferencing

• US 20060215016A1
• Filed: 03/22/2005
• Published: 09/28/2006
• Est. Priority Date: 03/22/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Cohen, Michael, Wang, Jue
• Granted Patent: US 7,583,287 B2
• Time in Patent Office: Days
• US Class Current: 348/14.120
• CPC Class Codes: H04N 7/148 Interfacing a video terminal to a particular transmission medium, e.g. ISDN

Abstract

A real-time low frame-rate video compression system and method that allows the user to perform face-to-face communication through an extremely low bandwidth network. At the encoder side, the system is able to automatically select only a few good faces from the original sequence with high visual quality and compress and transmit them. At the decoder side, the system use image-morphing based rendering method to generate a normal frame-rate video. Experimental results show that the system is superior to more traditional video codecs for low bit-rate face-to-face communication.

Claims

1. A process for encoding video data for face to face video conferencing comprising the process actions of:

inputting a video frame of a video frame sequence some images of which contain a face;

processing said video frame to locate said a face;

if a face is found processing the face to locate features, but if no face is found no longer processing said frame;

searching said face for features and using said found features to evaluate whether said frame is a good frame that should be encoded;

if the frame is not a good frame, no longer processing that frame;

subtracting said frame from said previously input frame to obtain a residual; and

encoding said residual with a video encoder.

2. The process of claim 1 further comprising the process action of transmitting the encoded residual with feature control parameters to a video conference participant.

3. The process of claim 1 further comprising the process action of, if the frame is a good frame, performing image morphing to align said frame with a previously input frame prior to subtracting said frame from said previously input frame.

4. The process of claim 1 wherein if the eyes are open designating said frame as a good frame.

5. The process of claim 1 wherein the whole frame is only encoded once and wherein in subsequent frames only the face is used in encoding.

6. The process of claim 1 wherein the encoded residual is transmitted in real time.

7. The process of claim 2 wherein the encoded residual is transmitted at very low bit rates.

8. The process of claim 1 wherein good frames are selected based on whether they contain a face and whether the eyes of the face are open.

9. The process of claim 1 wherein each good frame F_Gⁱ at time stamp t_Gⁱ is selected from the original input video frames based on the following criteria:

(a) t_min≦

t_Gⁱ−

t_G<sup>i−

1</sup>≦

t_max,

where t_min and t_max are parameters determining how frequently good frames are to be selected; and

(b) Both a face is found and the eyes of the face are open.

10. The process of claim 9 a random frame is sent every t_max time if a face is not found or the eyes are not open.

11. The process of claim 1 wherein the frame is only encoded if the person is not speaking.

12. The process of claim 2 wherein the face control parameters are time stamps and face feature positions.

13. The process of claim 1 further comprising the process actions of:

receiving the encoded residual with control parameters;

decoding said encoded residual and adding said decoded residual to a previously decoded frame to recover an image of said face;

using said control parameters to unmorph the face in a new frame to its location in the previously decoded frame;

putting the new frame in a buffer; and

rendering a current display by morphing consecutive images put in said buffer.

14. The process of claim 13 wherein cross-dissolving is performed in conjunction with morphing consecutive images when rendering said current display.

15. A computer-readable medium having computer-executable instructions for performing the process recited in claim 13.

17. The process of claim 15 wherein the current display is rendered in real-time.

18. The process of claim 15 wherein the encoded residual is received at very low bit rates.

16. A process for decoding video data for face-to-face video conferencing, comprising the process actions of:

receiving an encoded residual with control parameters based on features of a person'"'"'s face;

decoding said encoded residual and adding said decoded residual to a previously decoded frame to recover an image of a face;

using said control parameters to unmorph the face in a new frame to its location in the previously decoded frame;

putting the new frame in a buffer; and

rendering a current display by morphing consecutive images in said buffer.

19. A video conferencing system for streaming face-to-face video of video conference participants, comprising:

a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

input a video frame which possibly contains a face of a person participating in a video conference;

process said video frame to locate a face box around said possible face;

if a face box is found, process the face box to locate features, but if no face is found not process said frame any further;

use said found features to evaluate whether said frame is a good frame that should be encoded based on whether the eyes are open;

if frame is not a good frame, no longer process that frame;

if frame is a good frame, perform image morphing to align said frame with a previously input frame;

subtract said frame from said previously input frame to obtain a residual;

encode said residual with a video encoder; and

transmit said encoded residual to other video participants.

20. The system of claim 19 further comprising modules for:

receive the encoded residual with control parameters;

decode said encoded residual and add said decoded residual to a previously decoded frame to recover an image of said face;

use said control parameters to unmorph the face in a new frame to its location in the previously decoded frame;

put the new frame in a buffer; and

render a current display by morphing consecutive images in said buffer.