System and method for very low frame rate teleconferencing employing image morphing and cropping

• US 7,659,920 B2
• Filed: 03/22/2005
• Issued: 02/09/2010
• Est. Priority Date: 03/22/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Cohen, Michael, Wang, Jue
• Primary Examiner(s): Woo; Stella L
• Application Number:
  US11/087,355
• Publication Number:
  US 20060215014A1
• Time in Patent Office: 1,785 Days
• Field of Search: 348/14.12, 348/14.01, 348/14.08, 348/14.13, 382/100
• US Class Current: 348/14.13
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/537 Motion estimation other than block-based
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/148 Interfacing a video terminal to a particular transmission medium, e.g. ISDN

Abstract

A real-time low frame-rate video compression system and method that allows the user to perform face-to-face communication through an extremely low bandwidth network. The system and method employs image cropping and morphing to reduce frame rates. At the encoder side, the system is able to automatically select only a few good faces from the original sequence with high visual quality and compress and transmit them. At the decoder side, the system use image-morphing based rendering method to generate a normal frame-rate video. Experimental results show that the system is superior to more traditional video codecs for low bit-rate face-to-face communication.

Claims

1. A system for compressing data for video conferencing, comprising:

a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

capture a sequence of images some containing a person'"'"'s face;

locate the person'"'"'s face in each of the images in the sequence that contain a face;

select some of the images that contain a face; and

transmit only the face region including eyes of the selected images.

2. The system of claim 1 further comprising modules to:

receive a first image of the face region including eyes and mouth;

apply an image morph that aligns the eye and mouth positions in subsequently received images to the eyes and mouth positions in the first received image to provide motion compensation.

3. The system of claim 1 further comprising modules to:

receive said transmitted selected images; and

create intermediate images of the face region by morphing between the received images.

4. A process for low frame rate video conferencing, comprising the process actions of:

inputting a video frame which possibly contains a face;

processing said video frame to locate a face;

if a face is found, processing the face to locate features of the face, but if no face is found no longer processing said frame;

if features are found, using the features to perform image morphing to align said face in said frame with a face in a previous input frame;

subtracting said face in said frame from said face in said previously input frame to obtain a residual of said face;

encoding said residual of said face with a video encoder;

transmitting the encoded residual of the face with feature control parameters to a video conference participant.

5. The process of claim 4 further comprising the process actions of:

receiving the residual of the face at a decoder;

decoding the residual and adding it to the previous decoded frame;

morphing the face to its original location using the feature control parameters to create a new frame;

placing the new frame in a frame buffer; and

creating a real-time video by morphing between consecutively received frames in the frame buffer.

6. The process of claim 5 further comprising the process action of cross-dissolving consecutive frames in the frame buffer while morphing.

7. The process of claim 6 wherein the process action of cross-dissolving consecutive frames in the frame buffer comprises the process actions of:

creating an intermediate frame between consecutive frames in the frame buffer wherein one frame is a current frame and another frame is a previous frame; and

blending the corresponding pixels between the current frame and the intermediate frame; and

blending the corresponding pixels between the intermediate frame and the current frame.

8. The process of claim 7 wherein the intermediate frame is created by linear interpolation between the previous frame and the current frame.

9. The process of claim 8 wherein the linear interpolation uses feature points of the previous frame and the current frame.

10. The process of claim 5 wherein the morphed frames and the consecutively received good frames are displayed at the same rate as the original video.

11. A computer-readable medium having computer-executable instructions for performing the process recited in claim 5.

12. A process for encoding video to compensate for motion between frames, comprising the process actions of:

inputting a sequence of frames, some of said frames containing a face;

calculating the differences in eye locations between two consecutive frames which contain a face;

estimating the location of the mouth in the two consecutive frames based on an initially marked frame;

creating a morph mesh using said differences in eye locations and said estimated mouth locations between consecutive frames;

morphing each frame in the sequence to the previous consecutive frame in the sequence;

subtracting each morphed frame from the previous frame to obtain a residual image;

encoding the residual and the eye and mouth locations;

transmitting the encoded residual and the eye and mouth locations;

decoding the residual;

adding the residual to the previously received frame to produce a new frame;

unmorphing the new frame based on the transmitted eye and mouth locations; and

displaying the new frame while cross-dissolving from the previously received frame.

13. The process of claim 12 further comprising the process action of switching from displaying the previously received frame to the new frame.

14. The process of claim 12 further comprising the process action of cross-dissolving while morphing while transitioning from displaying the previously received frame to the new frame.