Method and system for real-time facial image enhancement

• US 7,227,976 B1
• Filed: 06/27/2003
• Issued: 06/05/2007
• Est. Priority Date: 07/08/2002
• Status: Expired

• Current Assignee: F Poszat Hu LLC (Intellectual Ventures LLC)
• Original Assignee: Videomining Corporation
• Inventors: Jung, Namsoon, Sharma, Rajeev
• Primary Examiner(s): Mancuso; Joseph
• Assistant Examiner(s): Akhavannik; Hadi
• Application Number:
  US10/609,245
• Time in Patent Office: 1,439 Days
• Field of Search: 382115-118, 382/103, 351/211
• US Class Current: 382/103
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation

Abstract

The present invention is a system and method for detecting facial features of humans in a continuous video and superimposing virtual objects onto the features automatically and dynamically in real-time. The suggested system is named Facial Enhancement Technology (FET). The FET system consists of three major modules, initialization module, facial feature detection module, and superimposition module. Each module requires demanding processing time and resources by nature, but the FET system integrates these modules in such a way that real time processing is possible. The users can interact with the system and select the objects on the screen. The superimposed image moves along with the user'"'"'s random motion dynamically. The FET system enables the user to experience something that was not possible before by augmenting the person'"'"'s facial images. The hardware of the FET system comprises the continuous image-capturing device, image processing and controlling system, and output display system.

Claims

1. A method for enhancing facial images of a person in a video by superimposing virtual object images onto said facial images automatically and dynamically in real-time, comprising the following steps of:

(a) capturing a plurality of images for said person with at least one means for capturing images,

(b) processing said plurality of images in order to detect the face of said person in said plurality of images for an initial face detection,

(c) tracking the detected face in real-time using a face tracking window,

(d) verifying said face within the boundary of the face tracking window using a second local face detection,

(e) calculating facial feature coordinate information from said face by applying a plurality of facial feature detection approaches and applying a block-processing,

(f) processing a touch-free interaction by said person to select virtual object images, and

(g) superimposing said virtual object images onto said facial images automatically and dynamically in real-time,

wherein the virtual object images comprise predefined virtual objects that can be attached to and enhance the facial images,

whereby the verification of said face within the boundary of the face tracking window makes more accurate facial images to be passed on to the next step while enabling real-time tracking of the face, and

whereby said touch-free interaction enables said person to choose said virtual object images on a means for displaying output to superimpose onto said facial images without touching any device.

2. The method according to claim 1, wherein the method further comprises a step of repeating the steps from (a) to (g) for a plurality of persons.

3. The method according to claim 1, wherein the method further comprises a step of estimating regions of interest for each facial feature in said face dynamically,

whereby said regions of interest change according to the results from the step of verifying said face within the boundary of the face tracking window, and

whereby said regions of interest are used as boundaries for detecting each facial feature, such as eyes, nose, and mouth on said face.

4. The method according to claim 1, wherein the method further comprises a step of applying fusion algorithms and geometrical constraints to said facial feature coordinate information.

5. The method according to claim 1, wherein the method further comprises a step of smoothing said facial feature coordinate information.

6. The method according to claim 5, wherein the method further comprises a step of storing a history of a plurality of said facial feature coordinate information and applying a smoothing algorithm for the current facial feature coordinate using said history of said facial feature coordinate information.

7. The method according to claim 1, wherein the method further comprises a step of combining the face detection process and the real-time face tracking process,

whereby the integration enables efficient and robust real-time facial image processing.

8. The method according to claim 1, wherein the method further comprises a step of applying said block-processing to each of said plurality of facial feature detection approaches.

9. The method according to claim 1, wherein the step of (g) superimposing said virtual object images onto said facial images automatically and dynamically in real-time further comprises steps of:

(a) preparing virtual object images,

(b) validating said facial feature coordinate information,

(c) smoothing,

(d) aesthetic processing, and

(e) processing final superimposition.

10. The method according to claim 9, wherein the step of (a) preparing virtual object images further comprises a step for setting pivot points in said virtual object images.

11. The method according to claim 9, wherein the step of (d) aesthetic processing further comprises a step for processing rotation and translation of said virtual object images.

12. An apparatus for enhancing facial images of a person in a video by superimposing virtual object images onto said facial images automatically and dynamically in real-time, comprising the following steps of:

(a) capturing a plurality of images for said person with at least one means for capturing images,

(b) processing said plurality of images in order to detect the face of said person in said plurality of images for an initial face detection,

(c) tracking the detected face in real-time using a face tracking window,

(d) verifying said face within the boundary of the face tracking window using a second local face detection,

(e) calculating facial feature coordinate information from said face by applying a plurality of facial feature detection approaches and applying a block-processing,

(f) processing a touch-free interaction by said person to select virtual object images, and

(g) superimposing said virtual object images onto said facial images automatically and dynamically in real-time,

wherein the virtual object images comprise predefined virtual objects which that can be attached to and enhance the facial images,

whereby the verification of said face within the boundary of the face tracking window makes more accurate facial images to be passed on to the next step while enabling real-time tracking of the face, and

whereby said touch-free interaction enables said person to choose said virtual object images on a means for displaying output to superimpose onto said facial images without touching any device.

13. The apparatus according to claim 12, wherein the apparatus further comprises means for repeatedly using the means from (a) to (g) for a plurality of persons.

14. The apparatus according to claim 12, wherein the apparatus further comprises means for estimating regions of interest for each facial feature in said face dynamically,

whereby said regions of interest change according to the results from using the means for verifying said face within the boundary of the face tracking window, and

whereby said regions of interest are used as boundaries for detecting each facial feature, such as eyes, nose, and mouth on said face.

15. The apparatus according to claim 12, wherein the apparatus further comprises means for applying fusion algorithms and geometrical constraints to said facial feature coordinate information.

16. The apparatus according to claim 12, wherein the apparatus further comprises means for smoothing said facial feature coordinate information.

17. The apparatus according to claim 16, wherein the apparatus further comprises means for storing a history of a plurality of said facial feature coordinate information and applying a smoothing algorithm for the current facial feature coordinate using said history of said facial feature coordinate information.

18. The apparatus according to claim 12, wherein the apparatus further comprises means for combining the face detection process and the real-time face tracking process,

whereby the integration enables efficient and robust real-time facial image processing.

19. The apparatus according to claim 12, wherein the apparatus further comprises means for applying said block-processing to each of said plurality of facial feature detection approaches.

20. The apparatus according to claim 12, wherein the means for (g) superimposing said virtual object images onto said facial images automatically and dynamically in real-time further comprises means for:

(a) preparing virtual object images,

(b) validating said facial feature coordinate information,

(c) smoothing,

(d) aesthetic processing, and

(e) processing final superimposition.

21. The apparatus according to claim 20, wherein the means for (a) preparing virtual object images further comprises means for setting pivot points in said virtual object images.

22. The apparatus according to claim 20, wherein the means for (d) aesthetic processing further comprises means for processing rotation and translation of said virtual object images.