Face recognition from video images

• US 6,301,370 B1
• Filed: 12/04/1998
• Issued: 10/09/2001
• Est. Priority Date: 04/13/1998
• Status: Expired due to Term

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Neven Vision, Inc. (Alphabet Inc.)
• Inventors: Maurer, Thomas, Neven, Hartmut, Nocera, Luciano Pasquale Agostino, Elagin, Egor Valerievich, Steffens, Johannes Bernhard
• Primary Examiner(s): Boudreau, Leo
• Assistant Examiner(s): Mariam, Daniel G.
• Application Number:
  US09/206,195
• Time in Patent Office: 1,040 Days
• Field of Search: 382/117, 382/118, 382/168, 382/181, 382/190, 382/195, 382/199, 382/103, 382/209, 382/216, 382/219, 382/279, 382/282, 382/115, 382/100, 382/276, 348/169, 342/90
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/262 using transform domain methods, e.g. Fourier domain methods
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/302 for viewing without the aid of special glasses, i.e. using autostereoscopic displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

The present invention is embodied in an apparatus, and related method, for detecting and recognizing an object in an image frame. The object may be, for example, a head having particular facial characteristics. The object detection process uses robust and computationally efficient techniques. The object identification and recognition process uses an image processing technique based on model graphs and bunch graphs that efficiently represent image features as jets. The jets are composed of wavelet transforms and are processed at nodes or landmark locations on an image corresponding to readily identifiable features. The system of the invention is particularly advantageous for recognizing a person over a wide variety of pose angles.

Claims

1. A process for recognizing objects in an image frame, comprising steps for:

detecting an object in the image frame and bounding a portion of the image frame associated with the object resulting in a bound portion of the image frame that is associated with the object and an unbound portion of the image frame that is not associated with the object;

transforming only the bound portion and not the unbound portion of the image frame using a wavelet transformation to generate a transformed image;

locating, on the transformed image, nodes associated with distinguishing features of the object defined by wavelet jets of a bunch graph generated from a plurality of representative object images;

identifying the object based on a similarity between wavelet jets associated with an object image in a gallery of object images and wavelet jets at the nodes on the transformed image.

2. A process for recognizing objects as defined in claim 1, further comprising sizing and centering the detected object within the bound portion of the image such that the detected object has a predetermined size and location within the bound portion.

3. A process for recognizing objects as defined in claim 1, further comprising a step for suppressing a background portion, of the bound portion of the image frame, that is not representing the object, prior to identifying the object.

4. A process for recognizing objects as defined in claim 3, wherein the suppressed background portion is gradually suppressed near edges of the object in the bound portion of the image frame.

5. A process for recognizing objects as defined in claim 1, wherein the object is a head of a person exhibiting a facial region.

6. A process for recognizing objects as defined in claim 1, wherein the bunch graph is based on a three-dimensional representation of the object.

7. A process for recognizing objects as defined in claim 1, wherein the wavelet transformation is performed using phase calculations that are performed using a hardware adapted phase representation.

8. A process for recognizing objects as defined in claims 1, wherein the locating step is performed using a coarse-to-fine approach.

9. A process for recognizing objects as defined in claim 1, wherein the bunch graph is based on predetermined poses.

10. A process for recognizing objects as defined in claim 1, wherein in the identifying step uses a three-dimensional representation of the object.

11. A process for recognizing objects as defined in claim 1, wherein the bound portion covers less than ten percent of the image frame.

12. A process for recognizing objects as defined in claim 1, wherein the step for detecting the object includes detecting a color associated with the object.

13. A process for recognizing objects in a sequence of image frames, comprising:

detecting an object in the image frames and bounding a portion of each image frame associated with the object;

transforming the bound portion of each image frame using a wavelet transformation to generate a transformed image;

locating, on the transformed images, nodes associated with distinguishing features of the object defined by wavelet jets of a bunch graph generated from a plurality of representative object images;

identifying the object based on a similarity between wavelet jets associated with an object image in a gallery of object images and wavelet jets at the nodes on the transformed images.

14. A process for recognizing objects as defined in claim 13, wherein the step of detecting an object further comprises tracking the object between image frames based on a trajectory associated with the object.

15. A process for recognizing objects as defined in claim 13, further comprising a preselecting process that chooses a most suitable view of an object out of sequence of views that belong to a particular trajectory.

16. A process for recognizing objects as defined in claim 13, wherein the step of locating the nodes includes tracking the nodes between image frames.

17. A process for recognizing objects as defined in claim 16, further comprising reinitializing a tracked node if the node'"'"'s position deviates beyond a predetermined position constraint between image frames.

18. A process for recognizing objects as defined in claim 17, wherein the predetermined position constraint is based on a geometrical position constraint associated with relative positions between the node locations.

19. A process for recognizing objects as defined in claim 13, wherein the image frames are stereo images and the step of detecting includes generating a disparity histogram and a silhouette image to detect the object.

20. A process for recognizing objects as defined in claim 19, wherein the disparity histogram and silhouette image generate convex regions which are associated with head movement and which are detected by a convex detector.

21. A process for recognizing objects as defined in claim 13, wherein the wavelet transformations are performed using phase calculations that are performed using a hardware adapted phase representation.

22. A process for recognizing objects as defined in claim 13, wherein the bunch graph is based on a three-dimensional representation of the object.

23. A process for recognizing objects as defined in claim 13, wherein the locating step is performed using a coarse-to-fine approach.

24. A process for recognizing objects as defined in claim 13, wherein the bunch graph is based on predetermined poses.

25. A process for recognizing objects as defined in claim 13, wherein the bound portion covers less than ten percent of the image frame.

26. A process for recognizing objects as defined in claim 13, wherein the step for detecting the object includes detecting a color associated with the object.

27. A process for recognizing objects as defined in claim 13, wherein the step for detecting the object includes detecting movement associated with the object in the sequence of image frames.

28. Apparatus for recognizing objects in an image, comprising:

means for detecting an object in the image frame and bounding a portion of the image frame associated with the object resulting in a bound portion of the image frame that is associated with the object and an unbound portion of the image frame that is not associated with the object;

means for transforming only the bound portion and not the unbound portion of the image frame using a wavelet transformation to generate a transformed image;

means for locating, on the transformed image, nodes associated with distinguishing features of the object defined by wavelet jets of a bunch graph generated from a plurality of representative object images;

means for identifying the object based on a similarity between wavelet jets associated with an object image in a gallery of object images and wavelet jets at the nodes on the transformed image.

29. Apparatus for recognizing objects as defined in claim 28, wherein the bound portion covers less than ten percent of the image frame.

30. Apparatus for recognizing objects as defined in claim 28, wherein the means for detecting the object includes a neural network.

31. Apparatus for recognizing objects as defined in claim 28, wherein the means for detecting the object includes means for detecting a color associated with the object.

32. Apparatus for recognizing objects as defined in claim 28, further comprising means for suppressing a background portion, of the bound portion of the image frame, that is not representing the object, prior to identifying the object.

33. Apparatus for recognizing objects as defined in claim 32, wherein the suppressed background portion is gradually suppressed near edges of the object in the bound portion of the image frame.

34. Apparatus for recognizing objects in a sequence of image frames, comprising:

means for detecting an object in the image frames and bounding a portion of each image frame associated with the object resulting in a bound portion of each image frame that is associated with the object and an unbound portion of each image frame that is not associated with the object;

means for transforming only the bound portion and not the unbound portion of each image frame using a wavelet transformation to generate a transformed image;

means for locating, on the transformed images, nodes associated with distinguishing features of the object defined by wavelet jets of a bunch graph generated from a plurality of representative object images;

means for identifying the object based on a similarity between wavelet jets associated with an object image in a gallery of object images and wavelet jets at the nodes on the transformed images.

35. Apparatus for recognizing objects as defined in claim 34, wherein the bound portion covers less than ten percent of each image frame.

36. Apparatus for recognizing objects as defined in claim 34, wherein the means for detecting the object includes a neural network.

37. Apparatus for recognizing objects as defined in claim 34, wherein the means for detecting the object includes means for detecting a color associated with the object.

38. Apparatus for recognizing objects as defined in claim 34, wherein the means for detecting an object further comprises means for tracking the object between image frames based on a trajectory associated with the object.

39. Apparatus for recognizing objects as defined in claim 34, wherein the means for locating the nodes includes means for tracking the nodes between image frames.

40. Apparatus for recognizing objects as defined in claim 39, further comprising means for reinitializing a tracked node if the node'"'"'s position deviates beyond a predetermined position constraint between image frames.

41. Apparatus for recognizing objects as defined in claim 40, wherein the predetermined position constraint is based on a geometrical position constraint associated with relative positions between the node locations.

42. Apparatus for recognizing objects as defined in claim 34, wherein the means for transforming uses a hardware adapted phase representation for performing phase calculations.