VIDEO-BASED FACE RECOGNITION USING PROBABILISTIC APPEARANCE MANIFOLDS

• US 20090041310A1
• Filed: 11/06/2003
• Published: 02/12/2009
• Est. Priority Date: 11/07/2002
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Lee, Kuang-Chih, Yang, Ming-Hsuan, Ho, Jeffrey
• Granted Patent: US 7,499,574 B1
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 10/242 by image rotation, e.g. by 90 degrees
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 40/169 Holistic features and representations, i.e. based on the facial image taken as a whole
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

The present invention meets these needs by providing temporal coherency to recognition systems. One embodiment of the present invention comprises a manifold recognition module to use a sequence of images for recognition. A manifold training module receives a plurality of training image sequences (e.g. from a video camera), each training image sequence including an individual in a plurality of poses, and establishes relationships between the images of a training image sequence. A probabilistic identity module receives a sequence of recognition images including a target individual for recognition, and identifies the target individual based on the relationship of training images corresponding to the recognition images. An occlusion module masks occluded portions of an individual'"'"'s face to prevent distorted identifications.

Claims

1. A method for recognizing a target individual, comprising the steps of:

receiving a first recognition image from a sequence of recognition images, said first recognition image including a first image of the target individual in a first pose, the first recognition image captured at a first time;

receiving a second recognition image of the sequence of recognition images, said second recognition image including a second image of the target individual in a second pose, the second recognition image captured at a second time;

generating a first identification information comprising a candidate identity of the target individual based on the first image; and

generating a second identification information comprising an updated identity of the target individual based on the first identification information and a likelihood that the second pose will follow the first pose in the sequence of recognition images.

3. The method of claim 1, wherein the second pose is a facial position having a two-dimensional rotation with respect to the first pose.

4. The method of claim 1, wherein the second pose is a facial position having a three-dimensional rotation with respect to the first pose.

5. The method of claim 1, wherein the step of generating the second identification information comprise generating the second identification information based on a conditional probability that defines a relationship between the first identification information and the second image.

6. The method of claim 1, wherein the step of generating the first identification information comprises the steps of:

generating a first vector representation of the first image; and

projecting the first vector representation onto a plurality of vector representations representing a plurality of individuals; and

selecting one or more vector representations from the plurality of vector representations having the least distance from the first vector representation.

7. The method of claim 6, wherein the plurality of vector representations are pose manifolds associated with the plurality of individuals.

8. The method of claim 6, further comprising the steps of:

generating the plurality of vector representations from a plurality of training image sequences including the plurality of individuals.

9. The method of claim 6, wherein the plurality of vector representations are linear approximations of a plurality of image sequences including the plurality of individuals.

10. The method of claim 1, further comprising the steps of:

if the second identification information exceeds an identification threshold for an individual, identifying the target individual as the individual.

11. The method of claim 1, further comprising the steps of:

receiving a third recognition image at a third time, the third recognition image including a third image of the target individual at a third time; and

generating a third identification information, said identification information based on the third image and the second identification information.

2. (canceled)

12. A method for recognizing a target individual, comprising the steps of:

receiving a first recognition image from a sequence of recognition images, said first recognition image including a first image of the target individual at a first time;

receiving a second recognition image of the sequence of recognition images, said second recognition image including a second image of the target individual at a second time;

generating a first identification information from the first image;

detecting that a portion of the second recognition image is at least partially occluded;

generating a weighted mask including the portion of the second image; and

generating a second identification information from the second image, as adjusted by the weighted mask, and the first identification information.

13. The method of claim 12, wherein the step of detecting the occlusion further comprises the step of:

determining that a group of pixels in the second image exceeds a variance threshold.

14. The method of claim 13, wherein the step of determining that the group of pixels in the second image exceeds a variance threshold comprises:

determining that a first group of pixels in the second image exceeds a variance threshold with respect to a corresponding group of pixels in the first image.

15. A method for recognizing a target individual from an image, comprising the steps of:

receiving a first identification information comprising a candidate identity of the target individual based on one or more recognition images at a previous time;

generating a second identification information comprising an updated identity of the target individual based on a recognition image at a current time; and

determining an identification of the target individual from a plurality of individuals based on a conditional probability given the first identification information and the recognition image at the current time.

16-17. -17. (canceled)

18. A computer program product, comprising:

a computer-readable medium having computer program instructions and data embodied thereon for recognizing a target individual from a plurality of individuals, comprising the steps of;

receiving a first recognition image from a sequence of recognition images, said first recognition image including a first image of the target individual in a first pose, the first recognition image captured at a first time;

receiving a second recognition image of the sequence of recognition images, said second recognition image including a second image of the target individual in a second pose, the second recognition image captured at a second time;

generating a first identification information comprising a candidate identity of the target individual from the first image; and

generating a second identification information comprising an updated identity of the target individual based on the first identification information and a likelihood that the second pose will follow the first pose in the sequence of recognition images.

20. The computer program product of claim 18, wherein the second pose is a facial position having a two-dimensional rotation with respect to the first pose.

21. The computer program product of claim 18, wherein the second pose is a facial position having a three-dimensional rotation with respect to the first pose.

22. The computer program product of claim 18, wherein the step of generating the second identification information comprises generating the second identification information based on a conditional probability that defines a relationship between the first identification information and the second image.

23. The computer program product of claim 18, wherein the step of generating the first identification information comprises the steps of:

generating a first vector representation of the first image; and

projecting the first vector representation onto a plurality of vector representations representing a plurality of individuals; and

selecting one or more vector representations from the plurality of vector representations having the least distance from the first vector representation.

24. The computer program product of claim 23, wherein the plurality of vector representations are pose manifolds associated with the plurality of individuals.

25. The computer program product of claim 23, further comprising the steps of:

generating the plurality of vector representations from a plurality of training image sequences including the plurality of individuals.

26. The computer program product of claim 23, wherein the plurality of vector representations are linear approximations of a plurality of image sequences including the plurality of individuals.

27. The computer program product of claim 18, further comprising the steps of:

if the second identification information exceeds an identification threshold for an individual, identifying the target individual as the individual.

28. The computer program product of claim 18, further comprising the steps of:

receiving a third recognition image at a third time, the third recognition image including a third image of the target individual at a third time; and

generating a third identification information, said third identification information based on the third image and the updated identification information.

19. (canceled)

29. A computer program product, comprising:

a computer-readable medium having computer program instructions and data embodied thereon for recognizing a target individual from a plurality of individuals, comprising the steps of;

receiving a first recognition image from a sequence of recognition images, said first recognition image including a first image of the target individual at a first time;

receiving a second recognition image of the sequence of recognition images, said second recognition image including a second image of the target individual at a second time;

generating a first identification information from the first image;

detecting that a portion of the second recognition image is at least partially occluded;

generating a weighted mask including the portion of the second image; and

generating the second identification information from the second image, as adjusted by the weighted mask, and the first identification information.

30. The computer program product of claim 29, wherein the step of detecting the occlusion further comprises the step of:

determining that a group of pixels in the second image exceeds a variance threshold.

31. The computer program product of claim 30, wherein the step of determining that the group of pixels in the second image exceeds a variance threshold comprises:

determining that a first group of pixels in the second image exceeds a variance threshold with respect to a corresponding group of pixels in the first image.

32. A recognition module for recognizing a target individual, comprising:

a video buffer to receive a first recognition image from a sequence of recognition images, said first recognition image including a first image of the target individual in a first pose, the first recognition image captured at a first time, and a second recognition image of the sequence of recognition images, said second recognition image including a second image of the target individual in a second pose, the second recognition image captured at a second time;

an identity module to generate a first identification information comprising a candidate identity of the target individual based on the first image, and generate a second identification information comprising an updated identity of the target individual based on the first identification information and a likelihood that the second pose will follow the first pose in the sequence of recognition images.

34. The recognition module of claim 32, wherein the second pose is a facial position having a two-dimensional rotation with respect to the first pose.

35. The recognition module of claim 32, wherein the second pose is a facial position having a three-dimensional rotation with respect to the first pose.

36. The recognition module of claim 32, wherein the identity module further comprises a transition module to define a relationship between the first identification information and the second image, and the identity module generates the second identification information based on the relationship.

37. The recognition module of claim 32, wherein the identity module further comprises an identification control module to generate a first vector representation of the first image, project the first vector representation onto a plurality of vector representations representing a plurality of individuals, and select one or more vector representations from the plurality of vector representations having the least distance from the first vector representation.

38. The recognition module of claim 37, wherein the plurality of vector representations are pose manifolds associated with the plurality of individuals.

39. The recognition module of claim 37, further comprising a linear approximation module to generate the plurality of vector representations from a plurality of training image sequences including the plurality of individuals.

40. The recognition module of claim 37, wherein the plurality of vector representations are linear approximations of a plurality of image sequences including the plurality of individuals.

41. The recognition module of claim 32, wherein if the second identification information exceeds an identification threshold for an individual, the identify module identifies the target individual as the individual.

42. The recognition module of claim 32, wherein the video buffer receives a third recognition image, said third recognition image including a third image of the target individual at a third time, and the identity module generates a third identification information, said third identification information based on the third image and the second identification information.

33. (canceled)

43. A recognition module for recognizing a target individual, comprising:

a video buffer to receive a first recognition image from a sequence of recognition images, said first recognition image including a first image of the target individual at a first time, and a second recognition image of the sequence of recognition images, said second recognition image including a second image of the target individual at a second time;

an identity module to generate a first identification information from the first image;

an occlusion module to detect that a portion of the second recognition image is at least partially occluded;

a mask generation module to generate a weighted mask including the portion; and

a mask adjustment module to generate the second identification information based on the second recognition image, as adjusted by the weighted mask, and the first identification information.

44. The recognition module of claim 43, wherein the mask generation means determines that a group of pixels in the second image exceeds a variance threshold.

45. The recognition module of claim 44, wherein the mask generation module determines that a first group of pixels in the second image exceeds a variance threshold with respect to a corresponding group of pixels in the first image.

46. A recognition module for recognizing a target individual, comprising:

a buffer means for receiving a first recognition image from a sequence of recognition images, said first recognition image including a first image of the target individual in a first pose, the first recognition image captured at a first time, and a second recognition image of the sequence of recognition images, said second recognition image including a second image of the target individual in a second pose, the second recognition image captured at a second time;

an identity means for generating a first identification information comprising a candidate identity of the target individual based on the first image, and generate a second identification information comprising an updated identity of the target individual based on the first identification information and a likelihood that the second pose will follow the first pose in the sequence of recognition images.

48. The recognition module of claim 46, wherein the second pose is a facial position having a two-dimensional rotation with respect to the first pose.

49. The recognition module of claim 46, wherein the second pose is a facial position having a three-dimensional rotation with respect to the first pose.

50. The recognition module of claim 46, wherein the identity means further comprises a transition means to define a relationship between the first identification information and the second image, and the identity means generates the second identification information based on the relationship.

51. The recognition module of claim 46, wherein the identity module further comprises an identification control means for generating a first vector representation of the first image, projecting the first vector representation onto a plurality of vector representations representing a plurality of individuals, and selecting one or more vector representations from the plurality of vector representations having the least distance from the first vector representation.

52. The recognition module of claim 51, wherein the plurality of vector representations are pose manifolds associated with the plurality of individuals.

53. The recognition module of claim 51, further comprising a linear approximation means for generating the plurality of vector representations from a plurality of training image sequences including the plurality of individuals.

54. The recognition module of claim 51, wherein the plurality of vector representations are linear approximations of a plurality of image sequences including the plurality of individuals.

55. The recognition module of claim 46, wherein if the second identification information exceeds an identification threshold for an individual, the identify module identifies the target individual as the individual.

56. The recognition module of claim 46, wherein the buffer means receives a third recognition image, the third recognition image including a third image of the target individual at a third time, and the identity means generates a third identification information, said third identification information based on the third image and the second identification information.

47. (canceled)

57. A recognition module for recognizing a target individual, comprising:

a buffer means for receiving a first recognition image from a sequence of recognition images, said first recognition image including a first image of the target individual at a first time, and a second recognition image of the sequence of recognition images, said second recognition image including a second image of the target individual at a second time;

an identity means for generating a first identification information from the first image;

an occlusion means for detecting that a portion of the second recognition image is at least partially occluded;

a mask generation means for generating a weighted mask including the portion of the second image; and

a mask adjustment means for generating the second identification information from the second image, as adjusted by the weighted mask, and the first identification information.

58. The recognition module of claim 57, wherein the mask generation means determines that a group of pixels in the second image exceeds a variance threshold.

59. The recognition module of claim 58, wherein the mask generation means determines that a first group of pixels in the second image exceeds a variance threshold with respect to a corresponding group of pixels in the first image.