Self learning face recognition using depth based tracking for database generation and update

• US 8,855,369 B2
• Filed: 06/22/2012
• Issued: 10/07/2014
• Est. Priority Date: 06/22/2012
• Status: Active Grant

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Kikkeri, Harshavardhana Narayana, Koenig, Michael F., Cole, Jeffrey
• Primary Examiner(s): NAKHJAVAN, SHERVIN K
• Application Number:
  US13/530,925
• Publication Number:
  US 20130343600A1
• Time in Patent Office: 837 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks
• CPC Class Codes: G06V 40/50 Maintenance of biometric data or enrolment thereof

Abstract

Face recognition training database generation technique embodiments are presented that generally involve collecting characterizations of a person'"'"'s face that are captured over time and as the person moves through an environment, to create a training database of facial characterizations for that person. As the facial characterizations are captured over time, they are will represent the person'"'"'s face as viewed from various angles and distances, different resolutions, and under different environmental conditions (e.g., lighting and haze conditions). Further, over a long period of time where facial characterizations of a person are collected periodically, these characterizations can represent an evolution in the appearance of the person. This produces a rich training resource for use in face recognition systems. In addition, since a person'"'"'s face recognition training database can be established before it is needed by a face recognition system, once employed, the training will be quicker.

Claims

1. A computer-implemented process for generating a face recognition training database for each person detected as being located in an environment, comprising:

using a computer to perform the following process actions;

(a) inputting a sequence of contemporaneously-captured frame pairs, each frame pair comprising a frame output from a color video camera and a frame output from a depth video camera;

(b) using a face detection method and the color video camera frames to detect potential persons in a environment;

(c) using a motion detection method and the depth video camera frames to detect potential persons in the environment;

(d) using detection results generated via the face detection method and motion detection method to determine the location of one or more persons in the environment, said detection results generated via the face detection method comprising for each person detected, a facial characterization of the portion of a color video camera frame depicting the person'"'"'s face;

(e) for each person detected solely via the motion detection method,

identifying the corresponding location of that person in the contemporaneously-captured frame of the color video camera,

generating said facial characterization of the portion of the color video camera frame depicting that person'"'"'s face;

(f) for each person detected in the environment,

assigning each facial characterization generated for that person to an unknown person identifier established for the person,

storing each of said facial characterizations in a memory associated with the computer,

attempting to ascertain the identity of the person, and

whenever the identity of the person is ascertained, re-assigning each facial characterization assigned to the unknown person identifier established for the person to a face recognition training database established for that person.

2. The process of claim 1, further comprising:

inputting a new sequence of contemporaneously-captured frame pairs, each frame pair comprising a frame output from a color video camera and a frame output from a depth video camera;

repeating process actions (b) through (e);

for each person detected in the environment and depicted in the new sequence of contemporaneously-captured frame pairs,

determining if the person corresponds to a person whose location was previously determined using a sequence of contemporaneously-captured frame pairs preceding said new sequence,

whenever it is determined that the person corresponds a person whose location was previously determined using a sequence of contemporaneously-captured frame pairs preceding said new sequence, determining if the identity of the person was previously ascertained,

whenever it is determined the identity of the person was previously ascertained, for each facial characterization generated from said new sequence of contemporaneously-captured frame pairs, determining if the facial characterization differs to a prescribed degree from each facial characterization assigned to the face recognition training database established for the person,

 

for each facial characterization generated from said new sequence of contemporaneously-captured frame pairs, whenever it is determined that the facial characterization differs to a prescribed degree from each facial characterization assigned to the face recognition training database established for the person, assigning that facial characterization to the face recognition training database established for the person and storing that facial characterization in a memory associated with the computer.

3. The process of claim 2, wherein the color video camera and depth video camera are disposed on a mobile robotic device which is capable of moving about the environment, and wherein the new sequence of contemporaneously-captured frame pairs was captured from a point of view within the environment that is different from the point of view that previous-captured frame pairs were captured, said new point of view being achieved by at least one of, the mobile robotic device changing the direction the color and depth video cameras are pointing without changing location within the environment, or the mobile robotic device changing location within the environment.

4. The process of claim 1, further comprising:

inputting a new sequence of contemporaneously-captured frame pairs, each frame pair comprising a frame output from a color video camera and a frame output from a depth video camera;

repeating process actions (b) through (e);

for each person detected in the environment and depicted in the new sequence of contemporaneously-captured frame pairs,

determining if the person corresponds to a person whose location was previously determined using a sequence of contemporaneously-captured frame pairs preceding said new sequence,

whenever it is determined that the person corresponds a person whose location was previously determined using a sequence of contemporaneously-captured frame pairs preceding said new sequence, determining if the identity of the person was previously ascertained,

whenever it is determined the identity of the person was not previously ascertained, for each facial characterization generated from said new sequence of contemporaneously-captured frame pairs, determining if the facial characterization differs to a prescribed degree from each facial characterization assigned to the unknown person identifier established for the person,

 

for each facial characterization generated from said new sequence of contemporaneously-captured frame pairs, whenever it is determined that the facial characterization differs to a prescribed degree from each facial characterization assigned to the unknown person identifier established for the person,

 

assigning that facial characterization to the unknown person identifier established for the person and storing that facial characterization in a memory associated with the computer,

 

attempting to ascertain the identity of the person, and

 

whenever the identity of the person is ascertained, re-assigning each facial characterization assigned to the unknown person identifier established for the person to a face recognition training database established for that person.

5. The process of claim 4, wherein the color video camera and depth video camera are disposed on a mobile robotic device which is capable of moving about the environment, and wherein the new sequence of contemporaneously-captured frame pairs was captured from a point of view within the environment that is different from the point of view that previous-captured frame pairs were captured, said new point of view being achieved by at least one of, the mobile robotic device changing the direction the color and depth video cameras are pointing without changing location within the environment, or the mobile robotic device changing location within the environment.

6. The process of claim 1, further comprising:

inputting a new sequence of contemporaneously-captured frame pairs, each frame pair comprising a frame output from a color video camera and a frame output from a depth video camera;

repeating process actions (b) through (e);

for each person detected in the environment and depicted in the new sequence of contemporaneously-captured frame pairs,

determining if the person corresponds to a person whose location was previously determined using a sequence of contemporaneously-captured frame pairs preceding said new sequence,

whenever it is determined that the person corresponds a person whose location was previously determined using a sequence of contemporaneously-captured frame pairs preceding said new sequence, determining if the identity of the person was previously ascertained,

whenever it is determined the identity of the person was not previously ascertained, determining how many times a sequence of contemporaneously-captured frame pairs had been input and processed without the identity of the person being ascertained, and determining if the number of times exceeds a prescribed maximum number,

 

whenever it is determined that the number of times a sequence of contemporaneously-captured frame pairs had been input and processed without the identity of the person being ascertained exceeds the prescribed maximum number, deleting from said memory each facial characterization assigned to the unknown person identifier established for the person.

7. The process of claim 1, further comprising:

inputting a new sequence of contemporaneously-captured frame pairs, each frame pair comprising a frame output from a color video camera and a frame output from a depth video camera;

repeating process actions (b) through (e);

for each person detected in the environment and depicted in the new sequence of contemporaneously-captured frame pairs,

determining if the person corresponds to a person whose location was previously determined using a sequence of contemporaneously-captured frame pairs preceding said new sequence,

whenever it is determined that the person does not correspond a person whose location was previously determined using a sequence of contemporaneously-captured frame pairs preceding said new sequence,

assigning each facial characterization generated for that person to an unknown person identifier established for the person,

storing each of said facial characterizations in a memory associated with the computer,

attempting to ascertain the identity of the person, and

whenever the identity of the person is ascertained, re-assigning each facial characterization assigned to the unknown person identifier established for the person to a face recognition training database established for that person.

8. The process of claim 7, wherein the color video camera and depth video camera are disposed on a mobile robotic device which is capable of moving about the environment, and wherein the new sequence of contemporaneously-captured frame pairs was captured from a point of view within the environment that is different from the point of view that previous-captured frame pairs were captured, said new point of view being achieved by at least one of, the mobile robotic device changing the direction the color and depth video cameras are pointing without changing location within the environment, or the mobile robotic device changing location within the environment.

9. The process of claim 1, further comprising performing, prior to performing the process action of assigning each facial characterization generated for the person to an unknown person identifier established for the person, the process actions of:

for each person detected at a distance from the color video camera that exceeds a prescribed maximum distance,

providing the location of the person to a controller that controls a color camera having zoom capability, said controller being capable of, based on the person'"'"'s location, zooming in on the face of the person to a degree proportional to the distance from the color video camera to the person, and capturing a zoomed image of the person'"'"'s face,

inputting said zoomed image of the person'"'"'s face, and

generating said facial characterization of the portion of the zoomed image depicting that person'"'"'s face.

10. The process of claim 1, wherein the process action of attempting to ascertain the identity of the person, comprises an action of employing the facial characterizations generated for that person and assigned to an unknown person identifier established for the person in attempting to ascertain the identity of the person.

11. The process of claim 1, further comprising the process actions of:

inputting an additional sequence of contemporaneously-captured frame pairs, each additional frame pair comprising a frame output from an additional color video camera and a frame output from an additional depth video camera, said additional color and depth video cameras capturing the same scene in the environment as the other color and depth video cameras but from a different point of view and each additional frame pair being captured substantially contemporaneously with a frame pair output from the other color and depth video cameras;

using a face detection method and the frames from the additional color video camera to detect potential persons in a environment;

using a motion detection method and the frames from the additional depth video camera to detect potential persons in the environment;

using detection results generated via the face detection method and motion detection method to determine the location of one or more persons in the environment, said detection results generated via the face detection method comprising for each person detected, a facial characterization of the portion of the color video camera frame depicting the person'"'"'s face;

for each person detected solely via the motion detection method,

identifying the corresponding location of that person in the contemporaneously-captured frame of the additional color video camera,

generating said facial characterization of the portion of the additional color video camera frame depicting that person'"'"'s face;

for each person detected in the environment based on frame pairs output from the additional color video camera and additional depth video camera,

determining based on identified location of the person whether the person has also been detected using the other color and depth video cameras,

whenever it is determined that the person has also been detected using the other color and depth video cameras, assigning each facial characterization generated for that person based on frame pairs output from the additional color video camera and additional depth video camera to the unknown person identifier established for that person based on the person'"'"'s detection using the other color and depth video cameras, and storing each of said facial characterizations generated for the person based on frame pairs output from the additional color video camera and additional depth video camera in the memory associated with the computer,

whenever it is determined that the person has not also been detected using the other color and depth video cameras, assigning each facial characterization generated for that person based on frame pairs output from the additional color video camera and additional depth video camera to an unknown person identifier established for that person, attempting to ascertain the identity of the person, and whenever the identity of the person is ascertained, re-assigning each facial characterization assigned to the unknown person identifier established for the person to a face recognition training database established for that person.

12. The process of claim 1, further comprising the process actions of:

inputting an additional sequence of contemporaneously-captured frame pairs, each additional frame pair comprising a frame output from an additional color video camera and a frame output from an additional depth video camera, said additional color and depth video cameras capturing a different scene in the environment than the other color and depth video cameras;

using a face detection method and the frames from the additional color video camera to detect potential persons in a environment;

using a motion detection method and the frames from the additional depth video camera to detect potential persons in the environment;

using detection results generated via the face detection method and motion detection method to determine the location of one or more persons in the environment, said detection results generated via the face detection method comprising for each person detected, a facial characterization of the portion of the color video camera frame depicting the person'"'"'s face;

for each person detected solely via the motion detection method,

identifying the corresponding location of that person in the contemporaneously-captured frame of the additional color video camera,

generating said facial characterization of the portion of the additional color video camera frame depicting that person'"'"'s face;

for each person detected in the environment based on frame pairs output from the additional color video camera and additional depth video camera,

determining if the person detected was previously detected in a different scene in the environment,

if the person was previously detected in a different scene in the environment, determining if the identity of the person was previously ascertained,

whenever it is determined the identity of the person was not previously ascertained, for each facial characterization generated from said additional sequence of contemporaneously-captured frame pairs, determining if the facial characterization differs to a prescribed degree from each facial characterization assigned to the unknown person identifier previously established for the person,

for each facial characterization generated from said additional sequence of contemporaneously-captured frame pairs, whenever it is determined that the facial characterization differs to a prescribed degree from each facial characterization assigned to the unknown person identifier previously established for the person, assigning that facial characterization to the unknown person identifier previously established for the person and storing that facial characterization in a memory associated with the computer,

attempting to ascertain the identity of the person, and

whenever the identity of the person is ascertained, re-assigning each facial characterization assigned to the unknown person identifier established for the person to a face recognition training database established for that person,

if the person was not previously detected in a different scene in the environment,

assigning each facial characterization generated for that person based on frame pairs output from the additional color video camera and additional depth video camera to an unknown person identifier newly established for that person,

storing each of said facial characterizations generated for that person based on frame pairs output from the additional color video camera and additional depth video camera in the memory associated with the computer,

attempting to ascertain the identity of the person, and

whenever the identity of the person is ascertained, re-assigning each facial characterization assigned to the unknown person identifier established for the person to a face recognition training database established for that person.

13. The process of claim 1, further comprising the process actions of:

inputting an additional sequence of contemporaneously-captured frame pairs, each additional frame pair comprising a frame output from an additional color video camera and a frame output from an additional depth video camera, said additional color and depth video cameras capturing a different scene in the environment than the other color and depth video cameras;

using a face detection method and the frames from the additional color video camera to detect potential persons in a environment;

using a motion detection method and the frames from the additional depth video camera to detect potential persons in the environment;

using detection results generated via the face detection method and motion detection method to determine the location of one or more persons in the environment, said detection results generated via the face detection method comprising for each person detected, a facial characterization of the portion of the color video camera frame depicting the person'"'"'s face;

for each person detected solely via the motion detection method,

identifying the corresponding location of that person in the contemporaneously-captured frame of the additional color video camera,

generating said facial characterization of the portion of the additional color video camera frame depicting that person'"'"'s face;

for each person detected in the environment based on frame pairs output from the additional color video camera and additional depth video camera,

determining if the person detected was previously detected in a different scene in the environment,

if the person was previously detected in a different scene in the environment,

determining if the identity of the person was previously ascertained,

 

whenever it is determined the identity of the person was previously ascertained, for each facial characterization generated from said additional sequence of contemporaneously-captured frame pairs, determining if the facial characterization differs to a prescribed degree from each facial characterization assigned to the face recognition training database established for the person,

 

for each facial characterization generated from said additional sequence of contemporaneously-captured frame pairs, whenever it is determined that the facial characterization differs to a prescribed degree from each facial characterization assigned to the face recognition training database established for the person, assigning that facial characterization to the face recognition training database established for the person and storing that facial characterization in a memory associated with the computer.

14. The process of claim 1, wherein the process action of using a motion detection method and the depth video camera frames to detect potential persons in the environment, comprises the actions of:

designating all the pixels in the first depth video camera frame as background pixels;

for each pixel of each of the subsequently-captured depth frames contained in the sequence of contemporaneously-captured frame pairs, in the order in which the frame was captured;

identifying if the depth value of the pixel has changed more than a prescribed amount from the value of a pixel in the depth frame captured immediately before the frame currently under consideration that represents the same location within the environment;

whenever the depth value of the pixel has changed more than the prescribed amount, designating the pixel to be a foreground pixel;

once the last frame contained in the sequence of contemporaneously-captured frame pairs has been processed to identify if its pixel depth values have changed more than the prescribed amount,

(i) establishing a seed point amongst the foreground pixels in said last frame and assigning the pixel associated therewith to be a part of a separate blob,

(ii) recursively determining for each pixel neighboring a pixel assigned to the blob, which is not already assigned to that blob, if its depth value is the same within a prescribed tolerance as the current average of the pixels assigned to the blob, and if so, assigning that neighboring pixel to be a part of the blob, until no neighboring pixel can be found that is unassigned to a blob and which has a depth value that is the same within said prescribed tolerance of the current average of the pixels assigned to the blob, and

(iii) whenever a neighboring pixel is found during the performance of the recursive determining action (ii) that is assigned to a different blob, combining the two blobs into one and continuing the recursive determining action (ii), and

(iv) repeating process actions (i) through (iii) for the unassigned foreground pixels, until no more blobs can be formed

once no more blobs can be formed, for each blob,

determining if the blob meets a set of prescribed criteria that is indicative of the blob representing a human,

eliminating each blob not meeting the set of prescribed criteria, and

designating each remaining blob to represent a different potential person located within the environment.

15. The process of claim 1, further comprising:

inputting a new sequence of frames output from the color video camera;

repeating process actions (b) through (e);

for each person detected in the environment and depicted in the new sequence of frames output from a color video camera,

determining if the person corresponds to a person whose location was previously determined using a sequence of color video camera frames which were capture before said new sequence,

whenever it is determined that the person corresponds a person whose location was previously determined, ascertaining if the identity of the person was previously ascertained,

whenever it is ascertained the identity of the person was previously ascertained, for each facial characterization generated from said new sequence of frames, determining if the facial characterization differs to a prescribed degree from each facial characterization assigned to the face recognition training database established for the person,

 

for each facial characterization generated from said new sequence of frames, whenever it is determined that the facial characterization differs to a prescribed degree from each facial characterization assigned to the face recognition training database established for the person, assigning that facial characterization to the face recognition training database established for the person and storing that facial characterization in a memory associated with the computer.

16. The process of claim 1, further comprising:

inputting a new sequence of frames output from the color video camera;

repeating process actions (b) through (e);

for each person detected in the environment and depicted in the new sequence of frames output from the color video camera,

determining if the person corresponds a person whose location was previously determined using a sequence of color video camera frames which were capture before said new sequence,

whenever it is determined that the person corresponds a person whose location was previously determined, determining if the identity of the person was previously ascertained,

whenever it is ascertained the identity of the person was not previously ascertained, for each facial characterization generated from said new sequence of frames, determining if the facial characterization differs to a prescribed degree from each facial characterization assigned to the unknown person identifier established for the person,

 

or each facial characterization generated from said new sequence of frames, whenever it is determined that the facial characterization differs to a prescribed degree from each facial characterization assigned to the unknown person identifier established for the person,

 

assigning that facial characterization to the unknown person identifier established for the person and storing that facial characterization in a memory associated with the computer,

 

attempting to ascertain the identity of the person, and

 

whenever the identity of the person is ascertained, re-assigning each facial characterization assigned to the unknown person identifier established for the person to a face recognition training database established for that person.

17. The process of claim 1, further comprising:

inputting a new sequence of frames output from the color video camera;

repeating process actions (b) through (e);

for each person detected in the environment and depicted in the new sequence of frames output from the color video camera,

determining if the person corresponds a person whose location was previously determined using a sequence of color video camera frames which were capture before said new sequence,

whenever it is determined that the person does not correspond a person whose location was previously determined,

assigning each facial characterization generated for that person to an unknown person identifier established for the person,

storing each of said facial characterizations in a memory associated with the computer,

attempting to ascertain the identity of the person, and

whenever the identity of the person is ascertained, re-assigning each facial characterization assigned to the unknown person identifier established for the person to a face recognition training database established for that person.

18. A computer-implemented process for generating a face recognition training database for each person detected as being located in an environment, comprising:

using a computer to perform the following process actions;

(a) inputting a sequence of frames output from a color video camera;

(b) using a face detection method and the color video camera frames to detect potential persons in a environment;

(c) using a motion detection method and the color video camera frames to detect potential persons in a environment;

(d) using detection results generated via the face detection method and motion detection method to determine the location of one or more persons in the environment, said detection results generated via the face detection method comprising for each person detected, a facial characterization of the portion of a color video camera frame depicting the person'"'"'s face;

(e) for each person detected solely via the motion detection method,

locating a portion of a color video camera frame depicting that person'"'"'s face, and

generating said facial characterization of the portion of the color video camera frame depicting that person'"'"'s face;

(f) for each person detected in the environment,

assigning each facial characterization generated for that person to an unknown person identifier established for the person,

storing each of said facial characterizations in a memory associated with the computer,

attempting to ascertain the identity of the person, and

whenever the identity of the person is ascertained, re-assigning each facial characterization assigned to the unknown person identifier established for the person to a face recognition training database established for that person.

19. A computer-implemented process for detecting persons located in an environment, comprising:

using a computer to perform the following process actions;

inputting a sequence of frames output from a depth video camera;

designating all the pixels in the first depth video camera frame as background pixels;

for each pixel of each of the subsequently-captured depth frames contained in the sequence of frames, in the order in which the frame was captured;

identifying if the depth value of the pixel has changed more than a prescribed amount from the value of a pixel in the depth frame captured immediately before the frame currently under consideration that represents the same location within the environment;

whenever the depth value of the pixel has changed more than the prescribed amount, designating the pixel to be a foreground pixel;

once the last frame contained in the sequence of frames has been processed to identify if its pixel depth values have changed more than the prescribed amount,

(i) establishing a seed point amongst the foreground pixels in said last frame and assigning the pixel associated therewith to be a part of a separate blob,

(ii) recursively determining for each pixel neighboring a pixel assigned to the blob, which is not already assigned to that blob, if its depth value is the same within a prescribed tolerance as the current average of the pixels assigned to the blob, and if so, assigning that neighboring pixel to be a part of the blob, until no neighboring pixel can be found that is unassigned to a blob and which has a depth value that is the same within said prescribed tolerance of the current average of the pixels assigned to the blob, and

(iii) whenever a neighboring pixel is found during the performance of the recursive determining action (ii) that is assigned to a different blob, combining the two blobs into one and continuing the recursive determining action (ii), and

(iv) repeating process actions (i) through (iii) for the unassigned foreground pixels, until no more blobs can be formed,

once no more blobs can be formed, for each blob,

determining if the blob meets a set of prescribed criteria that is indicative of the blob representing a human,

eliminating each blob not meeting the set of prescribed criteria, and

designating each remaining blob to represent a different potential person located within the environment.

20. The process of claim 19, wherein the process action of determining if a blob meets a set of prescribed criteria that is indicative of the blob representing a human, comprises determining if at least a portion of the blob exhibits a substantially rectangular shape.