Face recognition using depth based tracking

• US 9,317,762 B2
• Filed: 10/07/2014
• Issued: 04/19/2016
• Est. Priority Date: 06/22/2012
• Status: Active Grant

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kikkeri, Harshavardhana Narayana, Koenig, Michael F., Cole, Jeffrey
• Primary Examiner(s): NAKHJAVAN, SHERVIN K
• Application Number:
  US14/507,956
• Publication Number:
  US 20150092986A1
• Time in Patent Office: 560 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks
• CPC Class Codes: G06V 40/50 Maintenance of biometric data or enrolment thereof

Abstract

Face recognition training database generation technique embodiments are presented that generally involve collecting characterizations of a person'"'"'s face that are captured over time and as the person moves through an environment, to create a training database of facial characterizations for that person. As the facial characterizations are captured over time, they are will represent the person'"'"'s face as viewed from various angles and distances, different resolutions, and under different environmental conditions (e.g., lighting and haze conditions). Further, over a long period of time where facial characterizations of a person are collected periodically, these characterizations can represent an evolution in the appearance of the person. This produces a rich training resource for use in face recognition systems. In addition, since a person'"'"'s face recognition training database can be established before it is needed by a face recognition system, once employed, the training will be quicker.

Claims

1. A computer-implemented process for locating one or more persons in an environment, comprising:

using a computer to perform the following process actions;

inputting a sequence of contemporaneously-captured frame pairs, each frame pair comprising a frame output from a color video camera and a frame output from a depth video camera;

using a face detection method and the color video camera frames to detect potential persons in a environment;

using a motion detection method and the depth video camera frames to detect potential persons in the environment; and

using detection results generated via the face detection method and motion detection method to determine the location of one or more persons in the environment.

2. The process of claim 1, wherein the process action of using a motion detection method and the depth video camera frames to detect potential persons in the environment, comprises the actions of:

designating all the pixels in the first depth video camera frame as background pixels;

for each pixel of each of the subsequently-captured depth frames contained in the sequence of contemporaneously-captured frame pairs, in the order in which the frame was captured;

identifying if the depth value of the pixel has changed more than a prescribed amount from the value of a pixel in the depth frame captured immediately before the frame currently under consideration that represents the same location within the environment;

whenever the depth value of the pixel has changed more than the prescribed amount, designating the pixel to be a foreground pixel;

once the last frame contained in the sequence of contemporaneously-captured frame pairs has been processed to identify if its pixel depth values have changed more than the prescribed amount,

(i) establishing a seed point amongst the foreground pixels in said last frame and assigning the pixel associated therewith to be a part of a separate blob,

(ii) recursively determining for each pixel neighboring a pixel assigned to the blob, which is not already assigned to that blob, if its depth value is the same within a prescribed tolerance as the current average of the pixels assigned to the blob, and if so, assigning that neighboring pixel to be a part of the blob, until no neighboring pixel can be found that is unassigned to a blob and which has a depth value that is the same within said prescribed tolerance of the current average of the pixels assigned to the blob, and

(iii) whenever a neighboring pixel is found during the performance of the recursive determining action (ii) that is assigned to a different blob, combining the two blobs into one and continuing the recursive determining action (ii), and

(iv) repeating process actions (i) through (iii) for the unassigned foreground pixels, until no more blobs can be formed

once no more blobs can be formed, for each blob,

determining if the blob meets a set of prescribed criteria that is indicative of the blob representing a human,

eliminating each blob not meeting the set of prescribed criteria, and

designating each remaining blob to represent a different potential person located within the environment.

3. The process of claim 2, wherein the process action of determining if a blob meets a set of prescribed criteria that is indicative of the blob representing a human, comprises determining if the blob fits human body parameters in real space dimensions.

4. The process of claim 2, wherein the process action of determining if a blob meets a set of prescribed criteria that is indicative of the blob representing a human, comprises determining if at least a portion of the blob exhibits a substantially rectangular shape representing a human chest and head.

5. The process of claim 1, wherein the depth video camera is an infrared-based depth video camera.

6. The process of claim 1, further comprising tracking, over time, the location of each of said one or more persons in the environment.

7. The process of claim 1, wherein the color video camera and depth video camera are disposed on a mobile robotic device which is capable of moving about the environment.

8. A system for locating one or more persons in an environment, comprising:

a color video camera;

a depth video camera; and

a computing device which executes program modules of a computer program, the computing device being directed by the program modules of the computer program to,

input a sequence of contemporaneously-captured frame pairs, each frame pair comprising a frame output from a color video camera and a frame output from a depth video camera,

use a face detection method and the color video camera frames to detect potential persons in a environment,

use a motion detection method and the depth video camera frames to detect potential persons in the environment, and

use detection results generated via the face detection method and motion detection method to determine the location of one or more persons in the environment.

9. The system of claim 8, wherein the program module for using a motion detection method and the depth video camera frames to detect potential persons in the environment, comprises sub-modules for:

designating all the pixels in the first depth video camera frame as background pixels;

for each pixel of each of the subsequently-captured depth frames contained in the sequence of contemporaneously-captured frame pairs, in the order in which the frame was captured;

identifying if the depth value of the pixel has changed more than a prescribed amount from the value of a pixel in the depth frame captured immediately before the frame currently under consideration that represents the same location within the environment;

whenever the depth value of the pixel has changed more than the prescribed amount, designating the pixel to be a foreground pixel;

once the last frame contained in the sequence of contemporaneously-captured frame pairs has been processed to identify if its pixel depth values have changed more than the prescribed amount,

(i) establishing a seed point amongst the foreground pixels in said last frame and assigning the pixel associated therewith to be a part of a separate blob,

(ii) recursively determining for each pixel neighboring a pixel assigned to the blob, which is not already assigned to that blob, if its depth value is the same within a prescribed tolerance as the current average of the pixels assigned to the blob, and if so, assigning that neighboring pixel to be a part of the blob, until no neighboring pixel can be found that is unassigned to a blob and which has a depth value that is the same within said prescribed tolerance of the current average of the pixels assigned to the blob, and

(iii) whenever a neighboring pixel is found during the performance of the recursive determining action (ii) that is assigned to a different blob, combining the two blobs into one and continuing the recursive determining action (ii), and

(iv) repeating process actions (i) through (iii) for the unassigned foreground pixels, until no more blobs can be formed

once no more blobs can be formed, for each blob,

determining if the blob meets a set of prescribed criteria that is indicative of the blob representing a human,

eliminating each blob not meeting the set of prescribed criteria, and

designating each remaining blob to represent a different potential person located within the environment.

10. The system of claim 9, wherein the sub-module for determining if a blob meets a set of prescribed criteria that is indicative of the blob representing a human, comprises determining if the blob fits human body parameters in real space dimensions.

11. The system of claim 9, wherein the sub-module for determining if a blob meets a set of prescribed criteria that is indicative of the blob representing a human, comprises determining if at least a portion of the blob exhibits a substantially rectangular shape representing a human chest and head.

12. The system of claim 8, wherein the depth video camera is an infrared-based depth video camera.

13. The system of claim 8, further comprising a program module for tracking, over time, the location of each of said one or more persons in the environment.

14. The system of claim 8, wherein the color video camera and depth video camera are disposed on a mobile robotic device which is capable of moving about the environment.

15. A computer-implemented process for locating one or more persons in an environment, comprising:

using a computer to perform the following process actions;

inputting a sequence of contemporaneously-captured frame pairs, each frame pair comprising a frame output from a color video camera and a frame output from an infrared-based depth video camera, wherein the color video camera and depth video camera are disposed on a mobile robotic device which is capable of moving about the environment;

using a face detection method and the color video camera frames to detect potential persons in a environment;

using a motion detection method and the depth video camera frames to detect potential persons in the environment; and

using detection results generated via the face detection method and motion detection method to determine the location of one or more persons in the environment.

16. The process of claim 15, wherein the process action for using a motion detection method and the depth video camera frames to detect potential persons in the environment, comprises actions for:

designating all the pixels in the first depth video camera frame as background pixels;

for each pixel of each of the subsequently-captured depth frames contained in the sequence of contemporaneously-captured frame pairs, in the order in which the frame was captured;

identifying if the depth value of the pixel has changed more than a prescribed amount from the value of a pixel in the depth frame captured immediately before the frame currently under consideration that represents the same location within the environment;

whenever the depth value of the pixel has changed more than the prescribed amount, designating the pixel to be a foreground pixel;

once the last frame contained in the sequence of contemporaneously-captured frame pairs has been processed to identify if its pixel depth values have changed more than the prescribed amount,

(i) establishing a seed point amongst the foreground pixels in said last frame and assigning the pixel associated therewith to be a part of a separate blob,

(ii) recursively determining for each pixel neighboring a pixel assigned to the blob, which is not already assigned to that blob, if its depth value is the same within a prescribed tolerance as the current average of the pixels assigned to the blob, and if so, assigning that neighboring pixel to be a part of the blob, until no neighboring pixel can be found that is unassigned to a blob and which has a depth value that is the same within said prescribed tolerance of the current average of the pixels assigned to the blob, and

(iii) whenever a neighboring pixel is found during the performance of the recursive determining action (ii) that is assigned to a different blob, combining the two blobs into one and continuing the recursive determining action (ii), and

(iv) repeating process actions (i) through (iii) for the unassigned foreground pixels, until no more blobs can be formed

once no more blobs can be formed, for each blob,

determining if the blob meets a set of prescribed criteria that is indicative of the blob representing a human,

eliminating each blob not meeting the set of prescribed criteria, and

designating each remaining blob to represent a different potential person located within the environment.

17. The process of claim 16, wherein the process action for determining if a blob meets a set of prescribed criteria that is indicative of the blob representing a human, comprises determining if the blob fits human body parameters in real space dimensions.

18. The process of claim 16, wherein the process action for determining if a blob meets a set of prescribed criteria that is indicative of the blob representing a human, comprises determining if at least a portion of the blob exhibits a substantially rectangular shape representing a human chest and head.

19. The process of claim 15, further comprising a process action for tracking, over time, the location of each of said one or more persons in the environment.