Person detection and pose estimation system

• US 9,141,852 B1
• Filed: 08/28/2013
• Issued: 09/22/2015
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Inventors: Cosgun, Akansel, Sisbot, Emrah Akin
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): Lesnick, Alexander J
• Application Number:
  US14/012,194
• Time in Patent Office: 755 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A61H 3/061 with electronic detecting or guiding means
• CPC Class Codes: G01C 1/00 Measuring angles
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G06V 10/457 by analysing connectivity, e.g. edge linking, connected component analysis or slices
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G08B 6/00 Tactile signalling systems, e.g. personal calling systems

Abstract

A system for detecting a person and estimating pose information comprises a processor and a memory storing instructions causing the system to: retrieve depth data from a sensor, the depth data describing distance information associated with one or more objects detected by the sensor; cluster the depth data to determine two or more candidate leg clusters, each candidate leg cluster including a portion of the depth data that may represent a human leg detected by the sensor; identify a candidate leg cluster pair including two candidate leg clusters within a certain distance between each other; determine whether there is a connectivity between the two candidate leg clusters included in the candidate leg cluster pair; and responsive to determining that there is a connectivity between the two candidate leg clusters, determine that the candidate leg cluster pair is qualified to be a leg cluster pair representing a person.

Claims

1. A computer-implemented method, comprising:

retrieving depth data from a sensor, the depth data describing distance information associated with one or more objects detected by the sensor;

clustering the depth data to determine two or more candidate leg clusters, each candidate leg cluster including a portion of the depth data that may represent a human leg detected by the sensor;

identifying a candidate leg cluster pair based on the two or more candidate leg clusters, the candidate leg cluster pair including two candidate leg clusters within a certain distance between each other, the two candidate leg clusters including center points and neighbor points surrounding the center points;

propagating from the center points of the two candidate leg clusters included in the candidate leg cluster pair upwards to the neighbor points of the two candidate leg clusters that are represented by portions of the depth data;

determining a connectivity between the neighbor points of the two candidate leg clusters;

responsive to determining the connectivity between the neighbor points of the two candidate leg clusters, determining that there is a connectivity between the two candidate leg clusters included in the candidate leg cluster pair; and

responsive to determining that there is the connectivity between the two candidate leg clusters included in the candidate leg cluster pair, determining that the candidate leg cluster pair is qualified to be a leg cluster pair representing a person detected by the sensor.

2. The method of claim 1, wherein clustering the depth data to determine the two or more candidate leg clusters further comprises:

projecting the depth data into a two dimensional plane, the two dimensional plane including points that represent the depth data; and

grouping the points in the two dimensional plane into two or more object clusters, each object cluster including two or more points adjacent to each other.

3. The method of claim 2, wherein clustering the depth data to determine the two or more candidate leg clusters further comprises:

filtering the two or more object clusters based on sizes of the two or more object clusters; and

determining the two or more candidate leg clusters from the two or more object clusters based on the filtering.

4. The method of claim 1, wherein propagating from the center points of the two candidate leg clusters included in the candidate leg cluster pair upwards to the neighbor points of the two candidate leg clusters that are represented by portions of the depth data further comprises matching the center points of the two candidate leg clusters included in the candidate leg cluster pair to points in a three dimensional point cloud.

5. The method of claim 1, wherein the depth data is a three dimensional point cloud.

6. The method of claim 1 further comprising:

determining position information associated with the leg cluster pair; and

estimating a pose for the person detected by the sensor based on the position information associated with the leg cluster pair.

7. A system comprising:

a processor; and

a memory storing instructions that, when executed, cause the system to;

retrieve depth data from a sensor, the depth data describing distance information associated with one or more objects detected by the sensor;

cluster the depth data to determine two or more candidate leg clusters, each candidate leg cluster including a portion of the depth data that may represent a human leg detected by the sensor;

identify a candidate leg cluster pair based on the two or more candidate leg clusters, the candidate leg cluster pair including two candidate leg clusters within a certain distance between each other, the two candidate leg clusters including center points and neighbor points surrounding the center points;

propagate from the center points of the two candidate leg clusters included in the candidate leg cluster pair upwards to the neighbor points of the two candidate leg clusters that are represented by portions of the depth data;

determine a connectivity between the neighbor points of the two candidate leg clusters;

responsive to determining the connectivity between the neighbor points of the two candidate leg clusters, determine that there is a connectivity between the two candidate leg clusters included in the candidate leg cluster pair; and

responsive to determining that there is the connectivity between the two candidate leg clusters included in the candidate leg cluster pair, determine that the candidate leg cluster pair is qualified to be a leg cluster pair representing a person detected by the sensor.

8. The system of claim 7, wherein clustering the depth data to determine the two or more candidate leg clusters further comprises:

projecting the depth data into a two dimensional plane, the two dimensional plane including points that represent the depth data; and

grouping the points in the two dimensional plane into two or more object clusters, each object cluster including two or more points adjacent to each other.

9. The system of claim 8, wherein clustering the depth data to determine the two or more candidate leg clusters further comprises:

filtering the two or more object clusters based on sizes of the two or more object clusters; and

determining the two or more candidate leg clusters from the two or more object clusters based on the filtering.

10. The system of claim 7, wherein propagating from the center points of the two candidate leg clusters included in the candidate leg cluster pair upwards to the neighbor points of the two candidate leg clusters that are represented by portions of the depth data further comprises matching the center points of the two candidate leg clusters included in the candidate leg cluster pair to points in a three dimensional point cloud.

11. The system of claim 7, wherein the depth data is a three dimensional point cloud.

12. The system of claim 7, wherein the instructions, when executed, cause the system to also:

determine position information associated with the leg cluster pair; and

estimate a pose for the person detected by the sensor based on the position information associated with the leg cluster pair.

13. A computer program product comprising a non-transitory computer readable medium encoding instructions that, in response to execution by a computing device, cause the computing device to perform operations comprising:

retrieving depth data from a sensor, the depth data describing distance information associated with one or more objects detected by the sensor;

clustering the depth data to determine two or more candidate leg clusters, each candidate leg cluster including a portion of the depth data that may represent a human leg detected by the sensor;

identifying a candidate leg cluster pair based on the two or more candidate leg clusters, the candidate leg cluster pair including two candidate leg clusters within a certain distance between each other, the two candidate leg clusters including center points and neighbor points surrounding the center points;

propagating from the center points of the two candidate leg clusters included in the candidate leg cluster pair upwards to the neighbor points of the two candidate leg clusters that are represented by portions of the depth data;

determining a connectivity between the neighbor points of the two candidate leg clusters;

responsive to determining the connectivity between the neighbor points of the two candidate leg clusters, determining that there is a connectivity between the two candidate leg clusters included in the candidate leg cluster pair; and

responsive to determining that there is the connectivity between the two candidate leg clusters included in the candidate leg cluster pair, determining that the candidate leg cluster pair is qualified to be a leg cluster pair representing a person detected by the sensor.

14. The computer program product of claim 13, wherein clustering the depth data to determine the two or more candidate leg clusters further comprises:

projecting the depth data into a two dimensional plane, the two dimensional plane including points that represent the depth data; and

grouping the points in the two dimensional plane into two or more object clusters, each object cluster including two or more points adjacent to each other.

15. The computer program product of claim 14, wherein clustering the depth data to determine the two or more candidate leg clusters further comprises:

filtering the two or more object clusters based on sizes of the two or more object clusters; and

determining the two or more candidate leg clusters from the two or more object clusters based on the filtering.

16. The computer program product of claim 13, wherein propagating from the center points of the two candidate leg clusters included in the candidate leg cluster pair upwards to the neighbor points of the two candidate leg clusters that are represented by portions of the depth data further comprises matching the center points of the two candidate leg clusters included in the candidate leg cluster pair to points in a three dimensional point cloud.

17. The computer program product of claim 13, wherein the instructions, in response to execution by the computing device, cause the computing device to perform operations further comprising:

determining position information associated with the leg cluster pair; and

estimating a pose for the person detected by the sensor based on the position information associated with the leg cluster pair.