Image-based pedestrian detection

• US 10,108,867 B1
• Filed: 09/08/2017
• Issued: 10/23/2018
• Est. Priority Date: 04/25/2017
• Status: Active Grant

• Current Assignee: Aurora Operations Incorporated (Aurora Innovation, Inc.)
• Original Assignee: Uber Technologies, Inc.
• Inventors: Vallespi-Gonzalez, Carlos, Amato, Joseph, Bristow, Hilton
• Primary Examiner(s): Tucker, Wesley J
• Application Number:
  US15/698,824
• Publication Number:
  US 20180307921A1
• Time in Patent Office: 410 Days
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G01S 17/66 Tracking systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 17/93 for anti-collision purposes
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 7/4802 using analysis of echo signal for target characterisation; Target signature; Target cross-section
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 30/2504 Coarse or fine approaches, e.g. resolution of ambiguities or multiscale approaches
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

Object detection systems and methods can include identifying an object of interest within an image obtained from a camera, obtaining a first supplemental portion of data associated with the object of interest determining an estimated location of the object of interest within three-dimensional space based at least in part on the first supplemental portion of data and a known relative location of the camera, determining a portion of the LIDAR point data corresponding to the object of interest based at least in part on the estimated location of the object of interest within three-dimensional space, and providing one or more of at least a portion of the image corresponding to the object of interest and the portion of LIDAR point data corresponding to the object of interest as an output.

Claims

1. A method for detecting pedestrians in images, comprising:

identifying, by a computing system comprising one or more electronic devices, a pedestrian within an image obtained from a camera positioned relative to a vehicle;

determining, by the computing system, an estimated support location for the pedestrian within the image;

determining, by the computing system, a supplemental portion of data correlated with the pedestrian, the supplemental portion of data obtained from a sensor other than the camera;

determining, by the computing system, an estimated location of the pedestrian within three-dimensional space based at least in part on (i) the estimated support location for the pedestrian within the image, (ii) a known relative location of the camera to the vehicle, and (iii) the supplemental portion of data;

obtaining, by the computing system, map data descriptive of a geographic area surrounding the vehicle, the map data including relative locations of ground surfaces proximate to the vehicle;

determining, by the computing system, a revised estimated location of a pedestrian within three-dimensional space based at least in part on the estimated location and the map data; and

providing, by the computing system, at least a portion of the image corresponding to the pedestrian and the estimated location of the pedestrian within the three-dimensional space to an object classification application.

2. The method of claim 1 wherein the supplemental portion of data comprises a portion of LIDAR point data corresponding to the pedestrian.

3. The method of claim 2, wherein determining, by the computing system, the portion of LIDAR point data comprises identifying, by the computing system, a cluster of LIDAR data points of a predetermined size near a location within a point cloud corresponding to the estimated location of the pedestrian within the three-dimensional space.

4. The method of claim 1, wherein determining, by the computing system, the estimated support location for the pedestrian within the image comprises determining, by the computing system, a location within the image corresponding to feet of the pedestrian.

5. The method of claim 1, wherein determining, by the computing system, the estimated support location for the pedestrian within the image comprises:

determining, by the computing system, a location within the image corresponding to a head of the pedestrian; and

determining, by the computing system, the estimated support location for the pedestrian based at least in part on the location within the image corresponding to the head of the pedestrian and an estimated height of the pedestrian.

6. The method of claim 1, wherein determining a revised estimated location of the pedestrian within three-dimensional space based at least in part on the estimated location and the map data comprises adjusting the estimated location to a nearest point within three-dimensional space corresponding to one of the relative locations of ground surfaces identified within the map data.

7. The method of claim 1, wherein identifying, by the computing system, a pedestrian within an image obtained from a camera positioned relative to a vehicle comprises:

generating, by the computing system, a multi-scale image pyramid of multiple image samples having different scaling factors; and

analyzing, by the computing system, successive image patches within each of the multiple image samples using a sliding window of fixed size to identify pedestrians within the successive image patches.

8. The method of claim 1, wherein identifying, by the computing system, a pedestrian within an image obtained from a camera positioned relative to a vehicle comprises:

determining, by the computing system, a region proposal within the image, the region proposal having a smaller area than the image; and

identifying, by the computing system, the pedestrian within the region proposal.

9. The method of claim 8, wherein the region proposal is determined at least in part from locations within the image corresponding to locations for which corresponding LIDAR point data is also obtained.

10. The method of claim 8, wherein the region proposal is determined at least in part from locations within the image determined to correspond to foreground as opposed to background.

11. A detection system, comprising:

a camera positioned relative to a vehicle and configured to obtain images within an area proximate to the vehicle;

a LIDAR sensor positioned relative to a vehicle and configured to obtain LIDAR point data within an area proximate to the vehicle;

one or more electronic devices coupled to the camera and the LIDAR sensor, the one or more electronic devices configured to receive image data from the camera and LIDAR point data from the LIDAR sensor and perform operations comprising;

identifying an object of interest within an image obtained from the camera;

determining an estimated location of the object of interest within three-dimensional space based at least in part on a known relative location of the camera;

obtaining map data descriptive of a geographic area surrounding the vehicle; and

determining a revised estimated location of an object of interest within three-dimensional space based at least in part on the estimated location and the map data;

determining a portion of the LIDAR point data corresponding to the object of interest based at least in part on the estimated location of the object of interest within three-dimensional space; and

providing one or more of at least a portion of the image corresponding to the object of interest and the portion of the LIDAR point data corresponding to the object of interest as an output.

12. The detection system of claim 11, further comprising determining an estimated support location for the object of interest within the image, wherein:

when a lower bound of the object of interest is visible within the image, determining an estimated support location for the object of interest within the image comprises determining a location within the image corresponding to the lower bound of the object of interest; and

when the lower bound of the object of interest is not visible, determining an estimated support location for the object of interest within the image comprises;

determining a location within the image corresponding to an upper bound of the object of interest; and

determining the estimated support location for the object of interest based at least in part on the location within the image corresponding to the upper bound of the object of interest and an estimated height of the object of interest.

13. The detection system of claim 11, wherein determining a revised estimated location of an object of interest within three-dimensional space based at least in part on the estimated location and the map data comprises adjusting the estimated location to a nearest point within three-dimensional space corresponding to one of the relative locations of ground surfaces identified within the map data.

14. The detection system of claim 11, wherein identifying an object of interest within an image obtained from the camera comprises:

generating a multi-scale image pyramid of multiple image samples having different scaling factors; and

analyzing successive image patches within each of the multiple image samples using a sliding window of fixed size to identify objects of interest within the successive image patches.

15. The detection system of claim 11, wherein identifying an object of interest within an image obtained from the camera comprises:

determining a region proposal within the image, the region proposal having a smaller area than the image; and

identifying an object of interest within the region proposal.

16. An autonomous vehicle, comprising:

one or more cameras configured to obtain images within an environment proximate to an autonomous vehicle;

a field programmable gate array (FPGA) device coupled to the one or more cameras, the FPGA device configured to receive an image from the one or more cameras, generate a multi-scale image pyramid of multiple image samples having different scaling factors, and analyze successive image patches within each of the multiple image samples using a sliding window of fixed size to identify objects of interest within the successive image patches;

one or more processors; and

one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising;

determining a supplemental portion of data correlated with each object of interest, each supplemental portion of data obtained from a sensor other than the one or more cameras, each supplemental portion of data comprising map data descriptive of a geographic area surrounding the autonomous vehicle and including relative locations of ground surfaces proximate to the autonomous vehicle; and

providing at least a portion of the image corresponding to each object of interest and the supplemental portion of data associated with each object of interest to an object classification application.

17. The autonomous vehicle of claim 16, further comprising:

a LIDAR sensor positioned relative to the autonomous vehicle and configured to obtain LIDAR point data within an area proximate to the autonomous vehicle; and

wherein the operations further comprise providing at least a portion of the LIDAR point data corresponding to an estimated location of each object of interest within three-dimensional space to the object classification application.

18. The autonomous vehicle of claim 16, wherein the FPGA device is further configured to determine a region proposal within each image sample, each region proposal having a smaller area than its corresponding image sample, and to analyze successive image patches within the region proposal for each of the multiple image samples by using a sliding window of fixed size to identify objects of interest within the successive image patches.

19. The autonomous vehicle of claim 18, wherein the region proposal for each image sample is determined at least in part from locations within the image corresponding to locations for which corresponding LIDAR point data is also obtained.

20. The autonomous vehicle of claim 18, wherein the region proposal is determined at least in part from locations within the image determined to correspond to foreground as opposed to background.