Pedestrian detection and motion prediction with rear-facing camera

• US 10,055,652 B2
• Filed: 03/21/2016
• Issued: 08/21/2018
• Est. Priority Date: 03/21/2016
• Status: Active Grant

• Current Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Original Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Inventors: Myers, Scott Vincent, Nallapa, Venkatapathi Raju, Nariyambut Murali, Vidya, Goh, Madeline Jane
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): BRUMFIELD, SHANIKA M
• Application Number:
  US15/076,407
• Publication Number:
  US 20170270374A1
• Time in Patent Office: 883 Days
• Field of Search: 348148
• CPC Class Codes: B60Q 9/00 Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 , e.g. haptic signalling
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

Systems, methods, and devices for pedestrian detection are disclosed herein. A method includes receiving one or more images from a rear-facing camera on a vehicle. The method further includes determining that a pedestrian is present in the one or more images, predicting future motion of the pedestrian, and notifying a driver-assistance or automated driving system when a conflict exists between forward motion of the vehicle and the predicted future motion of the pedestrian.

Claims

1. A method comprising:

receiving one or more images from a rear-facing camera on a vehicle facing in a rearward direction;

determining, by a computer system, that a pedestrian is present in the one or more images;

predicting, by the computer system based on the one or more images, future motion of the pedestrian;

evaluating the future motion of the pedestrian with respect to a path of the vehicle in a forward direction opposite the rearward direction;

determining (a) that the future motion of the pedestrian is in conflict with the path of the vehicle in the forward direction; and

in response to (a), notifying, by the computer system, a driver-assistance or automated driving system that a conflict exists between forward motion of the vehicle and the future motion.

2. The method of claim 1, wherein the determining comprises:

processing, by the computer system, the one or more images using a first neural network to determine one or more locations where pedestrians are likely located within the one or more images; and

processing, by the computer system, the one or more locations of the one or more images using a second neural network to determine that the pedestrian is present.

3. The method of claim 2, wherein the predicting comprises determining, based on the one or more images, a velocity and direction of the pedestrian.

4. The method of claim 3, wherein the predicting further comprises determining, based on the one or more images, a location of the pedestrian in relation to the vehicle.

5. The method of claim 4, wherein the future motion comprises motion outside of the view of the rear-facing camera.

6. The method of claim 5, wherein the first neural network comprises a network trained to identify approximate locations within the one or more images that likely contain pedestrians.

7. The method of claim 6, wherein the first neural network generates a saliency map indicating most likely locations of pedestrians.

8. The method of claim 7, wherein the saliency map comprises a lower resolution than the one or more images.

9. The method of claim 8, wherein the second neural network processes the one or more locations within the one or more images at full resolution.

10. The method of claim 1, wherein:

the rear-facing camera is a back-up camera; and

the one or more images are captured by the back-up camera while the vehicle is in a forward gear.

11. A method for detecting pedestrians comprising:

receiving one or more images from a rear-facing camera on a vehicle, the one or more images being captured while the vehicle is in a forward gear;

identifying, by a first neural network of a computer system, the one or more images to determine one or more regions where pedestrians are likely located within the one or more images;

determining, by a second neural network of the computer system, that a pedestrian is present in at least one of the one or more regions;

predicting, by the computer system based on the one or more images, future motion of the pedestrian, the predicted future motion comprising motion outside of the view of the rear- facing camera;

evaluating, by the computer system, the future motion of the pedestrian with respect to a path of the vehicle in a forward direction;

determining, by the computer system, (a) that the future motion of the pedestrian is in conflict with the path of the vehicle in the forward direction; and

in response to (a), notifying, by the computer system, a driver-assistance or automated driving system that a conflict exists between forward motion of the vehicle and the predicted future motion.

12. The method of claim 11, wherein the predicting comprises determining, based on the one or more images, a velocity and direction of the pedestrian.

13. The method of claim 12, wherein the predicting further comprises determining, based on the one or more images, a location of the pedestrian in relation to the vehicle.

14. The method of claim 13, wherein:

the first neural network generates one or more saliency maps indicating the one or more regions;

the one or more saliency maps each comprise a lower resolution than the one or more images; and

the second neural network processes the one or more regions within the one or more images at full resolution.

15. A system comprising:

a rear-facing camera positioned on a vehicle to capture one or more images of a region behind the vehicle;

a saliency component configured to process the one or more images using a first neural network to generate one or more low resolution saliency maps indicating one or more regions where pedestrians are most likely located within the one or more images;

a detection component configured to process the one or more regions using a second neural network to determine, for each of one or more regions, whether a pedestrian is present;

a prediction component configured to process the one or more images to predict future motion of one or more pedestrians captured in the one or more images; and

a notification component configured to- evaluate the future motion of the pedestrian with respect to a path of the vehicle in a forward direction;

when (a) the future motion of the pedestrian is in conflict with the path of the vehicle in the forward direction, provide a notification indicating a conflict between forward motion of the vehicle of the predicted future motion of the one or more pedestrians.

16. The system of claim 15, wherein the prediction component is configured to:

determine, based on the one or more images, a velocity and direction for each of the one or more pedestrians;

determine, based on the one or more images, a location for each of the one or more pedestrians in relation to the vehicle; and

predict the future motion of one or more pedestrians based at least in part on the velocity, direction, and location for each of the one or more pedestrians.

17. The system of claim 16, wherein the one or more saliency maps comprise lower resolutions than the one or more images.

18. The system of claim 17, wherein the detection component uses the second neural network to process at full resolution the one or more regions within the one or more images.

19. The system of claim 18, wherein the second neural network comprises a deep neural network classifier that has been trained using cropped ground truth bounding boxes to determine that a pedestrian is or is not present.

20. The system of claim 15, wherein:

the rear-facing camera is a back-up camera; and

the one or more images are captured by the back-up camera while the vehicle is in a forward gear.