Video to radar

• US 10,495,753 B2
• Filed: 08/29/2017
• Issued: 12/03/2019
• Est. Priority Date: 09/19/2016
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Melvin, Iain, Cosatto, Eric, Durdanovic, Igor, Graf, Hans Peter
• Primary Examiner(s): Tsai, Tsung Yin
• Application Number:
  US15/689,656
• Publication Number:
  US 20180081053A1
• Time in Patent Office: 826 Days
• CPC Class Codes: B60G 2400/823 Obstacle sensing
• CPC Class Codes: B60Q 9/008 for anti-collision purposes
• CPC Class Codes: B60R 2300/301 combining image information with other obstacle sensor information, e.g. using RADAR/LIDAR/SONAR sensors for estimating risk of collision
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 2013/9316 combined with communication equipment with other vehicles or with base stations
• CPC Class Codes: G01S 2013/9323 Alternative operation using light waves
• CPC Class Codes: G01S 2013/93271 in the front of the vehicles
• CPC Class Codes: G01S 7/20 Stereoscopic displays; Three-dimensional displays; Pseudo-three-dimensional displays
• CPC Class Codes: G01S 7/2955 Means for determining the position of the radar coordinate system for evaluating the position data of the target in another coordinate system G01S7/24 takes precedence; sighting devices adapted for indirect laying of fire F41G3/16; inertial navigation G01C21/16
• CPC Class Codes: G01S 7/417 involving the use of neural networks
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 : Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

A computer-implemented method and system are provided. The system includes an image capture device configured to capture image data relative to an ambient environment of a user. The system further includes a processor configured to detect and localize objects, in a real-world map space, from the image data using a trainable object localization Convolutional Neural Network (CNN). The CNN is trained to detect and localize the objects from image and radar pairs that include the image data and radar data for different scenes of a natural environment. The processor is further configured to perform a user-perceptible action responsive to a detection and a localization of an object in an intended path of the user.

Claims

1. A system, comprising:

an image capture device configured to capture image data relative to an ambient environment of a user; and

a processor configured to detect and localize objects, in a real-world map space, from the image data using a trainable object localization Convolutional Neural Network (CNN),

wherein the CNN is trained to detect and localize the objects from image and radar pairs that include the image data and radar data for different scenes of a natural environment, and

wherein the processor is further configured to perform a user-perceptible action responsive to a detection and a localization of an object in an intended path of the user,

wherein the processor is further configured to generate an image showing positions of the objects in a top-down map-view perspective, and wherein the top-down map-view perspective is intentionally distorted in a pre-processing stage to correct for image capture related distortions.

2. The system of claim 1, wherein the processor is configured to train the CNN to detect and localize the objects also from image and object location list pairs that include (i) the image data or other image data and (ii) objection location data for the objects or other objects.

3. The system of claim 1, wherein the image showing the positions of the objects is a bitmap image.

4. The system of claim 1, wherein the processor is configured to use a detection window that is sized larger than an expected target object size, to enhance a detection context for object detection.

5. The system of claim 1, wherein the processor is configured to perform post-processing on the image to produce a discrete list of object detections and corresponding object positions in real world space.

6. The system of claim 1, wherein the processor is configured to use a decision making process to evaluate a scene as potentially dangerous such that the processor provides a user-perceptible object detection result indicative of a potentially dangerous scene.

7. The system of claim 1, wherein the system is comprised in an advanced driver-assistance system that uses a decision making process to evaluate a scene as potentially dangerous such that the processor automatically takes control over one or more systems of the motor vehicle to mitigate any potential harm.

8. The system of claim 1, wherein the processor is configured to directly output locations for all of the objects into a map-view space that has a direct one-to-one projection onto the real-world map space.

9. The system of claim 1, wherein the processor is configured to also detect and localize the objects, at all scales, in a single-pass.

10. The system of claim 1, wherein the processor is configured to perform non-maximal suppression on a set of probabilities that a particular object is at a particular location to obtain a list of object detections in an output space of the CNN.

11. The system of claim 1, wherein a last layer in the CNN collapses a Z-dimension of an output of the CNN using a max function to provide an output of the CNN only in X- and Y-dimensions.

12. The system of claim 1, wherein the processor detects and localizes the objects using the CNN by (i) inputting N features for each of input windows applied to the image data, and (ii) mapping the N features into a vector having a constrained length using 1×

1 kernels, wherein N is an integer.

13. A computer-implemented method, comprising:

capturing, by an image capture device, image data relative to an ambient environment of a user;

detecting and localizing, by a processor, objects, in a real-world map space, from the image data using a trainable object localization Convolutional Neural Network (CNN); and

providing, by the processor, performing a user-perceptible action responsive to a detection and a localization of an object in an intended path of the user,

wherein the CNN is trained to detect and localize the objects from image and radar pairs that include the image data and radar data for different scenes of a natural environment,

wherein the processor is further configured to generate an image showing positions of the objects in a top-down map-view perspective, and wherein the top-down map-view perspective is intentionally distorted in a pre-processing stage to correct for image capture related distortions.

14. The computer-implemented method of claim 13, wherein the CNN is trained to detect and localize the objects also from image and object location list pairs that include (i) the image data or other image data and (ii) objection location data for objects including and other than vehicles.

15. The computer-implemented method of claim 13, further comprising performing post-processing on the image to produce a discrete list of object detections and corresponding object positions in real world space.

16. The computer-implemented method of claim 13, wherein said detecting and localizing step detects and localizes the objects, at all scales, in a single-pass.

17. A computer program product, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:

capturing, by an image capture device, image data relative to an ambient environment of a user;

detecting and localizing, by a processor, objects, in a real-world map space, from the image data using a trainable object localization Convolutional Neural Network (CNN); and

providing, by the processor, performing a user-perceptible action responsive to a detection and a localization of an object in an intended path of the user,

wherein the CNN is trained to detect and localize the objects from image and radar pairs that include the image data and radar data for different scenes of a natural environment,

wherein the processor is further configured to generate an image showing positions of the objects in a top-down map-view perspective, and wherein the top-down map-view perspective is intentionally distorted in a pre-processing stage to correct for image capture related distortions.