Automated driving system that merges heterogenous sensor data

• US 10,599,931 B2
• Filed: 08/21/2017
• Issued: 03/24/2020
• Est. Priority Date: 08/21/2017
• Status: Active Grant

• Current Assignee: Blackberry Limited
• Original Assignee: 2236008 Ontario Inc. (Blackberry Limited)
• Inventors: Miville, Xavier, Montminy, Christian Joseph Eric
• Primary Examiner(s): Marc-Coleman, Marthe Y
• Application Number:
  US15/681,891
• Publication Number:
  US 20190057263A1
• Time in Patent Office: 946 Days
• Field of Search: 701301, 382218
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 2013/9318 Controlling the steering
• CPC Class Codes: G01S 2013/93185 Controlling the brakes
• CPC Class Codes: G01S 2013/9319 Controlling the accelerator
• CPC Class Codes: G01S 2013/9323 Alternative operation using light waves
• CPC Class Codes: G01S 7/2955 Means for determining the position of the radar coordinate system for evaluating the position data of the target in another coordinate system G01S7/24 takes precedence; sighting devices adapted for indirect laying of fire F41G3/16; inertial navigation G01C21/16
• CPC Class Codes: G05D 1/0061 for transition from automatic pilot to manual pilot and vice versa
• CPC Class Codes: G05D 1/024 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks

Abstract

An automated driving system comprising a first sensor providing first sensor data, a second sensor providing second sensor data, wherein the first and second sensors are different types of sensors and a processor communicatively coupled to the first sensor and the second sensor. The processor is configured to transform the first and second sensor data into a common three-dimensional coordinate system to create a representation of an image for a field of view corresponding to a direction of travel, identify one or more objects in the field of view, and generate one or more drive control signals to control a motion of the vehicle in response to the one or more objects in the field of view.

Claims

1. An automated driving system for a vehicle, the system comprising:

a first sensor providing first sensor data about an object of interest, wherein the first sensor data comprises 3D characteristics;

a second sensor providing second sensor data about the object of interest, wherein the first and second sensors are different types of sensors, wherein the second sensor data comprises 2D characteristics; and

a processor communicatively coupled to the first sensor and the second sensor, wherein the processor is configured to;

process the first and second sensor data for the object of interest by transforming the first or second sensor data into a common coordinate system, by comparing similarities between the characteristics of the object of interest and a fused object in the common coordinate system to determine if the object of interest and the fused object match, and augmenting the fused object with the characteristics of the object of interest if the object of interest and the fused object match; and

generate one or more drive control signals to control a motion of the vehicle in response to the fused object.

2. The system of claim 1 wherein the first sensor is RADAR and wherein the second sensor is a camera.

3. The system of claim 1 wherein the first sensor is LIDAR and wherein the second sensor is a camera.

4. The system of claim 1 wherein the processor is configured to apply a first weight to the first sensor data and a second weight to the second sensor data, wherein the first weight and the second weight are variable based on ambient light.

5. The system of claim 4 wherein the first weight and second weight are based on manufacturer-specified sensitivity data.

6. A non-transitory computer-readable medium comprising instructions in code which when loaded into a memory and executed by a processor of a vehicle cause the processor to:

receive first sensor data about an object of interest from a first sensor, wherein the first sensor data comprises 3D characteristics;

receive second sensor data about the object of interest from a second sensor of a type different from the first sensor, wherein the second sensor data comprises 2D characteristics;

process the first and second sensor data for the object of interest by transforming the first or second sensor data into a common coordinate system, by comparing similarities between the characteristics of the object of interest and a fused object in the common coordinate system to determine if the object of interest and the fused object match, and augmenting the fused object with the characteristics of the object of interest if the object of interest and the fused object match; and

generate one or more drive control signals to control a motion of the vehicle in response to the fused object.

7. The computer-readable medium of claim 6 wherein the first sensor is RADAR and wherein the second sensor is a camera.

8. The computer-readable medium of claim 6 wherein the first sensor is LIDAR and wherein the second sensor is a camera.

9. The computer-readable medium of claim 6 wherein the code causes the processor to apply a first weight to the first sensor data and a second weight to the second sensor data, wherein the first weight and the second weight are variable based on ambient light.

10. The computer-readable medium of claim 9 wherein the first weight and second weight are based on manufacturer-specified sensitivity data.

11. A vehicle comprising:

a powertrain;

a braking system;

a steering system;

an automated driving system for at least partially controlling one or more of the powertrain, braking system and steering system;

wherein the automated driving system includes;

a first sensor providing first sensor data about an object of interest, wherein the first sensor data comprises 3D characteristics;

a second sensor providing second sensor data about the object of interest, wherein the first and second sensors are different types of sensors, wherein the first sensor data comprises 2D characteristics; and

a processor communicatively coupled to the first sensor and the second sensor, wherein the processor is configured to;

process the first and second sensor data for the object of interest by transforming the first or second sensor data into a common coordinate system to compare similarities between the characteristics of the object of interest and a fused object in the common coordinate system to determine if the object of interest and the fused object match, and augmenting the fused object with the characteristics of the object of interest if the object of interest and the fused object match; and

generate one or more drive control signals to control a motion of the vehicle in response to the fused object.

12. The vehicle of claim 11 wherein the first sensor is RADAR and wherein the second sensor is a camera.

13. The vehicle of claim 11 wherein the first sensor is LIDAR and wherein the second sensor is a camera.

14. The vehicle of claim 11 wherein the processor is configured to apply a first weight to the first sensor data and a second weight to the second sensor data, wherein the first weight and the second weight are variable based on ambient light.

15. The vehicle of claim 14 wherein the first weight and second weight are based on manufacturer-specified sensitivity data.