Perceiving roadway conditions from fused sensor data

• US 10,394,237 B2
• Filed: 09/08/2016
• Issued: 08/27/2019
• Est. Priority Date: 09/08/2016
• Status: Active Grant

• Current Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Original Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Inventors: Xu, Wei, Syed, Fazal Urrahman, Nallapa, Venkatapathi Raju, Myers, Scott Vincent
• Primary Examiner(s): Oh, Harry Y
• Application Number:
  US15/259,513
• Publication Number:
  US 20180067487A1
• Time in Patent Office: 1,083 Days
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 17/66 Tracking systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 2013/9323 Alternative operation using light waves
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0242 using non-visible light signals, e.g. IR or UV signals
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0248 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0278 using satellite positioning signals, e.g. GPS

Abstract

The present invention extends to methods, systems, and computer program products for perceiving roadway conditions from fused sensor data. Aspects of the invention use a combination of different types of cameras mounted to a vehicle to achieve visual perception for autonomous driving of the vehicle. Each camera in the combination of cameras generates sensor data by sensing at least part of the environment around the vehicle. The sensor data form each camera is fused together into a view of the environment around the vehicle. Sensor data from each camera (and, when appropriate, each other type of sensor) is fed into a central sensor perception chip. The central sensor perception chip uses a sensor fusion algorithm to fuse the sensor data into a view of the environment around the vehicle.

Claims

1. A method comprising:

sensing sensor data by a camera included in a vehicle and having a first field-of-view of a roadway, the sensor data sensed within a distance threshold of the vehicle and within the first field-of-view;

sensing other sensor data by another camera included in the vehicle and having a second field-of-view of the roadway, the other sensor data sensed from beyond the distance threshold within the second field-of-view, the second field-of-view narrower than the first field-of-view;

fusing the sensor data and the other sensor data into fused sensor data;

a processor perceiving an object between the distance threshold and another distance threshold in a roadway view of the roadway from the fused sensor data, the other distance threshold being further away from the vehicle than the distance threshold; and

the processor adjusting configuration of vehicle operating components to adapt operation of the vehicle to the perceived object in the roadway view;

wherein sensing other sensor data by another camera comprises sensing image sensor data and LIDAR sensor data by the other camera as the other camera switches between a camera mode and a LIDAR mode.

2. The method of claim 1, wherein adjusting the configuration of vehicle operating components comprises controlling the vehicle by autonomously adjusting the configuration of the vehicle operating-components.

3. The method of claim 1, wherein sensing sensor data by a camera comprises sensing Infrared (IR) sensor data by the camera.

4. The method of claim 1, wherein fusing the sensor data and the other sensor data into fused sensor data comprises fusing the sensor data, the image sensor data, and the LIDAR sensor data into fused sensor data.

5. The method of claim 1, further comprising receiving further sensor data from a further one or more sensors, each of the further one or more sensors selected from among:

a RADAR sensor, an ultrasonic sensor, a global positioning system (GPS) sensor, and an inertial measurement unit (IMU) sensor; and

wherein fusing the sensor data and the other sensor data into fused sensor data comprises fusing the sensor data, the other sensor data, and the further sensor data into fused sensor data.

6. The method of claim 3, wherein sensing Infrared (IR) sensor data comprises sensing Red Green Blue-Infrared (RGB-IR) image data, including IR pixel information, by the camera.

7. The method of claim 6, wherein fusing the sensor data and the other sensor data into fused sensor data comprises fusing the IR pixel information with the other sensor data into a reduced light environment roadway view.

8. The method of claim 1, wherein perceiving an object comprises perceiving a tracked object from a three-dimensional point cloud.

9. The method of claim 1, wherein perceiving an object comprises perceiving an object selected from among:

a roadway marking, a bicycle, a pedestrian, another vehicle, or a sign.

10. The method of claim 1, wherein adjusting configuration of vehicle operating components comprises performing one of the following accelerating the vehicle, decelerating the vehicle, changing gears of the vehicle, or turning the vehicle.

11. A method comprising:

sensing sensor data by a camera included in a vehicle and having a first field-of-view of a roadway, the sensor data sensed within a distance threshold of the vehicle and within the first field-of-view;

sensing image sensor data and LIDAR sensor data by another camera included in the vehicle as the other camera switches between a camera mode and a LIDAR mode, the other camera having a second field-of-view, the image sensor data and LIDAR sensor data sensed from beyond the distance threshold within the second field-of-view, the second field-of-view narrower than the first field-of-view;

fusing the sensor data, the image sensor data, and the LIDAR sensor data into fused sensor data;

a processor perceiving an object in a roadway view of the roadway from the fused sensor data; and

the processor adjusting configuration of vehicle operating components to adapt operation of the vehicle to the perceived object in the roadway view.

12. The method of claim 11, wherein adjusting configuration of vehicle operating components comprises controlling the vehicle by autonomously adjusting the configuration of the vehicle operating components.

13. The method of claim 11, wherein sensing sensor data by a camera comprises sensing Infrared (IR) sensor data at the camera;

and

wherein fusing the sensor data, the image sensor data, and the LIDAR sensor data into fused sensor data comprises fusing the Infrared (IR) sensor data, the image sensor data, and the LIDAR sensor into fused sensor data.

14. The method of claim 13, wherein sensing Infrared (IR) sensor data comprises sensing Red Green Blue-Infrared (RGB-IR) image data, including IR pixel information, by the camera.

15. The method of claim 14, wherein fusing the Infrared (IR) sensor data, the image sensor data, and the LIDAR sensor into fused sensor data comprises fusing the IR pixel information, the image sensor data, and the LIDAR sensor data into a reduced light environment roadway view.

16. The method of claim 11, further comprising receiving further sensor data from a further one or more sensors, each of the further one or more sensors selected from among:

a RADAR sensor, an ultrasonic sensor, a global positioning system (GPS) sensor, and an inertial measurement unit (IMU) sensor; and

wherein fusing the sensor data, the image sensor data, and the LIDAR sensor data into fused sensor data comprises the sensor data, the image sensor data, the LIDAR, And the further sensor data into fused sensor.

17. The method of claim 11, wherein perceiving an object comprises perceiving a tracked object from a three-dimensional point cloud.

18. The method of claim 11, wherein perceiving an object comprises perceiving an object selected from among:

a roadway marking, a bicycle, a pedestrian, another vehicle, or a sign.

19. The method of claim 11, wherein adjusting configuration of vehicle operating components comprises performing one of the following:

accelerating the vehicle, decelerating the vehicle, changing gears of the vehicle, or turning the vehicle.