Driving assistance system and method of enhancing a driver's vision

• US 10,688,929 B2
• Filed: 05/25/2018
• Issued: 06/23/2020
• Est. Priority Date: 06/27/2017
• Status: Active Grant

• Current Assignee: Mindtronic AI Co.,Ltd., Shanghai XPT Technology Limited
• Original Assignee: Shanghai XPT Technology Limited
• Inventors: Huang, Mu-Jen, Tai, Ya-Li, Jiang, Yu-Sian
• Primary Examiner(s): Pham, Nam D
• Application Number:
  US15/989,208
• Publication Number:
  US 20180370434A1
• Time in Patent Office: 760 Days
• CPC Class Codes: B60R 1/24 in front of the vehicle
• CPC Class Codes: B60R 1/30 providing vision in the non-visible spectrum, e.g. night or infrared vision
• CPC Class Codes: B60R 2300/106 using night vision cameras
• CPC Class Codes: B60R 2300/301 combining image information with other obstacle sensor information, e.g. using RADAR/LIDAR/SONAR sensors for estimating risk of collision
• CPC Class Codes: B60R 2300/304 using merged images, e.g. merging camera image with stored images
• CPC Class Codes: B60R 2300/70 characterised by an event-triggered choice to display a specific image among a selection of captured images

Abstract

A driving assistance system includes a first camera disposed in the vehicle and configured to capture a visible image; a second camera disposed in the vehicle and configured to capture an invisible image; a display device configured to display images to the driver; and a processor coupled to the first camera, the second camera, and the display device, the processor conducts a process of image fusion to combine the visible image and the invisible image to obtain a fused image; wherein the display device displays the fused image only when the clarity of the visible image falls below a preset threshold.

Claims

1. A driving assistance system in a vehicle for assisting a driver, comprising:

a first camera disposed in the vehicle and configured to capture a visible image;

a second camera disposed in the vehicle and configured to capture an invisible image;

a display device configured to display images to the driver;

a plurality of sensors disposed in the vehicle and configured to collect environmental data surrounding the vehicle; and

a processor coupled to the first camera, the second camera, and the display device, the processor is configured to determine a weather condition based on the environmental data, determine whether a clarity of the visible image falls below a preset threshold, determine that a driver'"'"'s vision is affected when the weather condition is determined as misty or foggy, hazy or smoggy, and conduct a process of image fusion to combine the visible image and the invisible image to obtain a fused image;

wherein the display device displays the fused image when the clarity of the visible image falls below the preset threshold, and displays the fused image when the driver'"'"'s vision is determined to be affected.

2. The system of claim 1, wherein the visible image mimics the driver'"'"'s vision from the vehicle.

3. The system of claim 1, wherein the invisible image is a thermal image, while the second camera is a thermal imaging camera or an infrared camera.

4. The system of claim 1, wherein one of the plurality of sensors is configured to detect one or more objects, one of the plurality of sensors is a lidar or a radar, and the processor is further configured to generate an object image based on the detected objects.

5. The system of claim 4, wherein the processor conducts the process of image fusion to combine the visible image, the invisible image and the object image to obtain the fused image.

6. The system of claim 1, wherein the determination of the clarity of the visible image is made by reference to either a size, a size ratio or a pixel amount of glaring or foggy areas identified on the visible images, or intensities of an ambient light between a critical area and its neighboring area.

7. The system of claim 1, wherein two of the sensors include a temperature and humidity sensor configured to collect temperature and humidity degrees, and the processor is further configured to:

determine whether the weather condition is misty or foggy based on the temperature and humidity degrees; and

determine that the driver'"'"'s vision is affected when the weather condition is determined as misty or foggy.

8. The system of claim 1, wherein the display device is configured to display the environmental data together with the fused image to the driver of the vehicle.

9. The system of claim 1, wherein the plurality of sensors include a particulate matter (PM) sensor configured to collect PM concentration, and the processor is further configured to:

determine whether the weather condition is hazy or smoggy based on the PM concentration, and

determine that the driver'"'"'s vision is affected when the weather condition is determined as hazy or smoggy.

10. A method of enhancing a driver'"'"'s vision from a vehicle, comprising:

obtaining a visible image;

obtaining an invisible image;

determining a weather condition based on environmental data;

determining whether a clarity of the visible image falls below a preset threshold;

determining that the driver'"'"'s vision is affected when the weather condition is determined as misty or foggy, hazy or smoggy;

conducting a process of image fusion to combine the visible image and the invisible image and obtain a fused image;

displaying the fused image to the driver to enhance the driver'"'"'s vision if the clarity of the visible image falls below the preset threshold; and

displaying the fused image when the driver'"'"'s vision is determined to be affected.

11. The method of claim 10, wherein the step of determining the clarity of the visible image may comprise:

comparing either a size, a size ratio, or a pixel amount of glaring for foggy areas identified on the visible images, or intensities of an ambient light between a critical area and its neighboring area.

12. The method of claim 10, wherein the invisible image is a thermal image obtained from a thermal imaging camera, or an infrared camera.

13. The method of claim 10, further comprising detecting one or more objects by a lidar or a radar, and generating an object image based on the detected objects.

14. The method of claim 13, further comprising conducting the process of image fusion to combine the visible image, the invisible image and the detected object image to obtain the fused image.

15. The method of claim 10, further comprising collecting the environmental data surrounding the vehicle, wherein the method further comprises determining whether to display the fused image by reference to the collected environmental data.

16. The method of claim 15, the collected environmental data may include a temperature degree, a humidity degree, and a particulate matter (PM) concentration.

17. The method of claim 15, further comprising displaying the collected environmental data together with the fused image to the driver.

18. The method of claim 10, wherein the visible image serves to mimic the driver'"'"'s vision from the vehicle.

19. The method of claim 10, further comprising:

determining whether the weather condition is hazy or smoggy based on the PM concentration, and

determining that the driver'"'"'s vision is affected when the weather condition is determined as hazy or smoggy.