System, apparatus, and method of detecting and displaying obstacles and data associated with the obstacles

• US 9,580,014 B2
• Filed: 08/08/2014
• Issued: 02/28/2017
• Est. Priority Date: 08/08/2013
• Status: Active Grant

• Current Assignee: Convoy Technologies, Inc.
• Original Assignee: Convoy Technologies LLC
• Inventors: Lucas, Martin, Gasca, Blake, Cheng, Edward, Rebelo, Daniel, Sullivan, Christian
• Primary Examiner(s): SHAH, UTPAL D
• Application Number:
  US14/455,818
• Publication Number:
  US 20150043782A1
• Time in Patent Office: 935 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/30 providing vision in the non-visible spectrum, e.g. night or infrared vision
• CPC Class Codes: B60R 2300/103 using camera systems provided with artificial illumination device, e.g. IR light source
• CPC Class Codes: B60R 2300/107 using stereoscopic cameras
• CPC Class Codes: B60R 2300/202 displaying a blind spot scene on the vehicle part responsible for the blind spot
• CPC Class Codes: B60R 2300/802 for monitoring and displaying vehicle exterior blind spot views
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

A method of detecting and displaying the obstacles and data related to obstacles on a digital display starts with a detection processing unit receiving a digital video input signal from a first camera and depth data from a second camera. Digital input video input signal may include images of an area surrounding the vehicle. Digital video input signal may be analyzed to determine whether an obstacle is present in the area. When an obstacle is detected, the depth data may be analyzed to obtain a distance associated with the obstacle. Digital video output may be generated that includes the digital video input signal and a display of the distance associated with the obstacle. The display of the distance may be overlaid on the display of the obstacle. Digital video output may then be transmitted to the display device to be displayed to the user. Other embodiments are also described.

Claims

1. A method of detecting obstacles surrounding a vehicle and displaying the obstacles and data related to the obstacles on a digital display inside the vehicle, the method comprising:

receiving by a processor of a detection processing unit a digital video input signal from a first camera and depth data from a second camera, wherein the digital input video input signal includes images of an area surrounding the vehicle;

analyzing by the processor the digital video input signal to determine whether at least one obstacle is present in the area surrounding the vehicle; and

when the at least one obstacle is detected,

analyzing by the processor the depth data to obtain a distance associated with the at least one obstacle,

generating by the processor a digital video output including the digital video input signal and a display of the distance associated with the at least one obstacle overlaid on the obstacle in the digital video input signal, and

transmitting by the processor the digital video output to the display device, and when a plurality of obstacles is detected,

generating by the processor a list prioritizing the plurality of obstacles based on at least one of;

(i) an identification of the at least one obstacle,

(ii) the distance associated with the at least one obstacle, and

(iii) a calculated time the at least one obstacle will enter a blind spot.

2. The method of claim 1, wherein the at least one obstacle is a moving object or a non-moving object.

3. The method of claim 1, wherein analyzing the depth data to obtain the distance associated with the obstacle further comprises:

capturing by the second camera a video image of infrared (IR) light on the at least one obstacle, wherein the infrared light is outputted from an IR light source mounted on the vehicle; and

calculating by the processor a distance between the IR light source and the at least one obstacle based on the IR light captured.

4. The method of claim 1, further comprising:

receiving by the processor sensor data from 3-Dimensional (3D) sensors;

analyzing the sensor data to detect the presence of the at least one obstacle in the area surrounding the vehicle and to obtain the distance associated with the at least one obstacle.

5. The method of claim 4, further comprising:

calculating by the processor the calculated time at which the at least one obstacle will enter the blind spot of the vehicle.

6. The method of claim 5, wherein the digital video output further includes a display of the calculated time.

7. The method of claim 6, further comprising:

determining by the processor a relevance of the at least one obstacle based on at least one of the sensor data, the depth data, and the digital video input signal,

wherein determining the relevance of the at least one obstacle includes by the processor performing at least one of;

(i) comparing factors associated with an image of the at least one obstacle in the digital video input signal to known factors stored in a memory storage device to identify the at least one obstacle,

(ii) comparing the distance associated with the at least one obstacle with a threshold distance, and

(iii) comparing the time at which when the at least one obstacle will enter the blind spot with a threshold time.

8. The method of claim 4, wherein the digital video input signal, the depth data, and the sensor data are received via a wired or a wireless connection.

9. The method of claim 1, wherein the digital video output is transmitted to the display device via a wired or a wireless connection.

10. The method of claim 1, further comprising, when the at least one obstacle is detected, transmitting an audio output to the digital device, the audio output provides an audible alert.

11. A system of detecting obstacles surrounding a vehicle and displaying the obstacles and data related to the obstacles, the system comprising:

an infrared light source to generate an infrared light;

a first camera to generate a digital video input signal that includes images of an area surrounding the vehicle;

a second camera to generate depth data,

a detection processing unit coupled to the first and second cameras, the object detection processing unit including;

a processor,

a storage device including instructions which, when executed by the processor, causes the processor to;

analyze the digital video input signal to determine whether at least one obstacle is present in the area surrounding the vehicle,

when the at least one obstacle is detected,

(i) analyze the depth data to obtain a distance associated with the at least one obstacle, wherein the depth data generated by the second camera is associated with the at least one obstacle in the area surrounding the vehicle, wherein the depth data is a video image of the infrared (IR) light on at least one obstacle in the area surrounding the vehicle, and

(ii) generate a digital video output including the digital video input signal and a display of the distance associated with the at least one obstacle overlaid on the obstacle in the digital video input signal, and

when a plurality of obstacles is detected, a list prioritizing the plurality of obstacles based on at least one of;

(i) an identification of the at least one obstacle,

(ii) the distance associated with the at least one obstacle, and

(iii) a calculated time the at least one obstacle will enter a blind spot, and

a communication interface to

(i) receive the digital video input signal from the first camera and the depth data from the second camera, and

(ii) transmit the digital video output; and

a digital display to display the digital video output received from the communication interface of the detection processing unit.

12. The system of claim 11, further comprising:

a plurality of 3D sensors to generate sensor data, wherein the processor analyzes the sensor data to detect the presence of the at least one obstacle in the area surrounding the vehicle.

13. An apparatus for detecting obstacles surrounding a vehicle and displaying the obstacles and data related to the obstacles, the apparatus comprising:

a processor,

a communication interface coupled to the processor to receive a digital video input signal from a first camera and depth data from a second camera; and

a storage device including instructions which, when executed by the processor, causes the processor to;

analyze the digital video input signal to determine whether at least one obstacle is present in the area surrounding the vehicle,

when the at least one obstacle is detected,

(i) analyze the depth data to obtain a distance associated with the at least one obstacle, wherein the depth data generated by the second camera is associated with the at least one obstacle in the area surrounding the vehicle, wherein the depth data is a video image of the infrared (IR) light on at least one obstacle in the area surrounding the vehicle, and

(ii) generate a digital video output including the digital video input signal and a display of the distance associated with the at least one obstacle overlaid on the obstacle in the digital video input signal, and

(iii) transmit the digital video output to a digital device via the communication interface,

when a plurality of obstacles is detected, a list prioritizing the plurality of obstacles based on at least one of;

(i) an identification of the at least one obstacle,

(ii) the distance associated with the at least one obstacle, and

(iii) a calculated time the at least one obstacle will enter a blind spot.

14. The apparatus of claim 13, wherein analyzing the depth data to obtain the distance associated with the obstacle further comprises the processor

to signal to the second camera to capture a video image of infrared (IR) light on the at least one obstacle, wherein the infrared light is outputted from an IR light source mounted on the vehicle;

and

to calculate a distance between the IR light source and the at least one obstacle based on the IR light captured.

15. The apparatus of claim 13, wherein the instructions, when executed by the processor, further causes the processor to:

receive sensor data from 3-Dimensional (3D) sensors; and

analyze the sensor data to detect the presence of the at least one obstacle in the area surrounding the vehicle and to obtain the distance associated with the at least one obstacle.

16. The apparatus of claim 13, wherein the instructions, when executed by the processor, further causes the processor to:

calculate the calculated time at which the at least one obstacle will enter the blind spot of the vehicle.

17. The apparatus of claim 16, wherein the digital video output further includes a display of the calculated time.

18. The method of claim 17, wherein the instructions, when executed by the processor, further causes the processor to:

determine a relevance of the at least one obstacle based on at least one of the sensor data, the depth data, and the digital video input signal,

wherein determining the relevance of the at least one obstacle includes at least one of;

(iv) comparing factors associated with an image of the at least one obstacle in the digital video input signal to known factors stored in a memory storage device to identify the at least one obstacle,

(v) comparing the distance associated with the at least one obstacle with a threshold distance, and

(vi) comparing the time at which when the at least one obstacle will enter the blind spot with a threshold time.