COMBINED TIME-OF-FLIGHT AND IMAGE SENSOR SYSTEMS

• US 20120044093A1
• Filed: 08/20/2010
• Published: 02/23/2012
• Est. Priority Date: 08/20/2010
• Status: Active Grant

• Current Assignee: Denso International America Incorporated (DENSO Corporation)
• Original Assignee: Denso International America Incorporated (DENSO Corporation)
• Inventors: Pala, Silviu
• Granted Patent: US 8,648,702 B2
• Time in Patent Office: Days
• US Class Current: 340/963
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 7/4863 Detector arrays, e.g. charge-transfer gates
• CPC Class Codes: H04N 25/705 Pixels for depth measurement, e.g. RGBZ

Abstract

An object detection system of a vehicle includes a time-of-flight (TOF) sensor that receives a reflected object detection signal at a second time based on an object detection signal transmitted at a first time. An image sensor generates an image signal including an image of a detected object. The image sensor is distinct from and adjacent to the TOF sensor. A TOF control module generates distance data based on the first time and the second time and determines whether at least a portion of a detected object is within a predetermined distance of the vehicle. An image control module generates image data based on the image signal. A detection control module correlates the distance data with the image data to generate a warning indicator when at least a portion of the detected object is within the predetermined distance of the vehicle.

Claims

1. An object detection system of a vehicle comprising:

a time-of-flight (TOF) sensor receives a reflected object detection signal at a second time based on an object detection signal transmitted at a first time;

an image sensor that generates an image signal including an image of a detected object, wherein the image sensor is distinct from and adjacent to the TOF sensor;

a TOF control module that generates distance data based on the first time and the second time and determines whether at least a portion of a detected object is within a predetermined distance of the vehicle;

an image control module that generates image data based on the image signal; and

a detection control module that correlates the distance data with the image data to generate a warning indicator when at least a portion of the detected object is within the predetermined distance of the vehicle.

2. The object detection system of claim 1, further comprising:

a display; and

a graphics control module that displays the detected object and the warning indicator on the display,

wherein the warning indicator indicates that the detected object is within the predetermined distance.

3. The object detection system of claim 1, further comprising:

a first chip package comprising the TOF sensor; and

a second chip package comprising the image sensor,

wherein the second chip package is adjacent to and in contact with the first chip package.

4. The object detection system of claim 1, wherein the TOF sensor is within a predetermined distance of the image sensor.

5. The object detection system of claim 1, wherein the detection control module correlates the distance data with the image data based on at least one of an angular offset, a horizontal offset, a vertical offset, and a longitudinal offset between the TOF sensor and the image sensor.

6. The object detection system of claim 1, further comprising a graphics control module, wherein:

the detection control module correlates the distance data with the image data to generate resultant correlated data based on resolution of the TOF sensor and resolution of the image sensor; and

the graphics control module, based on the resultant correlated data, at least one of (i) alters pixels of the display associated with the portions of the detected object that are within the predetermined distance and (ii) displays the warning indicator over a portion of the detected object on the display.

7. The object detection system of claim 1, wherein the detection control module:

dithers in at least one of space and time the distance data to generate dithered data; and

correlates the dithered data with the image data.

8. An object detection system comprising:

a substrate comprising;

a time-of-flight (TOF) sensor that includes N pixel sensors and receives a reflected TOF signal at a second time based on a TOF signal transmitted at a first time, where N is an integer;

an image sensor that (i) includes M pixel sensors distinct from the N pixel sensors and (ii) generates an image signal and that is adjacent to the TOF sensor, where M is an integer;

a TOF control module that (i) generates distance data based on the first time and the second time and (ii) determines whether at least a portion of a detected object is within a predetermined distance of a vehicle;

an image control module that generates image data based on the image signal; and

a detection control module that correlates the distance data with the image data to generate a first warning indicator when at least a portion of the detected object is within the predetermined distance of the vehicle.

9. The object detection system of claim 8, further comprising a chip comprising the substrate.

10. The object detection system of claim 8, wherein:

the TOF sensor comprises a first array of the N pixel sensors; and

the image sensor comprises a second array of the M pixel sensors,

wherein the second array of the M pixel sensors are interlaced with the first array of the N pixel sensors.

11. The object detection system of claim 10, wherein the second array of the M pixel sensors is interlaced at least one of vertically and horizontally with the first array of the N pixel sensors.

12. The object detection system of claim 8, further comprising a graphics control module, wherein:

the detection control module correlates the distance data with the image data to generate resultant correlated data based on resolution of the TOF sensor and resolution of the image sensor; and

the graphics control module, based on the resultant correlated data, at least one of (i) alters pixels of the display associated with the portions of the detected object that are within the predetermined distance and (ii) displays the warning indicator over a portion of the detected object on the display.

13. The object detection system of claim 8, wherein the detection control module correlates the distance data with the image data based on groupings of the N pixel sensors with the M pixel sensors.

14. The object detection system of claim 8, further comprising:

a display; and

a graphics control module that displays the detected object and the first warning indicator on the display,

wherein the detection control module;

determines when portions of the detected object are with a warning distance and within a critical distance of a vehicle;

generates the first warning indicator when a portion of the detected object is within the warning distance;

generates a second warning indicator when a portion of the detected object is within the critical distance; and

displays the second warning indicator on the display, and

wherein the first warning indicator is different than the second warning indicator.

15. The object detection system of claim 8, wherein:

N is less than M; and

surface area of the N pixel sensors is equal to surface area of the M pixel sensors.

16. A method of operating an object detection system comprising:

generating a correspondence matrix between time-of-flight (TOF) pixel sensors of a TOF sensor and image pixel sensors of an image sensor;

receiving distance data from the TOF sensor;

receiving image data from the image sensor;

comparing the distance data to a warning distance and generating a warning indicator;

determining a first correspondence between a first subset of the TOF pixel sensors and at least one of (i) the image pixel sensors and (ii) pixels of a display, wherein the first subset of the TOF pixel sensors is associated with distances of a detected object that are within the warning distance; and

displaying the detected object and the warning indicator according to the first correspondence.

17. The method of claim 16, further comprising:

comparing the distance data to a critical distance and generating a critical indicator;

determining a second correspondence between a second subset of the TOF pixel sensors and at least one of (i) the image pixel sensors and (ii) pixels of the display, wherein the second subset of the TOF pixel sensors are associated with distances of the detected object that are within the critical distance; and

displaying the critical indicator according to the second correspondence,

wherein the warning indicator is different than the critical indicator.

18. The method of claim 17, further comprising:

increasing luminance and adjusting color of at least one of (i) the image pixel sensors and (ii) the pixels of the display corresponding to the first subset of the TOF pixel sensors when displaying the warning indicator; and

the critical indicator includes adjusting color of at least one of (i) the image pixel sensors and (ii) the pixels of the display corresponding to the second subset of the TOF pixel sensors when displaying the critical indicator.

19. The method of claim 18, further comprising:

highlighting at least one of (i) the image pixel sensors and (ii) the pixels of the display corresponding to the first subset of the TOF pixel sensors when displaying the warning indicator; and

blinking at least one of (i) the image pixel sensors and (ii) the pixels of the display corresponding to the second subset of the TOF pixel sensors when displaying the critical indicator.

20. The method of claim 19, further comprising:

generating a second warning indicator when distance data associated with TOF pixels of the TOF sensor indicates that a portion of the detected object is within the warning distance; and

generating a second critical indicator when distance data associated with TOF pixels of the TOF sensor indicates that a portion of the detected object is within the critical distance,

wherein the second warning indicator includes at least one of a first voice message and a first vibration indicator, and

wherein the second critical indicator includes at least one of a second voice message and a second vibration indicator.