Object detection system

• US 10,295,667 B2
• Filed: 06/29/2016
• Issued: 05/21/2019
• Est. Priority Date: 11/07/2007
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Lu, Yuesheng, Higgins-Luthman, Michael J.
• Primary Examiner(s): Ratcliffe, Luke D
• Application Number:
  US15/196,076
• Publication Number:
  US 20160306041A1
• Time in Patent Office: 1,056 Days
• CPC Class Codes: B60Q 9/008 for anti-collision purposes
• CPC Class Codes: B60R 2300/8033 for pedestrian protection
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: G01C 3/08 Use of electric radiation detectors
• CPC Class Codes: G01S 17/58 Velocity or trajectory determination systems; Sense-of-movement determination systems
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 7/4802 using analysis of echo signal for target characterisation; Target signature; Target cross-section
• CPC Class Codes: G01S 7/4817 relating to scanning
• CPC Class Codes: G01S 7/4865 Time delay measurement, e.g. time-of-flight measurement, time of arrival measurement or determining the exact position of a peak peak detection in noise, signal conditioning G01S7/487
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 2201/07 Target detection
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/81 : for suppressing or minimising disturbance in the image signal generation
• CPC Class Codes: H04N 5/33 : Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

An object detection system for a vehicle includes a camera vision module and a Lidar module. The camera vision module includes an imaging device viewing to the exterior of the vehicle and operable to capture image data representative of a scene exterior and forward of the vehicle. The Lidar module includes a Lidar device that, with the Lidar module mounted at a front exterior portion of the vehicle, scans a region forward of the vehicle that overlaps with the field of view of the imaging device. Based at least in part on processing of captured image data by an image processor using vision processing algorithms and based at least in part on distance data provided by the Lidar device, 3-dimensional information relative to the vehicle of an object present exterior of the vehicle is algorithmically constructed.

Claims

1. An object detection system suitable for use in a vehicle, said object detection system comprising:

a camera vision module and a Lidar module;

wherein said camera vision module is configured for mounting at an in-cabin side of a windshield of a vehicle equipped with said object detection system;

wherein said Lidar module is configured for mounting at a front exterior portion of the vehicle equipped with said object detection system;

wherein said camera vision module comprises an imaging device viewing to the exterior of the equipped vehicle when said camera vision module is mounted at the windshield of the equipped vehicle, said imaging device having a field of view exterior of the equipped vehicle and operable to capture image data representative of a scene exterior the equipped vehicle;

said imaging device comprising a two-dimensional pixelated imaging array having a plurality of photo-sensing pixels arranged on a semiconductor substrate;

wherein said Lidar module comprises a Lidar device that, with said Lidar module mounted at the front exterior portion of the equipped vehicle, scans a region forward of the equipped vehicle;

wherein the scanned region overlaps with the field of view of said imaging device;

wherein, based at least in part on processing of captured image data by an image processor using vision processing algorithms and based at least in part on distance data provided by said Lidar device, 3-dimensional information relative to the equipped vehicle of an object present exterior of the equipped vehicle is algorithmically constructed;

wherein said 3-dimensional information comprises (i) the X and Y coordinates of the object present exterior of the equipped vehicle, (ii) the distance (Z) between the object and the equipped vehicle and (iii) the time of possible collision of the object and the equipped vehicle; and

wherein vehicles present in the field of view of said imaging device are assigned high priority scores when they have one or more characteristic selected from the group comprising (i) are closing with the equipped vehicle and collision will happen in a pre-determined time, (ii) are in the same lane as that of the equipped vehicle and (iii) are moving to the same lane as that of the equipped vehicle so as to cut in the lane at a close distance in front of the equipped vehicle.

2. The object detection system of claim 1, wherein, responsive at least in part to image processing of captured image data by said image processor, a light beam emitted by a light source of the equipped vehicle impinges the object present exterior of the equipped vehicle.

3. The object detection system of claim 1, wherein said Lidar device comprises a light source, a detector and a scanner.

4. The object detection system of claim 3, wherein said scanner comprises at least one mirror.

5. The object detection system of claim 3, wherein said scanner comprises a micro mirror array.

6. The object detection system of claim 3, wherein said Lidar device is operable to determine distance to the object present exterior of the equipped vehicle via a time of flight calculation.

7. The object detection system of claim 1, wherein said image processor processes captured image data to detect multiple objects in the field of view of said imaging device, and wherein, at least in part responsive to image processing by said image processor of captured image data, said system prioritizes an individual object of said detected multiple objects based on at least one of (i) proximity of the individual object present exterior of the equipped vehicle to the equipped vehicle, (ii) speed of the individual object present exterior of the equipped vehicle relative to the equipped vehicle and (iii) location of the individual object present exterior of the equipped vehicle relative to the equipped vehicle.

8. The object detection system of claim 7, wherein the individual object present exterior of the equipped vehicle comprises a vehicle, and wherein, at least in part responsive to image processing by said image processor of captured image data, speed of the vehicle present exterior of the equipped vehicle relative to the equipped vehicle is calculated.

9. The object detection system of claim 7, wherein the individual object present in the field of view of said imaging device is assigned a priority score that represents the potential danger of the individual object to the equipped vehicle, and wherein the priority score is at least in part established responsive (i) to image processing by said image processor of captured image data and (ii) to distance between the individual object and the equipped vehicle.

10. The object detection system of claim 9, wherein the individual object is a pedestrian who is present in the field of view of said imaging device.

11. The object detection system of claim 10, wherein a light beam emitted from the equipped vehicle at least one of (a) tracks the pedestrian while the pedestrian moves in the field of view of said imaging device and (b) stays on the pedestrian while the pedestrian moves in the field of view of said imaging device.

12. The object detection system of claim 1, wherein, at least in part responsive to image processing by said image processor of captured image data, vehicles present in the field of view of said imaging device are assigned priority scores that represents their potential danger to the equipped vehicle.

13. The object detection system of claim 1, wherein, at least in part responsive to image processing by said image processor of captured image data, a pedestrian present in the field of view of said imaging device is assigned a high priority score when the pedestrian is moving toward the center of the field of view of said imaging device.

14. The object detection system of claim 1, wherein said front exterior portion at which said Lidar module is configured for mounting comprises a front fender portion of the equipped vehicle.

15. The object detection system of claim 1, wherein said 3-dimensional information relative to the equipped vehicle of the object present exterior of the equipped vehicle is algorithmically constructed based at least in part on distance data provided by said Lidar device determined by time of flight calculations.

16. The object detection system of claim 15, wherein said image processor processes captured image data to detect multiple objects in the field of view of said imaging device, and wherein, at least in part responsive to image processing by said image processor of captured image data, said system prioritizes an individual object of said detected multiple objects.

17. The object detection system of claim 16, wherein, at least in part responsive to image processing by said image processor of captured image data, said system prioritizes the individual object of said detected multiple objects based on (i) proximity of the individual object present exterior of the equipped vehicle to the equipped vehicle, (ii) speed of the individual object present exterior of the equipped vehicle relative to the equipped vehicle and (iii) location of the individual object present exterior of the equipped vehicle relative to the equipped vehicle.

18. The object detection system of claim 17, wherein the individual object of said detected multiple objects comprises a pedestrian.

19. An object detection system suitable for use in a vehicle, said object detection system comprising:

a camera vision module and a Lidar module;

wherein said camera vision module is configured for mounting at an in-cabin side of a windshield of a vehicle equipped with said object detection system;

wherein said Lidar module is configured for mounting at a front exterior portion of the vehicle equipped with said object detection system;

wherein said camera vision module comprises an imaging device viewing to the exterior of the equipped vehicle when said camera vision module is mounted at the windshield of the equipped vehicle, said imaging device having a field of view exterior of the equipped vehicle and operable to capture image data representative of a scene exterior the equipped vehicle;

said imaging device comprising a two-dimensional pixelated imaging array having a plurality of photo-sensing pixels arranged on a semiconductor substrate;

wherein said Lidar module comprises a Lidar device that, with said Lidar module mounted at the front exterior portion of the equipped vehicle, scans a region forward of the equipped vehicle;

wherein the scanned region overlaps with the field of view of said imaging device;

wherein, based at least in part on processing of captured image data by an image processor using vision processing algorithms and based at least in part on distance data provided by said Lidar device, 3-dimensional information relative to the equipped vehicle of an object present exterior of the equipped vehicle is algorithmically constructed;

wherein said image processor processes captured image data to detect multiple objects in the field of view of said imaging device;

wherein, at least in part responsive to image processing by said image processor of captured image data, said system prioritizes an individual object of said detected multiple objects based on at least one of (i) proximity of the individual object present exterior of the equipped vehicle to the equipped vehicle, (ii) speed of the individual object present exterior of the equipped vehicle relative to the equipped vehicle and (iii) location of the individual object present exterior of the equipped vehicle relative to the equipped vehicle;

wherein, at least in part responsive to image processing by said image processor of captured image data, said system prioritizes the individual object of said detected multiple objects based on at least one of (i) the individual object being closing with the equipped vehicle and collision will happen in a pre-determined time, (ii) the individual object being in the same lane as that of the equipped vehicle and (iii) the individual object being moving to the same lane as that of the equipped vehicle so as to cut in the lane at a close distance in front of the equipped vehicle; and

wherein the individual object present exterior of the equipped vehicle comprises a vehicle.

20. The object detection system of claim 19, wherein said 3-dimensional information comprises (i) the X and Y coordinates of the object present exterior of the equipped vehicle and (ii) the distance (Z) between the object and the equipped vehicle.

21. The object detection system of claim 20, wherein said 3-dimensional information comprises the time of possible collision of the object and the equipped vehicle.

22. The object detection system of claim 21, wherein, at least in part responsive to image processing by said image processor of captured image data, speed of the vehicle present exterior of the equipped vehicle relative to the equipped vehicle is calculated.

23. The object detection system of claim 19, wherein, at least in part responsive to image processing by said image processor of captured image data, said system prioritizes the individual object of said detected multiple objects based on (i) proximity of the individual object present exterior of the equipped vehicle to the equipped vehicle, (ii) speed of the individual object present exterior of the equipped vehicle relative to the equipped vehicle and (iii) location of the individual object present exterior of the equipped vehicle relative to the equipped vehicle.

24. An object detection system suitable for use in a vehicle, said object detection system comprising:

a camera vision module and a Lidar module;

wherein said camera vision module is configured for mounting at an in-cabin side of a windshield of a vehicle equipped with said object detection system;

wherein said Lidar module is configured for mounting at a front exterior portion of the vehicle equipped with said object detection system;

wherein said camera vision module comprises an imaging device viewing to the exterior of the equipped vehicle when said camera vision module is mounted at the windshield of the equipped vehicle, said imaging device having a field of view exterior of the equipped vehicle and operable to capture image data representative of a scene exterior the equipped vehicle;

said imaging device comprising a two-dimensional pixelated imaging array having a plurality of photo-sensing pixels arranged on a semiconductor substrate;

wherein said Lidar module comprises a Lidar device that, with said Lidar module mounted at the front exterior portion of the equipped vehicle, scans a region forward of the equipped vehicle;

wherein the scanned region overlaps with the field of view of said imaging device;

wherein, based at least in part on processing of captured image data by an image processor using vision processing algorithms and based at least in part on distance data provided by said Lidar device, 3-dimensional information relative to the equipped vehicle of an object present exterior of the equipped vehicle is algorithmically constructed;

wherein said image processor processes captured image data to detect multiple objects in the field of view of said imaging device;

wherein, at least in part responsive to image processing by said image processor of captured image data, said system prioritizes an individual object of said detected multiple objects based on at least one of (i) proximity of the individual object present exterior of the equipped vehicle to the equipped vehicle, (ii) speed of the individual object present exterior of the equipped vehicle relative to the equipped vehicle and (iii) location of the individual object present exterior of the equipped vehicle relative to the equipped vehicle;

wherein, at least in part responsive to image processing by said image processor of captured image data, said system prioritizes the individual object of said detected multiple objects based on at least one of (i) the individual object being closing with the equipped vehicle and collision will happen in a pre-determined time, (ii) the individual object being in the same lane as that of the equipped vehicle and (iii) the individual object being moving to the same lane as that of the equipped vehicle so as to cut in the lane at a close distance in front of the equipped vehicle; and

wherein the individual object present exterior of the equipped vehicle comprises a pedestrian present in the field of view of said imaging device.

25. The object detection system of claim 24, wherein, at least in part responsive to image processing by said image processor of captured image data, the pedestrian present in the field of view of said imaging device is assigned a high priority score when the pedestrian is moving toward the center of the field of view of said imaging device.

26. The object detection system of claim 24, wherein said Lidar device comprises a light source, a detector and a scanner, and wherein said scanner comprises at least one mirror, and wherein said Lidar device is operable to determine distance to the object present exterior of the equipped vehicle via a time of flight calculation.

27. The object detection system of claim 24, wherein, at least in part responsive to image processing by said image processor of captured image data, said system prioritizes the individual object of said detected multiple objects based on (i) proximity of the individual object present exterior of the equipped vehicle to the equipped vehicle, (ii) speed of the individual object present exterior of the equipped vehicle relative to the equipped vehicle and (iii) location of the individual object present exterior of the equipped vehicle relative to the equipped vehicle.

28. The object detection system of claim 24, wherein said 3-dimensional information relative to the equipped vehicle of the object present exterior of the equipped vehicle is algorithmically constructed based at least in part on distance data provided by said Lidar device determined by time of flight calculations.

29. The object detection system of claim 28, wherein said 3-dimensional information comprises (i) the X and Y coordinates of the object present exterior of the equipped vehicle and (ii) the distance (Z) between the object and the equipped vehicle.

30. The object detection system of claim 29, wherein said 3-dimensional information comprises the time of possible collision of the object and the equipped vehicle.

31. The object detection system of claim 24, wherein the individual object present in the field of view of said imaging device is assigned a priority score that represents the potential danger of the individual object to the equipped vehicle, and wherein the priority score is at least in part established responsive (i) to image processing by said image processor of captured image data and (ii) to distance between the individual object and the equipped vehicle.