Method for imaging the surrounding of a vehicle

• US 8,081,794 B2
• Filed: 08/24/2007
• Issued: 12/20/2011
• Est. Priority Date: 08/24/2006
• Status: Active Grant

• Current Assignee: Harman Becker Automotive Systems GmbH (Samsung Electronics Co. Ltd.)
• Original Assignee: Harman Becker Automotive Systems GmbH (Samsung Electronics Co. Ltd.)
• Inventors: Stiegler, Andreas
• Primary Examiner(s): Ahmed, Samir
• Assistant Examiner(s): RASHIDIAN, MOHAMMAD M
• Application Number:
  US11/844,890
• Publication Number:
  US 20080049975A1
• Time in Patent Office: 1,579 Days
• Field of Search: 382/100, 382/103, 382/104, 382/107, 382/224, 382/278, 348/143, 348/169, 345/7, 345/633, 180/168, 280/735, 367/137, 250/578.1, 701/1, 701/45
• US Class Current: 382/100
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

A method for imaging the surrounding of a vehicle is provided. The method includes the steps of (i) taking a series of two dimensional images of the vehicle surrounding, (ii) taking a series of three dimensional images of the vehicle surrounding that including depth information, (iii) determining at least one object in an area of interest in one of the series of images and tracking the object in the area of interest, (iv) processing the others of the series of images to retrieve object information from said other series of images, (v) adjusting the objects to be tracked in the one series of images in accordance with the object information received from processing the other series of images.

Claims

1. A method for imaging the surrounding of a vehicle, the method comprising the steps of:

taking a series of 2D images of the surrounding;

taking a series of 3D images of the surrounding, the 3D images including depth information;

determining at least one object in an area of interest in one of the series of images and tracking the object in the area of interest where objects are identified in the 3D images together with the object'"'"'s position and velocity relative to the vehicle and identified in the 2D images with their position, for which the change in distance relative to the vehicle is considered to be larger than a predetermined distance change;

processing the other of the series of images to retrieve object information from the other series of images;

adjusting the objects to be tracked in the one series of images in accordance with the object information received from processing the other series of images transmitting the position of said object to a 3D imaging unit;

requesting the determination of the relative distance to the vehicle of the object for which the position was transmitted;

determining the relative distance of the object for which the position was determined in the 2D images; and

tracking said object also in the 3D images.

2. The method of claim 1, where the 2D and the 3D images are processed to localize objects in a predetermined observation area of the images, the distance of which vary relative to the vehicle.

3. The method of claim 1, where the position and the velocity of the localized objects relative to the vehicle are determined.

4. The method of claim 1, where areas of interest are determined in the series of 2D images in which the movement of the objects is tracked.

5. The method of claim 1, where the distance of the objects relative to the vehicle is determined in the 3D images.

6. The method of claim 1, where the 3D images are processed to detect objects for which the distance change relative to the vehicle is larger than a predetermined value.

7. The method of claim 6, where the position and size information of the detected object retrieved from the 3D images is transmitted to a 2D imaging unit, the objects tracked in the 2D images being adapted to the information received from the 3D images.

8. The method of claim 6, where an additional area of interest in which the object detected in the 3D images is added in the 2D images.

9. The method of claim 1, where the 2D images are taken at a frame rate between 50 and 100 frames per second, whereas the 3D images are taken at a frame rate between 10 and 100 frames per second.

10. The method of claim 1, where objects in the images are detected by using different filter algorithms applied to the image data.

11. The method of claim 1, where for the 2D images a first frame rate is used for the image acquisition of the areas of interest with the tracked objects, whereas a second frame rate is used for the image acquisition of the whole 2D images.

12. The method of claim 11, where the second frame rate is 10 to 50 times smaller than the first frame rate.

13. The method of claim 1, where, when an object is tracked, the position vector in the recorded image and a velocity vector considering a series of images are determined for the tracked object.

14. The method of claim 1, where an alarm signal is generated when it is determined in one of the series of images that an object is approaching the vehicle a predetermined relative speed.

15. The method of claim 14, where the alarm signal is transmitted to a cruise control unit of the vehicle.

16. The method of claim 14, where the driver is informed when an object is closer to the vehicle than a predetermined distance.

17. The method of claim 1, where the series of images is processed taking into account the driving direction of the vehicle.

18. A system for imaging a vehicle surrounding, the system comprising:

a 2D image sensor generating a series of 2D images of the vehicle surrounding;

a 3D image sensor generating 3D images of the vehicle surrounding, where the 3D images have depth information;

an image processing unit processing one of the series of images for detecting objects in the images to be tracked and processing the other series of images to retrieve object information, where objects are identified in the 3D images together with the object'"'"'s position and velocity relative to the vehicle and identified in the 2D images with their position, for which the change in distance relative to the vehicle is considered to be larger than a predetermined distance change and where the image processing unit adjusts the objects to be tracked in one of series of images in accordance with the object information received from processing the other series of images and determining a relative distance to the vehicle of the objects by determination of the relative distance of said objects for which the position was determined in the 2D images;

transmitting the position of said object to a 3D imaging unit; and

tracking said object also in the 3D images.

19. The system of claim 18, where the 2D image sensor is a CMOS image sensor.

20. The system of claim 18, where the 2D image sensor is a CCED image sensor.

21. The system of claim 18, where the 3D image sensor is a photonic mixer device image sensor.

22. The system of claim 18, further including a 2D imaging unit comprising the 2D image sensor and a 2D sensor control unit.

23. The system of claim 18, further including a 3D imaging unit comprising the 3D image sensor and a 3D sensor control unit.

24. A cruise control system for a vehicle comprising a cruise control unit controlling the engine of the vehicle and further comprising:

a 2D image sensor generating a series of 2D images of the vehicle surroundings;

a 3D image sensor generating 3D images of the vehicle surroundings, the 3D images including depth information;

an image processing unit processing one of series of images for detecting objects in the images to be tracked and processing the other series of images to retrieve object information, where objects are identified in the 3D images together with the object'"'"'s position and velocity relative to the vehicle and identified in the 2D images with their position, for which the change in distance relative to the vehicle is considered to be larger than a predetermined distance change and where the image processing unit adjusts the objects to be tracked in one of the series of images in accordance with the object information received from processing the other series of images and determining a relative distance to the vehicle of the objects by determination of the relative distance of the object for which the position was determined in the 2D images;

transmitting the position of said object to a 3D imaging unit; and

tracking said object also in the 3D images.

25. The system of claim 24, where the 2D image sensor is a CMOS image sensor.

26. The system of claim 24, where the 2D image sensor is a CCED image sensor.

27. The system of claim 24, where the 3D image sensor is a photonic mixer device image sensor.