METHOD FOR IMAGING THE SURROUNDING OF A VEHICLE

• US 20080049975A1
• Filed: 08/24/2007
• Published: 02/28/2008
• Est. Priority Date: 08/24/2006
• Status: Active Grant

• Current Assignee: Harman Becker Automotive Systems GmbH (Samsung Electronics Co. Ltd.)
• Original Assignee: Harman Becker Automotive Systems GmbH (Samsung Electronics Co. Ltd.)
• Inventors: Stiegler, Andreas
• Granted Patent: US 8,081,794 B2
• Time in Patent Office: Days
• US Class Current: 382/104
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

A method for imaging the surrounding of a vehicle is provided. The method includes the steps of (i) taking a series of two dimensional images of the vehicle surrounding, (ii) taking a series of three dimensional images of the vehicle surrounding that including depth information, (iii) determining at least one object in an area of interest in one of the series of images and tracking the object in the area of interest, (iv) processing the others of the series of images to retrieve object information from said other series of images, (v) adjusting the objects to be tracked in the one series of images in accordance with the object information received from processing the other series of images.

Claims

1. A method for imaging the surrounding of a vehicle, the method comprising the steps of:

taking a series of 2D images of the surrounding;

taking a series of 3D images of the surrounding, the 3D images including depth information;

determining at least one object in an area of interest in one of the series of images and tracking the object in the area of interest;

processing the other of the series of images to retrieve object information from the other series of images;

adjusting the objects to be tracked in the one series of images in accordance with the object information received from processing the other series of images.

2. The method of claim 1, where the 2D and the 3D images are processed to localize objects in a predetermined observation area of the images, the distance of which vary relative to the vehicle.

3. The method of claim 1, where the position and the velocity of the localized objects relative to the vehicle are determined.

4. The method of claim 1, where areas of interest are determined in the series of 2D images in which the movement of the objects is tracked.

5. The method of claim 1, where the distance of the objects relative to the vehicle is determined in the 3D images.

6. The method of claim 1, where the 3D images are processed to detect objects for which the distance change relative to the vehicle is larger than a predetermined value.

7. The method of claim 6, where the position and size information of the detected object retrieved from the 3D images is transmitted to a 2D imaging unit, the objects tracked in the 2D images being adapted to the information received from the 3D images.

8. The method of claim 6, where an additional area of interest in which the object detected in the 3D images is added in the 2D images.

9. The method of claim 1, where objects are identified and tracked in the 3D images together with the object'"'"'s position and velocity relative to the vehicle.

10. The method of claim 9, where objects are identified in the 2D images with their position, for which the change in distance relative to the vehicle is considered to be larger than a predetermined distance change.

11. The method of claim 10, further comprising the steps of:

transmitting the position of said object to a 3D imaging unit;

requesting the determination of the relative distance to the vehicle of the object for which the position was transmitted;

determining the relative distance of the object for which the position was determined in the 2D images; and

tracking said object also in the 3D images.

12. The method of claim 1, where the 2D images are taken at a frame rate between 50 and 100 frames per second, whereas the 3D images are taken at a frame rate between 10 and 100 frames per second.

13. The method of claim 1, where objects in the images are detected by using different filter algorithms applied to the image data.

14. The method of claim 1, where for the 2D images a first frame rate is used for the image acquisition of the areas of interest with the tracked objects, whereas a second frame rate is used for the image acquisition of the whole 2D images.

15. The method of claim 14, where the second frame rate is 10 to 50 times smaller than the first frame rate.

16. The method of claim 1, where, when an object is tracked, the position vector in the recorded image and a velocity vector considering a series of images are determined for the tracked object.

17. The method of claim 1, where an alarm signal is generated when it is determined in one of the series of images that an object is approaching the vehicle a predetermined relative speed.

18. The method of claim 17, where the alarm signal is transmitted to a cruise control unit of the vehicle.

19. The method of claim 17, where the driver is informed when an object is closer to the vehicle than a predetermined distance.

20. The method of claim 1, where the series of images is processed taking into account the driving direction of the vehicle.

21. A system for imaging a vehicle surrounding, the system comprising:

a 2D image sensor generating a series of 2D images of the vehicle surrounding;

a 3D image sensor generating 3D images of the vehicle surrounding, where the 3D images have depth information; and

an image processing unit processing one of the series of images for detecting objects in the images to be tracked and processing the other series of images to retrieve object information, where the image processing unit adjusts the objects to be tracked in one of series of images in accordance with the object information received from processing the other series of images.

22. The system of claim 21, where the 2D image sensor is a CMOS image sensor.

23. The system of claim 21, where the 2D image sensor is a CCED image sensor.

24. The system of claim 21, where the 3D image sensor is a photonic mixer device image sensor.

25. The system of claim 21, further including a 2D imaging unit comprising the 2D image sensor and a 2D sensor control unit.

26. The system of claim 21, further including a 3D imaging unit comprising the 3D image sensor and a 3D sensor control unit.

27. A cruise control system for a vehicle comprising a cruise control unit controlling the engine of the vehicle and further comprising:

a 2D image sensor generating a series of 2D images of the vehicle surroundings;

a 3D image sensor generating 3D images of the vehicle surroundings, the 3D images including depth information; and

an image processing unit processing one of series of images for detecting objects in the images to be tracked and processing the other series of images to retrieve object information, where the image processing unit adjusts the objects to be tracked in one of the series of images in accordance with the object information received from processing the other series of images.

28. The system of claim 27, where the 2D image sensor is a CMOS image sensor.

29. The system of claim 27, where the 2D image sensor is a CCED image sensor.

30. The system of claim 27, where the 3D image sensor is a photonic mixer device image sensor.

31. A method for controlling a function in a vehicle, the method comprising the steps of:

detecting a series of 2D images of the vehicle surroundings;

detecting a series of 3D images of the vehicle surroundings, the 3D images including depth information;

processing one of the series of images to find objects to be tracked;

tracking the objects in the one series of images;

processing both series of images to retrieve object information;

adjusting the objects to be tracked in the one series of images according to the object information retrieved from processing the other series of images; and

controlling a function of the vehicle based on the retrieved object information.

32. The method of claim 31, where processing of both series of images comprises;

processing of at least one area of interest comprising at least one object to be tracked in the series of 2D images to retrieve object position or object type information; and

processing the series of 3D images to retrieve information comprising distance information on at least one tracked object.

33. The method of claim 31, where the controlling of a function includes controlling of an acceleration or deceleration of the vehicle.

34. The method of claim 31, where the controlling of a function includes the deployment of passenger or pedestrian protection equipment.

35. The method of claim 31, where the controlling a function includes notifying a vehicle passenger via a driver interface unit of an approaching object.