Free space monitoring system for motor vehicles

• US 7,136,754 B2
• Filed: 04/12/2004
• Issued: 11/14/2006
• Est. Priority Date: 04/11/2003
• Status: Expired due to Fees

• Current Assignee: Bayerische Motoren Werke AG
• Original Assignee: Daimler Chrysler Company LLC (Stellantis NV)
• Inventors: Toepfer, Christian Friedrich, Hahn, Stefan
• Primary Examiner(s): Zanelli, Michael J.
• Application Number:
  US10/822,513
• Publication Number:
  US 20040220724A1
• Time in Patent Office: 946 Days
• Field of Search: None
• US Class Current: 701/301
• CPC Class Codes: B60Q 9/005 using a video camera
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/26 to the rear of the vehicle
• CPC Class Codes: B60R 2300/30 characterised by the type of image processing
• CPC Class Codes: B60R 2300/305 merging camera image with lines or icons
• CPC Class Codes: B60R 2300/307 virtually distinguishing relevant parts of a scene from the background of the scene
• CPC Class Codes: B60R 2300/60 characterised by monitoring and displaying vehicle exterior scenes from a transformed perspective
• CPC Class Codes: B60R 2300/8086 for vehicle path indication
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: B62D 15/0295 by overlaying a vehicle path based on present steering angle over an image without processing that image

Abstract

In complex vehicle maneuvers it is often difficult for the driver of the motor vehicle to estimate along which tract the vehicle will move and how much free space is necessary in order to avoid collision. A process is provided for monitoring the environment space in the direction of travel of the own vehicle, in which image data of the environment around the vehicle in the direction of travel of the vehicle is recorded using a camera system the three-dimensional free space required for unimpeded travel is calculated in advance in a signal processing unit and the vehicle operator is informed regarding whether or not a sufficient free space is available to him for unimpeded travel.

Claims

1. A process for monitoring the free space in the direction of travel of a vehicle, comprising

acquiring image data of the vehicle environment in the direction of travel using a camera system,

additionally, on the basis of operating parameters and dimensions of the vehicle, pre-calculating a three dimensional free space required for unimpeded travel of the vehicle in a signal processing unit, and

displaying to the operator of the vehicle at least the part of the acquired image data corresponding to the pre-calculated required free space,

wherein the display of the image data occurs essentially as it would appear from the perspective of the operator of the vehicle directly viewing the vehicle environment in the direction of travel,

wherein at least the image data associated with the three dimensional free space is subjected to a further processing,

wherein as a result of this further processing the vehicle operator is informed with regard to whether or not sufficient free space is available for unimpeded travel,

wherein the informing of the vehicle operator with regard to the existence of a sufficient free space occurs by the pictorial display of at least a part of the further processed image data,

wherein the displayed image data has superimposed a symbolic representation of the edge boundaries of the required free space as determined by the pre-calculations in the signal processing unit, and

wherein the edge boundaries are visualized by the display of a grid, a sequence of columns or a checkerboard pattern.

2. The process according to claim 1, further comprising taking into consideration dynamic changes over time taking place in the image data during further processing.

3. The process according to claim 1, wherein the informing of the vehicle operator in the case that sufficient free space is not available for unimpeded travel occurs by warning means, wherein said warning means are optical, acoustic or haptic perceivable signal providers.

4. The process according to claim 1, wherein the informing of the vehicle operator in the case that sufficient free space is no longer available for an unimpeded travel occurs by pictorial display of at least a part of the further processed image data, wherein, in these image data, those objects which are located in an impeding manner in the free space required for the travel are represented enhanced or highlighted above the other image objects.

5. The process according to claim 4, wherein the enhancement occurs in such a manner that the objects which are located in an impeding manner in the free space required for the travel have symbols superimposed thereupon, which symbols represent the direction of travel and/or speed of movement of said objects.

6. The process according to claim 1, wherein the symbolic representation of the edge boundaries occurs by symbolic representation of walls.

7. The process according to claim 1, wherein the symbolic display of the edge boundaries occurs by the symbolic representation of walls and a ceiling area and/or floor area, forming a tube or tunnel.

8. The process according to claim 1, wherein the wall area and/or ceiling area and/or the floor area is so designed that it

has a geometric distinct and unambiguously recognizable structure,

permits a “

through view”

to the video scene lying there-behind, and

is so calculated, that it appears to be standing still even while the vehicle is moving in the longitudinal direction.

9. The process according to claim 1, further comprising recording the scene displayed to the operator of the vehicle and intermediately storing such for at least a specified elapsed time interval or for a particular traveled-through vehicle path, such that said stored information can be called up by the vehicle operator for repeated display.

10. The process according to claim 1, wherein the camera image data having superimposed thereon the edge boundaries of the required free space are supplied to a video processing process based on optical flow, which compares the movement of the spatially-fixed edge boundaries in the image data with the actual movement of objects in the image,

wherein those objects in the image, for which the model flow is less than the actual measured flow, are assessed as potential collision objects.

11. The process according to claim 10, wherein in the case of the presence of a potential collision object, this object is optically accentuated or highlighted, and/or results in a warning being given to the vehicle operator via a warning signal.

12. The process according to claim 11, wherein the object is optically accentuated or highlighted with color.

13. A device for monitoring the free space in the direction of travel of a vehicle, including:

a camera system for acquiring image data from the vehicle environment in the direction of travel of the vehicle,

a signal processing unit, which is in communication with sensors for detecting operating parameters of the vehicle and which includes a memory in which data is recorded regarding dimensions of the vehicle, the signal processing unit pre-calculating a three dimensional free space required for unimpeded travel of the vehicle on the basis of the operating parameters and the dimensions of the vehicle,

a display unit in communication with the signal processing unit, for displaying to the operator of the vehicle at least that part of the acquired image data corresponding to the pre-calculated required three-dimensional free space, wherein the displayed image data has superimposed a symbolic representation of the edge boundaries of the required free space as determined by the pre-calculations in the signal processing unit, and wherein the edge boundaries are visualized by the display of a grid, a sequence of columns or a checkerboard pattern,

wherein the memory associated with the signal processing unit is so designed and arranged, that a number of data sets are recorded in the memory regarding different vehicle dimensions, each of which can be specifically selected and made available to the signal processing unit by a means for specific selection of a set of data stored in memory, and/or the memory is provided with a means via which the at least one data set stored in memory can be changed.

14. The device according to claim 13, wherein the means for specific selection of a set of data stored in memory include a keypad operable by the vehicle operator.

15. The device according to claim 13, wherein the means for changing the at least one set of data stored in memory includes a communication device for the wireless transmission of data.

16. The device according to claim 13, wherein the camera system includes at least one catadioptric camera.

17. The device according to claim 13, wherein the device includes a memory for intermediate storage of at least a part of the image data displayed to the operator of the vehicle, and wherein means is provided via which the vehicle operator can call up this intermediate stored image data for repeated viewing.

18. The device according to claim 13, wherein the image data has edge boundaries of the required free space superimposed thereon and the device includes a unit for image processing on the basis of optical flow, via which the image data having the edge boundaries of the required free space superimposed thereon can be processed to the extent that movement of the positionally fixed edge boundaries in the image can be compared with the actual movement of objects in the image, so that those locations in the image, at which the optical flow is less than the actual measured flow, can be assessed as potential collision objects.

19. The device according to claim 18, wherein the unit for image processing is in communication with a display for the vehicle operator and/or a warning signal generator,

such that a potential collision object can be accentuated in the represented image or the vehicle operator can be warned of these objects.

20. A process for monitoring the space behind a vehicle during backing-up or repositioning of the vehicle, comprising:

acquiring image data of the vehicle environment in the direction of travel using a camera system,

additionally, on the basis of operating parameters and dimensions of the vehicle, pre-calculating a three dimensional free space required for unimpeded travel of the vehicle in a signal processing unit, and

displaying to the operator of the vehicle at least the part of the acquired image data corresponding to the pre-calculated required free space,

wherein the display of the image data occurs essentially as it would appear from the perspective of the operator of the vehicle directly viewing the vehicle environment in the direction of travel,

wherein at least the image data associated with the three dimensional free space is subjected to a further processing, and

wherein as a result of this further processing the vehicle operator is informed with regard to whether or not sufficient free space is available for unimpeded travels,

wherein the informing of the vehicle operator with regard to the existence of a sufficient free space occurs by the pictorial display of at least a part of the further processed image data,

wherein the displayed image data has superimposed a symbolic representation of the edge boundaries of the required free space as determined by the pre-calculations in the signal processing unit, and

wherein the edge boundaries are visualized by the display of a grid, a sequence of columns or a checkerboard pattern.

21. A device for monitoring the space behind a vehicle during backing-up or repositioning of the vehicle,

said device including:

a camera system for acquiring image data from the vehicle environment in the direction of travel of the vehicle,

a signal processing unit, which is in communication with sensors for detecting operating parameters of the vehicle and which includes a memory in which data is recorded regarding dimensions of the vehicle, the signal processing unit pre-calculating a three dimensional free space required for unimpeded travel of the vehicle on the basis of the operating parameters and the dimensions of the vehicle,

a display unit in communication with the signal processing unit, for displaying to the operator of the vehicle at least that part of the acquired image data corresponding to the pre-calculated required three-dimensional free space, wherein the displayed image data has superimposed a symbolic representation of the edge boundaries of the required free space as determined by the pre-calculations in the signal processing unit, and

wherein the edge boundaries are visualized by the display of a arid, a sequence of columns or a checkerboard pattern,

wherein the memory associated with the signal processing unit is so designed and arranged, that a number of data sets are recorded in the memory regarding different vehicle dimensions, each of which can be specifically selected and made available to the signal processing unit, and/or the memory is provided with a means via which the at least one data set stored in memory can be changed.