Video image presentation system

• US 9,013,578 B2
• Filed: 11/16/2007
• Issued: 04/21/2015
• Est. Priority Date: 11/21/2006
• Status: Active Grant

• Current Assignee: Harman Becker Automotive Systems GmbH (Samsung Electronics Co. Ltd.)
• Original Assignee: Harman Becker Automotive Systems GmbH (Samsung Electronics Co. Ltd.)
• Inventors: Mohr, Ulrich, Stiegler, Andreas
• Primary Examiner(s): Ashraf, Waseem
• Assistant Examiner(s): MURPHY, CHARLES C
• Application Number:
  US11/941,643
• Publication Number:
  US 20080239078A1
• Time in Patent Office: 2,713 Days
• Field of Search: 701/49, 701/45, 701/301, 701/93, 701/36, 701/33, 701/96, 701/532, 701/29.7, 382/104, 382/100, 382/103, 250/221, 250/208.1, 348/148, 348/135, 348/143, 340/425.5, 702/143
• US Class Current: 348/148
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30248 Vehicle exterior or interior
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

A system for presenting video images from a vehicle environment includes taking video images of the vehicle environment, detecting sensor data of the vehicle environment or the vehicle, processing the sensor data, recognizing objects in the video images, and visualizing the processed sensor data in the displayed video images by changing the representation of recognized objects in the displayed video images.

Claims

1. A method for presenting video images from a vehicle environment outside of a detecting vehicle, the method comprising:

taking video images of the vehicle environment

detecting sensor data of the vehicle environment or the detecting vehicle itself;

processing the sensor data;

determining the distance of an object shown in the video images relative to the detecting vehicle from where the distance is determined;

recognizing objects in the video images; and

visualizing the processed sensor data in the displayed video images by changing the representation of recognized objects in the video images,

when it is detected that the distance of the object relative to the detecting vehicle is decreasing, changing the representation of the object in the displayed video image to visually alert a driver of the detecting vehicle of the object,

where the representation of the recognized objects is changed by introducing an element in an object in the displayed video images, and where the element is not in the video images of the environment of the vehicle, and where the element is an element that could be present in another driving situation, where the element is a representation of a component of a detected object in the environment, and where the alert visually changes or modifies the element that is not present visibly or invisibly in the vehicle environment as viewed by a driver of the detecting vehicle.

2. The method of claim 1, where objects in the video images are detected and interpreted by post processing the video images and are classified in object classes.

3. The method of claim 1, where the representation of the recognized objects is changed by introducing an element in an object in the displayed video images, and where the element is actually not present in the vehicle environment as viewed by a driver of the detecting vehicle through a windscreen of the vehicle.

4. The method of claim 1, where the video images are taken from the vehicle environment in front of the detecting vehicle.

5. The method of claim 1, where the video images are taken from the vehicle environment in back of the detecting vehicle.

6. The method of claim 1, where the step of recognizing objects includes the step of recognizing vehicles in the video images.

7. The method of claim 1, where the step of detecting sensor data includes the step of determining the distance of an object shown in the video images relative to the detecting vehicle from where the distance is determined.

8. The method of claim 7 where, when it is detected that the distance of the object relative to the detecting vehicle is decreasing, changing the representation of the object in the displayed video image to visually alert a driver of the detecting vehicle of the object.

9. The method of claim 1, where the step of detecting sensor data includes the step of determining the velocity of a vehicle in front of the detecting vehicle.

10. The method of claim 9, where, when it is detected that the velocity of the vehicle in front of the detecting vehicle is lower than a predetermined threshold velocity, changing the representation of the front vehicle by emphasizing visualization of braking lights of the front vehicle in the displayed video images.

11. The method of claim 9, where, when it is detected that the velocity of the vehicle in front of the detecting vehicle is lower than a predetermined threshold velocity, changing the representation of the front vehicle by adding flashing warning lights to the front vehicle in the displayed video images.

12. The method of claim 1, where the step of detecting sensor data includes the step of determining the outside visibility, and when the visibility is lower than a predetermined threshold visibility and when the distance to the vehicle in front is greater than a predetermined distance, the representation of the vehicle in front of the detecting vehicle is changed.

13. The method of claim 1, where the step for detecting sensor data of the vehicle includes the step of determining the position of the vehicle and to locate the vehicle in a digital map and to emphasize objects in the video images depending on information from the map data.

14. The method of claim 13, where object information is retrieved from the map data, and the objects in the video images are emphasized depending on the received sensor data.

15. A system for displaying video images from a vehicle environment, the system comprising:

an image sensor taking video images from the environment of the vehicle;

at least one sensor detecting data of the vehicle environment or the vehicle itself;

a sensor processing unit processing the sensor data;

an image processing unit post-processing the video images and recognizing objects in the video images; and

a graphics processing unit visualizing the processed sensor data in the displayed video images by changing the representation of recognized objects in the video images, when it is detected that the distance of the object relative to the detecting vehicle is decreasing, changing the representation of the object in the displayed video image to visually alert a driver of the detecting vehicle of the object,

where the representation of the recognized objects is changed by introducing an element in an object in the displayed video images, and where the element is not in the video images of the environment of the vehicle, and where the element is an element that could be present in another driving situation, where the element is a representation of a component of a detected object in the environment, and where the alert visually changes or modifies the element that is not present visibly or invisibly in the vehicle environment as viewed by a driver of the detecting vehicle.

16. The system of claim 15, where the sensor detecting data of the vehicle environment is selected from the group consisting of a distance sensor, a sensor detecting the velocity of vehicles driving in front of the vehicle, and a position detecting sensor.

17. The system of claim 15, where the image processing unit recognizes objects in the video images by applying edge detection filters to the video images and comparing the detected objects to predetermined types of objects stored in a database.

18. A method for presenting video images from a vehicle environment outside of a detecting vehicle, the method comprising:

taking video images of the vehicle environment;

recognizing a vehicle in front of the detecting vehicle in the video images;

determining the velocity of the recognized vehicle;

processing the determined velocity; and

visualizing the processed velocity in the displayed video images by changing the representation of the recognized vehicle in the video images to visually alert the driver of the detecting vehicle, where, when it is detected that the velocity of the recognized vehicle is lower than a predetermined threshold velocity, braking lights of the recognized vehicle as shown in the video images are displayed in an activated state although not actually physically in the active state, where the activated brake lights could be present in another driving situation.

19. A method for presenting video images from a vehicle environment outside of a detecting vehicle, the method comprising:

taking video images of the vehicle environment;

recognizing a vehicle in front of the detecting vehicle in the video images;

determining the velocity of the recognized vehicle relative to the detecting vehicle;

processing the determined velocity; and

visualizing the processed velocity in the displayed video images by changing the representation of the recognized vehicle in the displayed video images to visually alert the driver of the detecting vehicle, where the representation of the recognized vehicle is changed in the video images by adding flashing warning lights to the recognized vehicle as shown in the video images, when it is detected that the velocity is lower than a predetermined threshold velocity although the flashing warning lights are not physically in an active state in the recognized vehicle, where the flashing warning lights could be present in another driving situation.

20. A system for displaying video images from a vehicle environment outside of a detecting vehicle, the system comprising:

an image sensor taking video images from the vehicle environment;

at least one velocity sensor detecting the velocity of other vehicles in the vehicle environment;

a velocity sensor processing unit processing the velocity of the other vehicles relative to the detecting vehicle;

an image processing unit post-processing the video images and recognizing the other vehicles in the video images; and

a graphics processing unit visualizing the processed sensor data in the video images to visually alert the driver of the detecting vehicle, where, when the velocity sensor detects that the velocity of a vehicle moving in front of the detecting vehicle is lower than a predetermined threshold velocity, the graphics processor displays braking lights of the front vehicle in an activated state although not actually physically in the active state, where the activated brake lights could be present in another driving situation.

21. A system for displaying video images from a vehicle environment outside of a detecting vehicle, the system comprising:

an image sensor taking video images from the vehicle environment;

at least one velocity sensor detecting the velocity of other vehicles in the vehicle environment;

a velocity sensor processing unit processing the velocity of the other vehicles relative to the detecting vehicle;

an image processing unit post-processing the video images and recognizing the other vehicles in the video images; and

a graphics processing unit visualizing the processed sensor data in the video images to visually alert the driver of the detecting vehicle, where, when the velocity sensor detects that the velocity of a vehicle moving in front of the detecting vehicle is lower than a predetermined threshold velocity, the graphics processor adds flashing warning lights to the front vehicle in the displayed video images although the flashing warning lights are not physically in an active state in the recognized vehicle, where the flashing warning lights could be present in another driving situation.

22. A method for presenting video images from a vehicle environment outside of a detecting vehicle to a driver of the detecting vehicle, the method comprising:

taking a video image of the vehicle environment in front of the detecting vehicle;

recognizing a plurality of objects in the video image;

presenting the video image including the recognized objects on a display of the detecting vehicle;

detecting sensor data of the vehicle environment or the detecting vehicle itself by operating a sensor of the detecting vehicle;

based on the detected sensor data, determining whether one or more of the detected sensor data is to be selected for communication to the driver via the presented video image;

if it is determined that one or more of the detected sensor data is to be selected for communication to the driver to visually alert the driver of the detecting vehicle, changing the representation of one or more of the recognized objects in the video image to visualize the selected sensor data although the representation is physically different than the recognized object, where the representation of one or more of the recognized objects could be present in another driving situation.

23. The method of claim 22, where detecting sensor data includes detecting data selected from the group consisting of the velocity of the detecting vehicle, the position of the detecting vehicle, the velocity of another vehicle, the position of another vehicle or of one or more of the recognized objects, the distance from the detecting vehicle to another vehicle or to one or more of the recognized objects, a temperature of the vehicle environment, a moisture level of the vehicle environment, a visibility of the vehicle environment, and combinations of two or more of the foregoing.

24. The method of claim 22, where detecting sensor data includes operating a navigation unit of the detecting vehicle.

25. The method of claim 22, where detecting sensor data includes accessing digital map data.

26. The method of claim 22, where detecting sensor data includes determining the position of the detecting vehicle, and further including comparing the position of the detecting vehicle relative to one or more of the recognized objects, and where changing the representation of the one or more recognized objects is based on the relative position.

27. The method of claim 22, where changing the representation includes rendering the visualization of the one or more recognized objects such that the one or more recognized objects are presented in the video image more prominently than other objects presented in the video image.

28. The method of claim 22, where changing the representation includes adding an additional object to the displayed video image, and where the additional object is not actually present in the vehicle environment or not actually visible by the driver while viewing the vehicle environment from the detecting vehicle.

29. The method of claim 22, where changing the representation includes adding an element to one or more of the recognized objects presented in the video and where the element is not actually present in the vehicle environment or not actually visible by the driver while viewing the vehicle environment from the detecting vehicle.

30. The method of claim 22, where determining whether one or more of the detected sensor data is to be selected includes determining whether one or more of the detected sensor data falls outside of a predetermined value or range of values.