Method And System For Generating Augmented Reality With A Display Of A Motor Vehicle

• US 20130162639A1
• Filed: 12/20/2012
• Published: 06/27/2013
• Est. Priority Date: 12/21/2011
• Status: Active Grant

• Current Assignee: Harman Becker Automotive Systems GmbH (Samsung Electronics Co. Ltd.)
• Original Assignee: Harman Becker Automotive Systems GmbH (Samsung Electronics Co. Ltd.)
• Inventors: Muench, Tobias, Schmauderer, Philipp, Benz, Christoph, Koerner, Andreas
• Granted Patent: US 9,517,415 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/525 Changing parameters of virtual cameras
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/803 Driving vehicles or craft, e.g. cars, airplanes, ships, robots or tanks
• CPC Class Codes: A63F 2300/205 for detecting the geographical location of the game platform
• CPC Class Codes: A63F 2300/6661 for changing the position of the virtual camera
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A method and system for generating augmented reality with a display of a vehicle is provided. The method comprises recording an image of an environment of the vehicle in the form of image data, determining a virtual space in three dimensions from the image data, and detecting a real object in the image data. The method further comprises determining a first coordinate range of the real object in the three dimensional virtual space, adding a virtual element to the three dimensional virtual space, and controlling the virtual element in the three dimensional virtual space based on a user input. The method further comprises outputting the environment and the controlled virtual element in combined form in an output image, modifying the output image when the first coordinate range of the real object and a second coordinate range of the controlled virtual element form an intersection area, and displaying the output image.

Claims

1. A method for generating an augmented reality with a display of a motor vehicle, the method comprising the steps of:

recording an image of an environment of the motor vehicle in the form of image data;

determining a virtual space in three dimensions from the image data;

detecting a real object in the image data;

determining a first coordinate range of the real object in the three dimensional virtual space;

adding a virtual element to the three dimensional virtual space, the virtual element having a second coordinate range;

controlling the virtual element in the three dimensional virtual space based on a user input;

outputting the environment and the controlled virtual element in combined form in an output image;

modifying the output image when the first coordinate range of the real object and the second coordinate range of the controlled virtual element form an intersection area; and

outputting the output image by means of the display of the motor vehicle.

2. The method according to claim 1, wherein the virtual element is controlled in three dimensions in the three dimensional virtual space.

3. The method according to claim 1, wherein the three dimensional virtual space is determined in three dimensions on the basis of a geometry of a traffic route.

4. The method according to claim 1, wherein the output image is modified by supplementing or replacing object image data of the real object with a virtual object.

5. The method according to claim 1, wherein the output image is modified by generating a virtual object from the object image data of the real object and outputting the virtual object in the output image.

6. The method according to claim 1, wherein a viewer position within the three dimensional virtual space is controlled by adapting the image data and the controlled virtual element to the viewer position.

7. An augmented reality system of a motor vehicle, the system comprising:

an image capture device which is configured to record an image of an environment of the motor vehicle in the form of image data;

an arithmetic unit which is connected to the image capture device;

a control unit which is connected to the arithmetic unit; and

a display which is connected to the arithmetic unit displaying an output image;

wherein the arithmetic unit is configured to

determine a three dimensional virtual space from the image data;

detect a real object in the image data;

determine a first coordinate range of the real object in the three dimensional virtual space;

add a virtual element having a second coordinate range to the three dimensional virtual space;

control the second coordinate range of the virtual element in the three dimensional virtual space by means of the control unit based on a user input;

combine the environment and the controlled virtual element in the output image; and

modify the output image when the first coordinate range of the real object and the second coordinate range of the controlled virtual element form an intersection area.

8. The system according to claim 7, wherein the virtual element is controlled in three dimensions in the three dimensional virtual space.

9. The system according to claim 7, wherein the three dimensional virtual space is determined in three dimensions on the basis of a geometry of a traffic route.

10. The system according to claim 7, wherein the output image is modified by one of supplementing and replacing object image data of the real object with a virtual object.

11. The system according to claim 7, wherein the output image is modified by generating a virtual object from the object image data of the real object and outputting the virtual object in the output image.

12. The system according to claim 7, wherein a viewer position within the three dimensional virtual space is controlled by adapting the image data and the controlled virtual element to the viewer position.

13. The system according to claim 7, wherein the display comprises a projector for projecting the output image onto a window pane.

14. An augmented reality system in a vehicle, the system comprising:

an image capture device configured to record an image of an environment about the vehicle to provide image data;

a display for providing an output image; and

an arithmetic unit operably coupled to the image capture device and to the display, the arithmetic unit being configured to;

determine a three dimensional virtual space from the image data;

detect a real object in the image data;

determine a first coordinate range of the real object in the three dimensional virtual space;

add a virtual element having a second coordinate range to the three dimensional virtual space;

control the second coordinate range of the virtual element in the three dimensional virtual space based on a user input;

combine the environment and the controlled virtual element in the output image; and

modify the output image when the first coordinate range of the real object and the controlled second coordinate range of the virtual element form an intersection area.

15. The system of claim 14, wherein the virtual element is controlled in three dimensions in the three dimensional virtual space.

16. The system of claim 14, wherein the three dimensional virtual space is determined in three dimensions on the basis of a geometry of a traffic route.

17. The system of claim 14, wherein the output image is modified by one of supplementing and replacing object image data of the real object with a virtual object.

18. The system of claim 14, wherein the output image is modified by generating a virtual object from the object image data of the real object and outputting the virtual object in the output image.

19. The system of claim 14, wherein a viewer position within the three dimensional virtual space is controlled by adapting the image data and the controlled virtual element to the viewer position.

20. The system of claim 14, wherein the display comprises a projector for projecting the output image onto a window pane.