Method for Representing Virtual Information in a Real Environment

• US 20150332505A1
• Filed: 12/21/2012
• Published: 11/19/2015
• Est. Priority Date: 12/21/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Metaio GmbH (Apple Inc.)
• Inventors: Wang, Lejing, Meier, Peter, Misslinger, Stefan
• Granted Patent: US 9,558,581 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 15/06 Ray-tracing
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A method for representing virtual information in a view of a real environment is provided that includes the following steps: providing a system setup comprising at least one display device, determining a position of a viewing point relative to at least one component of the real environment, providing a geometry model of the real environment, providing at least one item of virtual information and its position, determining a visualization mode of blending in the at least one item of virtual information on the display device according to the position of the viewing point and the geometry model, calculating a ray between the viewing point and the item of virtual information, and determining a number of boundary intersections by the ray, wherein if the number of boundary intersections is less than 2, the item of virtual information is blended in a non-occlusion mode, otherwise in an occlusion mode.

Claims

1. A method for representing virtual information in a view of a real environment, comprising the following steps:

providing a system setup comprising at least one display device, wherein the system setup is adapted for blending in virtual information on the display device;

determining a position of a viewing point relative to at least one component of the real environment;

providing a geometry model of the real environment, the geometry model comprising multiple representations of respective built-in real objects of the real environment, each representation comprising at least one 2D or 3D geometrical shape having at least one boundary delimiting the respective object, wherein one side of the boundary is delimiting the object toward its inner side and the other side of the boundary is delimiting the object toward its outer side;

providing at least one item of virtual information and a position of the at least one item of virtual information;

determining a visualization mode of blending in the at least one item of virtual information on the display device according to the position of the viewing point and the geometry model;

calculating a ray between the viewing point and the item of virtual information; and

determining a number of boundary intersections by the ray;

wherein if the number of boundary intersections is less than 2, the item of virtual information is blended in in a non-occlusion mode, otherwise in an occlusion mode, and/or the position of the item of virtual information is moved to the front of the intersected boundary with respect to the viewing point.

2. The method according to claim 1, wherein the method is performed iteratively, where in each iteration a height of the item of virtual information is increased and the iteration is stopped, when the number of boundary intersections by the ray becomes less than 2 or when the height exceeds a predefined limit.

3. The method according to claim 1, wherein the determination of the visualization mode of items of virtual information is influenced by the distance of the virtual information to the viewing point and in case the item of virtual information is determined to be blended in in a non-occlusion mode, a graphical representation of the item of virtual information is added to the geometry model and is considered when determining the visualization mode of a following item of virtual information.

4. The method according to claim 1, wherein the geometry model is derived from a depth sensor source associated with the display device.

5. The method according to claim 1, wherein the geometry model is a 2D model of a real environment which describes a 2D geometry of the real environment with respect to a particular plane.

6. The method according to claim 1, wherein the geometry model describes at least one of the following attributes of the real environment:

shape, symmetry, geometrical size, structure.

7. The method according to claim 1, wherein the geometry model is representative of a city map, a city blueprint, a floor plan of a building, or a blueprint of a shopping mall.

8. The method according to claim 1, wherein the geometry model is a 3D model of a real environment which describes a 3D geometry of the real environment.

9. The method according to claim 8, wherein the 3D model of the real environment is obtained from a 2D model and a dimension orthogonal to a ground plane of the 2D model.

10. The method according to claim 1, wherein in the geometry model the built-in real objects of the real environment refer to real objects represented by at least one of 2D and 3D geometrical shapes having at least one closed boundary.

11. The method according to claim 1, wherein items of virtual information related to occluded real objects are shown semi-transparent, or in dash lines, and items of virtual information related to at least partially visible real objects are shown opaquely, in solid lines, or in a different color.

12. The method according to claim 1, wherein the system setup is adapted for providing at least one view of the real environment by means of taking a picture and blending in virtual information on the display device such that the virtual information is superimposed with the real environment in at least part of the view.

13. The method according to claim 1, wherein the system setup is adapted for providing at least one rendered view of the real environment and blending in virtual information on the display device such that the virtual information is superimposed with the real environment in at least part of the view.

14. The method according to claim 1, wherein the system setup is adapted for providing at least one view of the real environment and blending in virtual information on a semi-transparent display device such that the virtual information is superimposed with the real environment in at least part of the view.

15. A non-transitory computer readable medium comprising software code sections adapted to perform a method for representing virtual information in a view of a real environment, which method comprises the following steps:

providing a system setup comprising at least one display device, wherein the system setup is adapted for blending in virtual information on the display device;

determining a position of a viewing point relative to at least one component of the real environment;

providing a geometry model of the real environment, the geometry model comprising multiple representations of respective built-in real objects of the real environment each representation comprising at least one 2D or 3D geometrical shape having at least one boundary delimiting the respective object, wherein one side of the boundary is delimiting the object toward its inner side and the other side of the boundary is delimiting the object toward its outer side;

providing at least one item of virtual information and a position of the at least one item of virtual information;

determining a visualization mode of blending in the at least one item of virtual information on the display device according to the position of the viewing point and the geometry model;

calculating a ray between the viewing point and the item of virtual information; and

determining a number of boundary intersections by the ray;

wherein if the number of boundary intersections is less than 2, the item of virtual information is blended in in a non-occlusion mode, otherwise in an occlusion mode, and/or the position of the item of virtual information is moved to the front of the intersected boundary with respect to the viewing point.