Method of displaying virtual information in a view of a real environment

• US 9,170,766 B2
• Filed: 02/28/2011
• Issued: 10/27/2015
• Est. Priority Date: 03/01/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Metaio GmbH (Apple Inc.)
• Inventors: Meier, Peter
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): CHEN, FRANK S
• Application Number:
  US13/582,314
• Publication Number:
  US 20130057581A1
• Time in Patent Office: 1,702 Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A method of displaying virtual information in a view of a real environment comprising the following steps: providing a system for displaying of virtual information in a view of a real environment, determining a current pose of at least one part of the system relative to at least one part of the real environment and providing accuracy information of the current pose, providing multiple pieces of virtual information, and assigning a respective one of the pieces of virtual information to one of different parameters indicative of different pose accuracy information, and displaying at least one of the pieces of virtual information in the view of the real environment according to the accuracy information of the current pose in relation to the assigned parameter of the at least one of the pieces of virtual information.

Claims

1. A method of displaying virtual information in a view of a real environment comprising the following steps:

providing a system for displaying of virtual information in a view of a real environment;

determining a current pose of at least one part of the system relative to at least one part of the real environment and providing accuracy information of the current pose;

providing multiple pieces of virtual information related to at least part of the real environment contained in the view of the real environment, and assigning one of different parameters indicative of pose accuracy information to each of the multiple pieces of virtual information; and

displaying at least one of the pieces of virtual information in the view of the real environment according to the accuracy information of the current pose in relation to the assigned parameter of the at least one of the pieces of virtual information;

wherein at least one of the multiple pieces of virtual information is not displayed when the accuracy information of the current pose is equal to or beyond a certain threshold; and

wherein a first number of the multiple pieces of virtual information are displayed when the accuracy information of the current pose is within a first range, and a second number of the multiple pieces of virtual information are displayed when the accuracy information of the current pose is within a second range different from the first range.

2. The method of claim 1, wherein the accuracy information contains information on an uncertainty.

3. The method of claim 1, further comprising the step of:

displaying at least one of the pieces of virtual information in the view of the real environment according to a combination of the accuracy information of the current pose and a distance of at least one part of the system to the position of the at least one of the pieces of virtual information in relation to the real environment.

4. The method of claim 1, further comprising the step of:

providing an information system which is capable of providing, in a first mode, an augmented reality view and, in a second mode, a map-view,

wherein the step of displaying at least one of the pieces of virtual information according to the accuracy information of the current pose is performed in the augmented reality view, but not in the map view.

5. The method of claim 1, wherein the accuracy information and/or a change in accuracy information of the current pose is determined by a tracking system'"'"'s own accuracy mechanism, by a time elapsed since last successful pose estimation of a pose-estimation mechanism, or by a position or a change in position of the system.

6. The method of claim 1, wherein the accuracy information is determined for each component of the pose individually or for orientation and position individually, and assigning a respective one of the pieces of virtual information to one of different parameters indicative of different pose accuracy information.

7. A method of displaying virtual information in a view of a real environment comprising the following steps:

providing a system for displaying of virtual information in a view of a real environment;

determining a current pose of at least one part of the system relative to at least one part of the real environment and providing accuracy information of the current pose;

providing multiple pieces of virtual information, and assigning a respective one of the pieces of virtual information to one of different parameters indicative of different pose accuracy information;

displaying at least one of the pieces of virtual information in the view of the real environment according to the accuracy information of the current pose in relation to the assigned parameter of the at least one of the pieces of virtual information;

clustering a number of the pieces of virtual information;

creating a superordinate piece of virtual information from the clustered number of pieces of virtual information; and

assigning the superordinate piece of virtual information to a parameter indicative of pose accuracy information which is different from the parameters assigned to the clustered number of pieces of virtual information.

8. A method of displaying virtual information in a view of a real environment comprising the following steps:

providing a system for displaying of virtual information in a view of a real environment;

determining a current pose of at least one part of the system relative to at least one part of the real environment and providing accuracy information of the current pose;

providing multiple pieces of virtual information, and assigning a respective one of the pieces of virtual information to one of different parameters indicative of different pose accuracy information;

displaying at least one of the pieces of virtual information in the view of the real environment according to the accuracy information of the current pose in relation to the assigned parameter of the at least one of the pieces of virtual information;

assigning a respective one of the pieces of virtual information to a respective second parameter indicative of a distance;

determining a current distance of at least one part of the system to at least one part of the real environment; and

displaying at least one of the pieces of virtual information in the view of the real environment according to the current distance in relation to the assigned second parameter of the at least one of the pieces of virtual information.

9. A method of displaying virtual information in a view of a real environment comprising the following steps:

providing a system for displaying of virtual information in a view of a real environment;

determining a current pose of at least one part of the system relative to at least one part of the real environment and providing accuracy information of the current pose;

providing multiple pieces of virtual information, and assigning a respective one of the pieces of virtual information to one of different parameters indicative of different pose accuracy information; and

displaying at least one of the pieces of virtual information in the view of the real environment according to the accuracy information of the current pose in relation to the assigned parameter of the at least one of the pieces of virtual information,

wherein providing accuracy information of the current pose includes differentiating the accuracy information of the current pose depending on whether an optical or a non-optical pose determination method is used.

10. A method of displaying virtual information in a view of a real environment comprising the following steps:

providing a system for displaying of virtual information in a view of a real environment;

determining a current pose of at least one part of the system relative to at least one part of the real environment and providing accuracy information of the current pose;

providing multiple pieces of virtual information, and assigning a respective one of the pieces of virtual information to one of different parameters indicative of different pose accuracy information;

displaying at least one of the pieces of virtual information in the view of the real environment according to the accuracy information of the current pose in relation to the assigned parameter of the at least one of the pieces of virtual information; and

displaying at least one of the pieces of virtual information in the view of the real environment using different appearances or visualization methods depending on the accuracy information of the current pose.

11. A method of displaying virtual information in a view of a real environment comprising the following steps:

providing a system for displaying of virtual information in a view of a real environment;

determining a current pose of at least one part of the system relative to at least one part of the real environment and providing accuracy information of the current pose;

providing multiple pieces of virtual information, and assigning a respective one of the pieces of virtual information to one of different parameters indicative of different pose accuracy information;

displaying at least one of the pieces of virtual information in the view of the real environment according to the accuracy information of the current pose in relation to the assigned parameter of the at least one of the pieces of virtual information; and

displaying at least one of the pieces of virtual information in the view of the real environment perspectively correctly overlayed on the real environment in case of a higher accuracy of the current pose and non-correctly overlayed in case of a lower accuracy of the current pose.

12. A method of displaying virtual information in a view of a real environment comprising the following steps:

providing a system for displaying of virtual information in a view of a real environment;

determining a current pose of at least one part of the system relative to at least one part of the real environment and providing accuracy information of the current pose;

providing multiple pieces of virtual information, and assigning a respective one of the pieces of virtual information to one of different parameters indicative of different pose accuracy information;

displaying at least one of the pieces of virtual information in the view of the real environment according to the accuracy information of the current pose in relation to the assigned parameter of the at least one of the pieces of virtual information; and

determining the visibility of drawing style of a piece of virtual information not only according to the accuracy information of the current pose, but also according to a distance and/or information about occluding objects in the real environment and/or a current speed of a display of the system with respect to a coordinate system which is fixed with respect to the real environment.