Transitions between body-locked and world-locked augmented reality

• US 9,361,732 B2
• Filed: 05/01/2014
• Issued: 06/07/2016
• Est. Priority Date: 05/01/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Ebstyne, Michael John, Schaffalitzky, Frederik, Latta, Stephen, Lalonde, Paul Albert, Steedly, Drew, Kipman, Alex, Eade, Ethan
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): CHEN, FRANK S
• Application Number:
  US14/267,624
• Publication Number:
  US 20150317831A1
• Time in Patent Office: 768 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Various embodiments relating to controlling a see-through display are disclosed. In one embodiment, virtual objects may be displayed on the see-through display. The virtual objects transition between having a position that is body-locked and a position that is world-locked based on various transition events.

Claims

1. A method of operating a see-through display, the method comprising:

displaying via the see-through display one or more virtual objects with a world-locked position in which a position of each virtual object appears to be fixed relative to real-world objects viewable through the see-through display and the position of each virtual object appears to be moveable relative to a wearer of the see-through display; and

responsive to transitioning from tracking a location and an orientation of the see-through display in six degrees of freedom to tracking the orientation of the see-through display in three degrees of freedom due to the location and orientation of the head-mounted display device in six degrees of freedom not meeting a quality metric, displaying via the see-through display the one or more virtual objects with a body-locked position in which the position of each virtual object appears to be fixed relative to the wearer of the see-through display and the position of each virtual object appears to be moveable relative to the real-world objects.

2. The method of claim 1, further comprising:

visually transitioning the one or more virtual objects over a series of image frames from the world-locked position to the body-locked position via an animation.

3. The method of claim 2, wherein the one or more virtual objects are a plurality of virtual objects and the plurality of virtual objects visually transition to a predefined layout in the body-locked position.

4. The method of claim 1, further comprising:

visually transitioning the one or more virtual objects from a 3D view of the one or more virtual objects to a 2D view of the one or more virtual objects.

5. The method of claim 4, wherein the one or more virtual objects are displayed as volumetric holograms in the 3D view and the one or more virtual objects are displayed in a screen viewport of a virtual world in the 2D view.

6. The method of claim 1, wherein the transition event includes the see-through display moving from a visually recognized real-world location to a visually unrecognized real-world location.

7. The method of claim 6, wherein the one or more virtual objects are a plurality of virtual objects and the plurality of virtual objects is displayed according to a location-specific visual layout corresponding to the recognized real-world location.

8. A method of operating a see-through display, the method comprising:

displaying via the see-through display one or more virtual objects with a body-locked position in which a position of each virtual object appears to be fixed relative to a wearer of the see-through display and the position of each virtual object appears to be moveable relative to real-world objects viewable through the see-through display; and

responsive to transitioning from tracking a location and an orientation of the see-through display in six degrees of freedom to tracking the orientation of the see-through display in three degrees of freedom due to the location and orientation of the head-mounted display device in six degrees of freedom not meeting a quality metric, displaying via the see-through display the one or more virtual objects with a world-locked position in which the position of each virtual object appears to be fixed relative to the real-world objects and the position of each virtual object appears to be moveable relative to the wearer of the see-through display.

9. The method of claim 8, further comprising:

visually transitioning the one or more virtual objects over a series of image frames from the body-locked position to the world-locked position via an animation.

10. The method of claim 9, wherein the one or more virtual objects are a plurality of virtual objects and the plurality of virtual objects visually transition to a predefined layout in the body-locked position.

11. The method of claim 8, further comprising:

visually transitioning the one or more virtual objects from a 2D view of the one or more virtual objects to a 3D view of the one or more virtual objects.

12. The method of claim 11, wherein the one or more virtual objects are displayed in a screen viewport of a virtual world in the 2D view and the one or more virtual objects are displayed as volumetric holograms in the 3D view.

13. The method of claim 8, wherein the transition event includes the see-through display moving from a visually unrecognized real-world location to a visually recognized real-world location.

14. The method of claim 13, wherein the one or more virtual objects are a plurality of virtual objects and the plurality of virtual objects is displayed according to a location-specific visual layout corresponding to the recognized real-world location.

15. A head-mounted display device comprising:

an inertial measurement unit (IMU) configured to determine an orientation of the head-mounted display device in three degrees of freedom;

an optical sensor configured to capture image data of a real-world environment and cooperate with the IMU to determine a location and the orientation of the head-mounted display device in six degrees of freedom;

a see-through display through which a real-world environment is viewable by a wearer, the see-through display being configured to display virtual objects to the wearer such that virtual objects appear to be integrated with the real-world environment, the see-through display being configured to;

responsive to a transition from tracking the orientation of the head-mounted display device in three degrees of freedom to tracking the location and the orientation of the see-through display in six degrees of freedom, display via the see-through display one or more virtual objects with a world-locked position in which a position of each virtual object appears to be fixed relative to real-world objects viewable through the see-through display and the position of each virtual object appears to be moveable relative to a wearer of the see-through display, the world-locked position being derived from the location and orientation of the head-mounted display device in six degrees of freedom; and

responsive to a transition from tracking the location and the orientation of the head-mounted display device in six degrees of freedom to tracking the orientation of the see-through display in three degrees of freedom, displaying via the see-through display the one or more virtual objects with a body-locked position in which the position of each virtual object appears to be fixed relative to the wearer of the see-through display and the position of each virtual object appears to be moveable relative to the real-world objects, the body-locked position being derived from the orientation of the head-mounted display device in three degrees of freedom.

16. The head-mounted display device of claim 15, wherein the transition from tracking the orientation of the head-mounted display device in three degrees of freedom to tracking the location and the orientation of the see-through display in six degrees of freedom is performed responsive to the location and the orientation of the head-mounted display device in six degrees of freedom meeting the quality metric, and the transition from tracking the location and the orientation of the head-mounted display device in six degrees of freedom to tracking the orientation of the see-through display in three degrees of freedom is performed responsive to the location and orientation of the head-mounted display device in six degrees of freedom not meeting the quality metric.