Hybrid world/body locked HUD on an HMD

• US 10,175,483 B2
• Filed: 06/18/2013
• Issued: 01/08/2019
• Est. Priority Date: 06/18/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Salter, Tom G., Sugden, Ben J., Deptford, Daniel, Crocco, Jr., Robert L., Keane, Brian E., Massey, Laura K., Kipman, Alex Aben-Athar, Kinnebrew, Peter Tobias, Kamuda, Nicholas Ferianc
• Primary Examiner(s): Wang, Jin-Cheng
• Application Number:
  US13/921,116
• Publication Number:
  US 20140368535A1
• Time in Patent Office: 2,030 Days
• Field of Search: 345632, 345633, 345634, 345419
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A system and method are disclosed for displaying virtual objects in a mixed reality environment in a way that is optimal and most comfortable for a user to interact with the virtual objects. When a user is not focused on the virtual object, which may be a heads-up display, or HUD, the HUD may remain body locked to the user. As such, the user may explore and interact with a mixed reality environment presented by the head mounted display device without interference from the HUD. When a user wishes to view and/or interact with the HUD, the user may look at the HUD. At this point, the HUD may change from a body locked virtual object to a world locked virtual object. The user is then able to view and interact with the HUD from different positions and perspectives of the HUD.

Claims

1. A system for presenting a mixed reality experience to one or more users, the system comprising:

a display device including a display unit for displaying a virtual object; and

a computing system operatively coupled to the display device, the computing system configured to;

generate the virtual object for display on the display device,

determine whether a field of view of the display device includes the virtual object for at least a determined period of time,

position, based on determining that the field of view does not include the virtual object for at least the determined period of time, the virtual object as being body locked with respect to the display device, and

position, based on determining that the field of view includes the virtual object for at least the determined period of time, the virtual object as being world locked.

2. The system of claim 1, wherein the computing system comprises at least one of a hub computing system or one or more processing units.

3. The system of claim 1, wherein the computing system is configured to switch the position of the virtual object from being body locked to being world locked in response to determining that the virtual object remains within the field of view of the display device for the determined period of time.

4. The system of claim 1, wherein the computing system is configured to switch the position of the virtual object from being world locked to being body locked in response to determining that the virtual object remains outside of the field of view of the display device for the determined period of time.

5. The system of claim 1, wherein the computing system is configured to switch the position of the virtual object from being world locked to being body locked in response to determining that the user is not looking in a direction of the virtual object.

6. The system of claim 1, wherein the virtual object is a heads-up display (HUD) including content.

7. The system of claim 1, wherein the computing system is configured to allow a user to select the virtual object when the virtual object is world locked, and move the virtual object to a new position in three dimensional space with a gesture.

8. The system of claim 7, wherein the virtual object is selected by the user performing a gesture with the user'"'"'s hands, body or eyes.

9. A system for presenting a mixed reality experience, the system comprising:

a display device including a first set of sensors for sensing data relating to a position of the display device and a display unit for displaying a virtual object to a user of the display device; and

a computing system operatively coupled to the display device, the computing system including a second set of sensors for sensing data relating to a position of the user, and the computing system configured to;

generate the virtual object for display on the display device,

determine whether a field of view of the display device includes the virtual object for at least a determined period of time,

maintain the virtual object in a body locked position outside of the field of view of the display device in response to determining that the field of view does not include the virtual object for at least the determined period of time, and

maintain the virtual object in world locked position within the field of view of the display device in response to determining that the field of view includes the virtual object for at least the determined period of time.

10. The system of claim 9, wherein a user may view the virtual object from different perspectives and positions when the virtual object is world locked.

11. The system of claim 9, wherein the virtual object is body locked by remaining a fixed distance away from the user'"'"'s head as the user moves around.

12. The system of claim 9, wherein the virtual object is body locked by remaining in front of a user'"'"'s head and above a horizontal plane in which the user'"'"'s head resides.

13. The system of claim 9, wherein the virtual object is body locked by pivoting with the user'"'"'s head as the user'"'"'s head turns from side to side.

14. The system of claim 9, wherein the virtual object is body locked by pivoting with the user'"'"'s head as the user'"'"'s head nods up and down.

15. The system of claim 9, wherein the virtual object is body locked by pivoting with the user'"'"'s head as the user'"'"'s head tilts from side to side.

16. The system of claim 9, wherein the virtual object is a first virtual object, the system further including a second virtual object, the computing system further configured to:

maintain the second virtual object in a body locked position outside of the field of view of the display device in response to determining that the field of view does not include the virtual object for at least the determined period of time, and

maintain the second virtual object in world locked position within the field of view of the display device in response to determining that the field of view includes the virtual object for at least the determined period of time.

17. A method of presenting a mixed reality experience to one or more users, the method comprising:

determining whether a field of view of a display device includes a virtual object for at least a determined period of time,

locking a position of the virtual object in a fixed position with respect to a portion of a user'"'"'s body in response to determining that the field of view does not include the virtual object for at least the determined period of time; and

locking a position of the virtual object in a fixed position in three-dimensional space in response to determining that the field of view includes the virtual object for at least the determined period of time.

18. The method of claim 17, wherein locking the position of the virtual object comprises locking the position of the virtual object with respect to the user'"'"'s head.

19. The method of claim 17, further comprising switching the virtual object from the body locked position to the world locked position upon determining that the field of view includes the virtual object for at least the determined period of time.

20. The method of claim 17, further comprising switching the virtual object from the world locked position to the body locked position upon determining that the field of view does not include the virtual object for at least the determined period of time.

21. A system for presenting a mixed reality experience to a user, the system comprising:

a display device including a display unit for displaying a virtual object; and

a processor operatively coupled to the display device, the processor configured to;

generate the virtual object for display on the display device,

determine whether a field of view of the display device includes the virtual object for at least a determined period of time,

operate, in response to determining that the field of view does not include the virtual object for at least a determined period of time, a first mode of operation where the virtual object is positioned outside of a central portion of a field of view of the display device, the virtual object moving with a body part of the user other than a head of the user so as to remain accessible to the user as the user moves, and such that the user can look toward and away from the virtual object in the first mode of operation,

operate, in response to determining that the field of view includes the virtual object for at least a determined period of time, in a second mode of operation where the virtual object is world locked and the user can move around a fixed position of the virtual object, and

switch from the first mode to the second mode by focusing on the virtual object in the first mode for a predetermined period of time.

22. The system of claim 21, wherein the the processor is further configured to switch the position of the virtual object from being world locked to being body locked in response to determining that the field of view does not include the virtual object for at least a determined period of time.

23. The system of claim 21, wherein the the processor is further configured to switch the position of the virtual object from being world locked to being body locked in response to determining that the user is not looking in a direction of the virtual object.

24. The system of claim 21, wherein the virtual object is a heads-up display (HUD) including content.

25. The system of claim 21, wherein the processor is further configured to allow a user to select the virtual object in response to the virtual object being world locked, and to move the virtual object to a new position in three dimensional space with a gesture.