Portable holographic user interface for an interactive 3D environment

• US 9,952,656 B2
• Filed: 08/21/2015
• Issued: 04/24/2018
• Est. Priority Date: 08/21/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Poulos, Adam Gabriel, Brown, Cameron Graeme, Krauss, Aaron Daniel, Ghaly, Marcus, Thomas, Michael, Paulovich, Jonathan, McCulloch, Daniel Joseph
• Primary Examiner(s): SALVUCCI, MATTHEW D
• Application Number:
  US14/832,961
• Publication Number:
  US 20170052507A1
• Time in Patent Office: 977 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0174 holographic
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G03H 1/2249 Holobject properties
• CPC Class Codes: G03H 2001/0061 in haptic applications when the observer interacts with the holobject
• CPC Class Codes: G03H 2001/2284 Superimposing the holobject with other visual information
• CPC Class Codes: G06F 3/00 Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Disclosed are a method and corresponding apparatus to enable a user of a display system to manipulate holographic objects. Multiple holographic user interface objects capable of being independently manipulated by a user are displayed to the user, overlaid on a real-world view of a 3D physical space in which the user is located. In response to a first user action, the holographic user interface objects are made to appear to be combined into a holographic container object that appears at a first location in the 3D physical space. In response to the first user action or a second user action, the holographic container object is made to appear to relocate to a second location in the 3D physical space. The holographic user interface objects are then made to appear to deploy from the holographic container object when the holographic container object appears to be located at the second location.

Claims

1. A method comprising:

displaying to a user, by a head-mounted display (HMD) device, a plurality of holographic user interface objects overlaid on a real-world view of a three-dimensional (3D) physical space in which the user is located, wherein each of the holographic user interface objects is a different tool usable by the user to affect a holographic target object, each of the plurality of holographic user interface objects being capable of being independently moved and used by the user;

in response to a first user action, causing, by the HMD device, the plurality of holographic user interface objects to appear to the user to be physically placed into a 3D holographic container object that appears at a first location in the 3D physical space;

in response to the first user action or a second user action, causing, by the HMD device, the 3D holographic container object to appear to relocate to a second location in the 3D physical space; and

causing, by the HMD device, the plurality of holographic user interface objects to appear to the user to deploy from within the 3D holographic container object when the 3D holographic container object appears to be located at the second location in the 3D physical space.

2. The method of claim 1, wherein the 3D holographic container object is moveable in three dimensions in the 3D physical space.

3. The method of claim 1, wherein at least one of the first user action or the second user action comprises a gesture or a spoken command.

4. The method of claim 1, wherein causing the 3D holographic object to appear to relocate to the second location in the 3D physical space is performed automatically in response to the user changing locations within the 3D physical space.

5. The method of claim 1, further comprising:

maintaining the plurality of holographic user interface objects in a constant orientation relative to the user while the user changes location or orientation within the 3D physical space.

6. The method of claim 1, further comprising:

maintaining the plurality of holographic user interface objects at a constant distance or within a constant range of distances from the user while the user changes location within the 3D physical space.

7. The method of claim 1, further comprising:

determining locations of physical objects within the 3D physical space;

wherein causing the holographic object to appear to relocate to the second location in the 3D physical space comprises at least one of;

selecting a display location for the holographic object so as to avoid visual occlusion of the holographic object by a physical or holographic object;

or

selecting a display location for the holographic object so as to avoid visual occlusion of another holographic object by the holographic object.

8. The method of claim 1, further comprising:

determining locations of physical objects in the 3D physical space; and

causing the 3D holographic container object to appear attached to a surface of a physical object.

9. The method of claim 1, further comprising:

determining locations of physical objects in the 3D physical space;

detecting a condition that a holographic object is occluded by a physical object, wherein the holographic object comprises at least one of the plurality of user interface objects or the 3D holographic container object; and

in response to detecting the condition, displaying to the user a visual cue to locate the holographic object.

10. A head-mounted display (HMD) device comprising:

a head fitting to enable the head-mounted display device to be worn on the head of a user;

a display device coupled to the head fitting and disposed to be positioned in front of the eyes of the user when the HMD device is worn by the user, the display device being at least partially transparent; and

a processor coupled to the display device and configured to

cause the display device to display to the user a plurality of holographic tools overlaid on a real-world view of a three-dimensional (3D) physical space in which the user is located, wherein each of the holographic tools is usable by the user to affect a holographic target object, each of the plurality of holographic tools being capable of being independently moved and operated by the user, each of the tools having a separate predetermined functionality usable by the user in a holographic workspace;

in response to a first user action, cause the plurality of holographic tools to appear to the user to be physically placed into a 3D holographic container object that appears at a first location in the 3D physical space;

in response to the first user action or a second user action, cause the 3D holographic container object to appear to relocate to a second location in the 3D physical space; and

cause the plurality of holographic tools to appear to the user to deploy from within the 3D holographic container object when the 3D holographic container object appears to be located at the second location in the 3D physical space.

11. The HMD device of claim 10, wherein causing the holographic object to appear to relocate to the second location in the 3D physical space is performed automatically in response to the user changing locations within the 3D physical space.

12. The HMD device of claim 10, wherein the processor is further configured to cause the display device to display the plurality of holographic tools in a constant orientation relative to the user while the user changes location or orientation within the 3D physical space.

13. The HMD device of claim 10, wherein the processor is further configured to cause the display device to display the plurality of holographic tools at a constant distance or range of distances from the user while the user changes location within the 3D physical space.

14. A non-transitory machine-readable storage medium storing instructions, an execution of which by a processor causes a head-mounted display (HMD) device to perform operations comprising:

displaying, to a user of the HMD device, a plurality of holographic user interface objects overlaid on a real-world view of a three-dimensional (3D) physical space in which the user is located, each of the plurality of holographic user interface objects being a different tool independently movable and usable by the user to create or edit a holographic target object;

in response to a first user input from the user, causing the plurality of holographic user interface objects to appear to the user to be physically placed into a 3D holographic container object that appears at a first location in the 3D physical space, wherein the 3D holographic container object is moveable in three dimensions in the 3D physical space;

in response to a second user input from the user indicating a relocation action, causing the 3D holographic container object to appear to relocate to a second location in the 3D physical space; and

causing the plurality of holographic user interface objects to appear to the user to deploy from within the 3D holographic container object when the 3D holographic container object appears to be located at the second location in the 3D physical space.

15. The non-transitory machine-readable storage medium of claim 14, wherein causing the holographic object to appear to relocate to the second location in the 3D physical space is performed automatically in response to the user changing locations within the 3D physical space.

16. The non-transitory machine-readable storage medium of claim 14, the operations further comprising:

causing the display device to display the plurality of holographic tools in a constant orientation relative to the user while the user changes location or orientation within the 3D physical space.

17. The non-transitory machine-readable storage medium of claim 14, the operations further comprising:

causing the display device to display the plurality of holographic tools at a constant distance or range of distances from the user while the user changes location within the 3D physical space.