Location-based holographic experience

• US 10,210,661 B2
• Filed: 04/25/2016
• Issued: 02/19/2019
• Est. Priority Date: 04/25/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kohler, Jeffrey, Wright, Shawn Crispin, McManus, Michael W., Ambrus, Anthony, Dack, James, Hoffman, Craig Lee, Hogan, Alexander James, Krauskopf, Korey, Zibits, Andrew, Omotani, Todd Alan, Scott, Jason Bradley
• Primary Examiner(s): Lam, Vinh T
• Application Number:
  US15/137,960
• Publication Number:
  US 20170307888A1
• Time in Patent Office: 1,030 Days
• Field of Search: 345 7, 345 8, 345633, 706 10, 706 45
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0174 holographic
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Examples disclosed herein relate to providing a location-based holographic experience. One example provides a head-mounted display device comprising a see-through display, one or more position sensors, a logic subsystem, and a storage subsystem comprising instructions executable by the logic subsystem to obtain data representing a plurality of holographic objects, acquire sensor data via the one or more position sensors to monitor a position of the head-mounted display device along a path of a holographic experience, detect that the position of the head-mounted display device meets a first position-based condition regarding a first holographic object, and display the first holographic object at a corresponding location for the first holographic object.

Claims

1. A head-mounted display device configured to provide a holographic experience associated with an experience space, the head-mounted display device comprising:

a see-through display;

one or more position sensors configured to provide data relating to a position of the head-mounted display device within the experience space, the one or more position sensors comprising a depth image sensor;

a logic subsystem comprising a processor; and

a storage subsystem comprising a storage device storing instructions executable by the processor of the logic subsystem to

obtain a surface reconstruction mesh of the experience space from one or more of the depth image sensor, a server computing device, and the storage device of the storage subsystem, and also obtain, from one or more of the server computing device and the storage device of the storage subsystem, image data representing a plurality of holographic objects, each holographic object being associated with a corresponding location in the experience space based upon the obtained surface reconstruction mesh of the experience space,

control the depth image sensor to acquire sensor data monitor the position of the head-mounted display device along a path extending from a beginning location to an end location of the holographic experience by tracking a position of one or more real-world objects using the sensor data from the depth image sensor and the surface reconstruction mesh of the experience space,

from the sensor data from the depth image sensor, detect that the position of the head-mounted display device meets a first position-based condition regarding a first holographic object, the first position-based condition comprising a threshold distance from the head-mounted display device to a corresponding location for the first holographic object as determined by tracking the position of the one or more real objects using the sensor data from the depth image sensor, and

in response, control the see-through display to display the first holographic object at the corresponding location for the first holographic object in the experience space via the see-through display, the first holographic object providing guidance information along the path of the holographic experience.

2. The head-mounted display device of claim 1, wherein the instructions are further executable to:

from the sensor data, detect that the position of the head-mounted display device meets a second position-based condition regarding a second holographic object, and

in response, display the second holographic object at a corresponding location for the second holographic object in the experience space via the see-through display, the second holographic object providing further guidance information along the path of the holographic experience.

3. The head-mounted display device of claim 1, wherein the instructions are further executable to dynamically update the first holographic object based upon a change in position of the see-through display within the holographic experience as detected via the sensor data.

4. The head-mounted display device of claim 3, wherein the instructions executable to dynamically update the first holographic object comprise instructions executable to display updated guidance information regarding a distance between the see-through display and a target location within the holographic experience.

5. The head-mounted display device of claim 1, wherein the instructions are further executable to send a request to trigger a physical effect output device located within the experience space to output a physical effect associated with a selected holographic object.

6. The head-mounted display device of claim 1, wherein the instructions are further executable to display the first holographic object to avoid occlusion of the first holographic object by other objects in the experience space.

7. The head-mounted display device of claim 1, further comprising one or more image sensors, and wherein the instructions are further executable to detect via image data acquired by the one or more image sensors an optically-readable label within the holographic experience, and display a second holographic object based upon a location of the optically-readable label.

8. The head-mounted display device of claim 1, wherein the instructions are further executable to send at least a portion of the sensor data to a remote server.

9. The head-mounted display device of claim 1, wherein the instructions are further executable to obtain information regarding one or more physical characteristics of a user of the head-mounted display device and display the first holographic object based on the one or more physical characteristics of the user.

10. The head-mounted display device of claim 9, wherein the instructions are executable to display the first holographic object based on one or more of a height and an interpupillary distance of the user.

11. On a display device, a method of providing a holographic experience associated with an experience space, the display device comprising a computing device comprising a storage device and processor, and the display device also comprising one or more position sensors, the one or more position sensors comprising a depth image sensor, the method comprising:

obtaining a surface reconstruction mesh of the experience space from one or more of the depth image sensor, the storage device of the display device, and a remote server;

obtaining from one or more of the storage device of the display device and the remote server data representing a plurality of holographic objects associated with the holographic experience, each holographic object being associated with a corresponding location in the experience space based upon the obtained surface reconstruction mesh of the experience space;

controlling the depth image sensor to acquire sensor data from the depth image sensor to monitor motion of the display device along a path extending from a beginning location to an end location of the holographic experience by tracking a position of one or more real-world objects using the sensor data from the depth image sensor and the surface reconstruction mesh of the experience space;

from the sensor data, detecting that the display device meets a first position-based condition with regard to a first holographic object, the first position-based condition comprising a threshold distance from the display device to a corresponding location for the first holographic object as determined by tracking the position of the one or more real objects using the sensor data from the depth image sensor;

in response, control the display device to display the first holographic object at the corresponding location in the experience space for the first holographic object via the display device, the first holographic object providing guidance information along the path of the holographic experience; and

from the sensor data, detecting that the display device meets a second position-based condition with regard to a second holographic object, and in response displaying the second holographic object at a corresponding location in the experience space for the second holographic object via the display device, the second holographic object providing further guidance information along the path of the holographic experience.

12. The method of claim 11, further comprising dynamically updating one or more of the first holographic object and the second holographic object based upon a change in position of the display device within the holographic experience as detected via the sensor data.

13. The method of claim 11, further comprising sending a request to trigger a physical effect within the experience space to accompany one or more of the first holographic object and the second holographic object based upon the corresponding location in the experience space associated with one or more of the first holographic object and the second holographic object.

14. The method of claim 11, further comprising sending at least a portion of the sensor data to the remote server.

15. A computing system configured to facilitate presentation of a holographic experience associated with an experience space via each of a plurality of display devices configured to present the holographic experience, the computing system comprising:

a communications subsystem configured to connect to each of the plurality of display devices;

a logic system comprising a processor; and

a storage system comprising a storage device having instructions executable by the processor to

receive input of a plurality of holographic objects to be displayed during the holographic experience, each holographic object being associated with a corresponding location in the experience space and providing guidance information along a path extending from a beginning location to an end location of the holographic experience,

determine an identity of each display device of the plurality of display devices,

control the communications subsystem to send to a first display device a first set of holographic objects selected from the plurality of holographic objects,

control the communications subsystem to send to a second display device a second set of holographic objects selected from the plurality of holographic objects, the second set of holographic objects sent to the second display device being at least partially non-overlapping with the first set of holographic objects sent to the first display device,

control the communications subsystem to receive sensor data from each display device, the sensor data for each display device comprising data relating to a position of the display device, and

based upon the sensor data, monitor a position of each of the first display device and the second display device within the holographic experience.

16. The computing system of claim 15, wherein the second set of holographic objects is at least partially non-overlapping with the first set of holographic objects such that the first set and the second set provide different guidance information at a portion of the holographic experience.

17. The computing system of claim 15, wherein the instructions are further executable to send an instruction to a physical effect output device located within the experience space to output a physical effect to accompany a selected holographic object.

18. The computing system of claim 15, wherein the instructions are further executable to, for each display device of the plurality of display devices, receive an interpupillary distance of a user of the display device and associate the interpupillary distance with the identity of the display device.

19. The computing system of claim 18, wherein the instructions are further executable to, for a selected display device, receive an indication that a user of the selected display device has switched to a different display device, and send the interpupillary distance associated with the selected display device to the different display device.

20. The computing system of claim 15, wherein the instructions are further executable to receive telemetric data from one or more display devices of the plurality of display devices upon occurrence of an error in the presentation of the holographic experience.