Mixed reality experience sharing

• US 9,342,929 B2
• Filed: 01/22/2013
• Issued: 05/17/2016
• Est. Priority Date: 01/22/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kamuda, Nicholas, Kinnebrew, Peter Tobias
• Primary Examiner(s): Merouan, Abderrahim
• Assistant Examiner(s): Chin, Michelle
• Application Number:
  US13/747,328
• Publication Number:
  US 20140204077A1
• Time in Patent Office: 1,211 Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/497 Partially or entirely replaying previous game actions
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/205 for detecting the geographical location of the game platform
• CPC Class Codes: A63F 2300/407 Data transfer via internet
• CPC Class Codes: A63F 2300/63 for controlling the execution of the game in time
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: G02B 27/00 Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment

Abstract

Embodiments that relate to presenting a textured shared world model of a physical environment are disclosed. One embodiment includes receiving geo-located crowd-sourced structural data items of the physical environment. The structural data items are stitched together to generate a 3D spatial shared world model. Geo-located crowd-sourced texture data items are also received and include time-stamped images or video. User input of a temporal filter parameter is used to temporally filter the texture data items. The temporally-filtered texture data items are applied to surfaces of the 3D spatial shared world model to generate a textured shared world model of the physical environment. The textured shared world model is then provided for display by a display device.

Claims

1. A mixed reality experience sharing system for presenting a textured shared world model of a physical environment, the textured shared world model including a plurality of crowd-sourced structural data items that are geo-located at the physical environment, and a plurality of crowd-sourced texture data items that are geo-located at the physical environment, the mixed reality experience sharing system comprising:

a computing device comprising a processor; and

a mixed reality experience sharing program executed by the processor of the computing device, the mixed reality experience sharing program configured to;

receive the plurality of crowd-sourced structural data items including depth information;

stitch together the plurality of crowd-sourced structural data items to generate a 3D spatial shared world model of the physical environment;

receive the plurality of crowd-sourced texture data items from a plurality of sources, each of the texture data items comprising a time-stamped image or video, wherein the plurality of crowd-sourced texture data items comprise at least one hologram and metadata describing the hologram;

receive user input of a temporal filter parameter;

filter the plurality of crowd-sourced texture data items by the temporal filter parameter to generate a temporally-filtered subgroup of the crowd-sourced texture data items;

apply to surfaces of the 3D spatial shared world model of the physical environment the temporally-filtered subgroup of the crowd-sourced texture data items to generate the textured shared world model of the physical environment;

provide the textured shared world model of the physical environment to a display program for display by a display device; and

locate the hologram within the textured shared world model of the physical environment.

2. The mixed reality experience sharing system of claim 1,

wherein the temporal filter parameter corresponds to a predated time that is earlier than a current time.

3. The mixed reality experience sharing system of claim 2, wherein the mixed reality experience sharing program is further configured to:

filter the plurality of crowd-sourced structural data items by the temporal filter parameter to generate a temporally-filtered subgroup of the crowd-sourced structural data items; and

stitch together the temporally-filtered subgroup of the crowd-sourced structural data items to generate the 3D spatial shared world model of the physical environment.

4. The mixed reality experience sharing system of claim 1, wherein the mixed reality experience sharing program is further configured to receive the plurality of crowd-sourced texture data items from at least a user of the display device and a third party source.

5. The mixed reality experience sharing system of claim 1, wherein each of the plurality of crowd-sourced texture data items is selected from the group consisting of light illuminating one or more of the crowd-sourced structural data items, an image of an object in the physical environment, a video of an object in the physical environment, and audio of the physical environment.

6. The mixed reality experience sharing system of claim 1, wherein the mixed reality experience sharing program is further configured to prioritize the plurality of crowd-sourced structural data items according to one or more of a fidelity, an age, a trustworthiness of a source, and a social proximity of the source of the crowd-sourced structural data items.

7. The mixed reality experience sharing system of claim 1, wherein the mixed reality experience sharing program is further configured to receive the plurality of crowd-sourced structural data items from at least one head-mounted display device.

8. The mixed reality experience sharing system of claim 1, wherein the mixed reality experience sharing program is further configured to prioritize the plurality of crowd-sourced texture data items according to one or more of a trustworthiness of a source and a social proximity of the source of the crowd-sourced texture data items.

9. A method for presenting a textured shared world model of a physical environment via a display device, the method comprising:

receiving a plurality of crowd-sourced structural data items that are geo-located at the physical environment and include depth information;

stitching together the plurality of crowd-sourced structural data items to generate a 3D spatial shared world model of the physical environment;

receiving from a plurality of sources a plurality of crowd-sourced texture data items that are geo-located at the physical environment, each of the crowd-sourced texture data items comprising a time-stamped image or video, wherein the plurality of crowd-sourced texture data items comprise at least one hologram and metadata describing the hologram;

receiving user input of a temporal filter parameter;

filtering the plurality of crowd-sourced texture data items by the temporal filter parameter to generate a temporally-filtered subgroup of the crowd-sourced texture data items;

applying to surfaces of the 3D spatial shared world model of the physical environment the temporally-filtered subgroup of the crowd-sourced texture data items to generate the textured shared world model of the physical environment;

providing the textured shared world model of the physical environment to a display program for display by the display device; and

locating the hologram within the textured shared world model of the physical environment.

10. The method of claim 9, wherein the temporal filter parameter corresponds to a predated time that is earlier than a current time.

11. The method of claim 10, further comprising:

filtering the plurality of crowd-sourced structural data items by the temporal filter parameter to generate a temporally-filtered subgroup of the crowd-sourced structural data items; and

wherein stitching together the plurality of crowd-sourced structural data items comprises stitching together the temporally-filtered subgroup of the crowd-sourced structural data items to generate the 3D spatial shared world model of the physical environment.

12. The method of claim 9, further comprising receiving the plurality of crowd-sourced texture data items from at least a user of the display device and a third party source.

13. The method of claim 9, wherein each of the plurality of crowd-sourced texture data items are selected from the group consisting of light illuminating one or more of the crowd-sourced structural data items, an image of an object in the physical environment, a video of an object in the physical environment, and audio of the physical environment.

14. The method of claim 9, further comprising prioritizing the plurality of crowd-sourced structural data items according to one or more of a fidelity, an age, a trustworthiness of a source, and a social proximity of the source of the crowd-sourced structural data items.

15. The method of claim 9, further comprising prioritizing the plurality of crowd-sourced texture data items according to one or more of a trustworthiness of a source and a social proximity of the source of the crowd-sourced texture data items.

16. A mixed reality experience sharing system for presenting a textured shared world model of a physical environment, the textured shared world model including a plurality of crowd-sourced structural data items that are geo-located at the physical environment, and a plurality of crowd-sourced texture data items that are geo-located at the physical environment, the mixed reality experience sharing system comprising:

a computing device comprising a processor; and

a mixed reality experience sharing program executed by the processor of the computing device, the mixed reality experience sharing program configured to;

receive the plurality of crowd-sourced structural data items including depth information;

stitch together the plurality of crowd-sourced structural data items to generate a 3D spatial shared world model of the physical environment;

receive the plurality of crowd-sourced texture data items from a plurality of sources, each of the texture data items comprising a time-stamped image or video, wherein the plurality of crowd-sourced texture data items comprise at least one hologram and metadata describing the hologram;

receive user input of a temporal filter parameter corresponding to a predated time that is earlier than a current time;

filter the plurality of crowd-sourced texture data items by the temporal filter parameter to generate a temporally-filtered subgroup of the crowd-sourced texture data items;

apply to surfaces of the 3D spatial shared world model of the physical environment the temporally-filtered subgroup of the crowd-sourced texture data items to generate the textured shared world model of the physical environment;

provide the textured shared world model of the physical environment to a display program for display by a display device; and

locate the hologram within the textured shared world model of the physical environment.

17. The mixed reality experience sharing system of claim 16, wherein the mixed reality experience sharing program is further configured to prioritize the plurality of crowd-sourced texture data items according to one or more of a trustworthiness of a source and a social proximity of the source of the crowd-sourced texture data items.