MIXED REALITY DATA COLLABORATION

• US 20140354685A1
• Filed: 06/03/2013
• Published: 12/04/2014
• Est. Priority Date: 06/03/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Schaffalitzky, Frederik, Hou, Dan, Lazarow, Gavin
• Granted Patent: US 9,685,003 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G09G 2350/00 Solving problems of bandwidth in display systems
• CPC Class Codes: G09G 2370/022 Centralised management of display operation, e.g. in a server instead of locally

Abstract

Embodiments that relate to sharing mixed reality experiences among multiple display devices are disclosed. In one embodiment, a method includes receiving current versions of a plurality of data subtypes geo-located at a keyframe location. A world map data structure is updated to include the current versions, and a neighborhood request including the keyframe location is received from a display device. Based on the keyframe location, an identifier and current version indicator for each data subtype is provided to the device. A data request from the device for two or more of the data subtypes is received, and the two or more data subtypes are prioritized based on a priority hierarchy. Based on the prioritization, current versions of the data subtypes are sequentially provided to the device for augmenting an appearance of a mixed reality environment.

Claims

1. A data collaborator for enabling two or more display devices to share mixed reality experiences, the data collaborator comprising:

a synchronization program comprising instructions stored in a mass storage of the data collaborator, the synchronization program configured to;

receive a plurality of data subtypes from a first display device, each of the data subtypes being geo-located at a keyframe location and having a current version;

update a world map data structure to include the current versions of the data subtypes at the keyframe location;

receive a neighborhood request including the keyframe location from a second display device;

in response to the neighborhood request and based on the keyframe location, provide to the second display device an identifier for each of the plurality of data subtypes and a current version indicator for each of the plurality of data subtypes;

receive a data request from the second display device for two or more of the plurality of data subtypes;

prioritize the two or more data subtypes based on a priority hierarchy; and

based on the prioritization, sequentially provide current versions of the two or more data subtypes to the second display device, wherein an appearance of a mixed reality environment generated via the second display device is augmented using the current versions of the two or more data subtypes.

2. The data collaborator of claim 1, wherein the data request includes the identifier for each of the two or more data subtypes and a non-current version indicator for each of the two or more data subtypes.

3. The data collaborator of claim 2, wherein the synchronization program is further configured to:

compare the non-current version indicators to the current version indicators for the corresponding two or more data subtypes; and

based on the comparison, select the current versions of the two or more data subtypes for provision to the second display device.

4. The data collaborator of claim 1, wherein the plurality of data subtypes are prioritized by the first display device based on the priority hierarchy, and the plurality of data subtypes are sequentially received by the synchronization program from the first display device based on the priority hierarchy.

5. The data collaborator of claim 1, wherein the plurality of data subtypes are prioritized by the second display device based on the priority hierarchy, and the data request from the second display device includes a prioritization of the two or more data subtypes based on the priority hierarchy.

6. The data collaborator of claim 1, wherein prior to receiving the neighborhood request from the second display device and in response to receiving the plurality of data subtypes from the first display device, the synchronization program is further configured to provide to the second display device the identifiers that identify each of the plurality of data subtypes and the current version of each of the plurality of data subtypes received from the first display device.

7. The data collaborator of claim 1, wherein the synchronization program is further configured to:

determine that a previous version of one of the requested two or more data subtypes was previously received from the second display device; and

based on this determination, refrain from providing the previous version of the one of the requested data subtypes to the second display device.

8. The data collaborator of claim 1, wherein the data subtypes are selected from the group consisting of pose link transforms, anchor transforms, tracking images, scene images, correspondences between two-dimensional image coordinates and three-dimensional locations, keyframe metadata, keyframe descriptors, three-dimensional keyframe relocalization points, quantizations of keyframe descriptors, and rig-to-rig transforms.

9. The data collaborator of claim 1, wherein the priority hierarchy comprises a first priority that includes anchor transforms, a second priority that includes scene images, and a third priority that includes three-dimensional keyframe relocalization points.

10. The data collaborator of claim 1, wherein the second display device is a head-mounted display device.

11. A method for enabling two or more display devices to share mixed reality experiences, the method comprising:

receiving a plurality of data subtypes from a first display device, each of the data subtypes being geo-located at a keyframe location and having a current version;

updating a world map data structure to include the current versions of the data subtypes at the keyframe location;

receiving a neighborhood request including the keyframe location from a second display device;

in response to the neighborhood request and based on the keyframe location, providing to the second display device an identifier for each of the plurality of data subtypes and a current version indicator for each of the plurality of data subtypes;

receiving a data request from the second display device for two or more of the plurality of data subtypes;

prioritizing the two or more data subtypes based on a priority hierarchy; and

based on the prioritization, sequentially providing current versions of the two or more data subtypes to the second display device, wherein an appearance of a mixed reality environment generated via the second display device is augmented using the current versions of the two or more data subtypes.

12. The method of claim 11, wherein the data request includes the identifier for each of the two or more data subtypes and a non-current version indicator for each of the two or more data subtypes.

13. The method of claim 12, further comprising:

comparing the non-current version indicators to the current version indicators for the corresponding two or more data subtypes; and

based on the comparison, selecting the current versions of the two or more data subtypes for provision to the second display device.

14. The method of claim 11, wherein the plurality of data subtypes are prioritized by the first display device based on the priority hierarchy, and the method further comprises sequentially receiving the plurality of data subtypes from the first display device based on the priority hierarchy.

15. The method of claim 11, wherein the plurality of data subtypes are prioritized by the second display device based on the priority hierarchy, and the data request from the second display device includes a prioritization of the two or more data subtypes based on the priority hierarchy.

16. The method of claim 11, further comprising, prior to receiving the neighborhood request from the second display device and in response to receiving the plurality of data subtypes from the first display device, providing to the second display device the identifiers that identify each of the plurality of data subtypes and the current version of each of the plurality of data subtypes received from the first display device.

17. The method of claim 11, further comprising:

determining that a previous version of one of the requested two or more data subtypes was previously received from the second display device; and

based on this determination, refraining from providing the previous version of the one of the requested data subtypes to the second display device.

18. The method of claim 11, wherein the data subtypes are selected from the group consisting of pose link transforms, anchor transforms, tracking images, scene images, correspondences between two-dimensional image coordinates and three-dimensional locations, keyframe metadata, keyframe descriptors, three-dimensional keyframe relocalization points, quantizations of keyframe descriptors, and rig-to-rig transforms.

19. The method of claim 11, wherein the priority hierarchy comprises a first priority that includes anchor transforms, a second priority that includes scene images, and a third priority that includes three-dimensional keyframe relocalization points.

20. A method for enabling two or more head-mounted display devices to share mixed reality experiences, the method comprising:

receiving a plurality of data subtypes from a first head-mounted display device, each of the data subtypes being geo-located at a keyframe location and having a current version;

updating a world map data structure to include the current versions of the data subtypes at the keyframe location;

receiving a neighborhood request including the keyframe location from a second head-mounted display device;

in response to the neighborhood request and based on the keyframe location, providing to the second display head-mounted device an identifier for each of the plurality of data subtypes and a current version indicator for each of the plurality of data subtypes;

receiving a data request from the second head-mounted display device for two or more of the plurality of data subtypes, the data request including an identifier for each of the two or more data subtypes and a non-current version indicator for each of the two or more data subtypes;

comparing the non-current version indicators to the current version indicators for the corresponding data subtypes;

based on the comparison, selecting the current versions of the two or more data subtypes for provision to the second head-mounted display device;

prioritizing the two or more data subtypes based on a priority hierarchy; and

based on the prioritization, sequentially providing the current versions of the two or more data subtypes to the second head-mounted display device, wherein an appearance of a mixed reality environment generated via the second head-mounted display device is augmented using the current versions of the two or more data subtypes.