Shared scene mesh data synchronization

• US 9,544,537 B2
• Filed: 06/30/2016
• Issued: 01/10/2017
• Est. Priority Date: 01/21/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Birjukov, Andrei, Reitel, Juri, Ellis, Martin, Miao, ZhiCheng, Menezes, Ryan S.
• Primary Examiner(s): Anwah, Olisa
• Application Number:
  US15/199,337
• Publication Number:
  US 20160316171A1
• Time in Patent Office: 194 Days
• Field of Search: 348/14.01, 348/14.03, 348/14.07, 348/14.08, 348/14.11, 348/14.12, 348/14.13, 348/14.15
• US Class Current: 1/1
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: H04L 65/1089 by adding media; by removing media
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 21/23406 involving management of server-side video buffer
• CPC Class Codes: H04N 21/236 Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream multiplexing of data packets for data networks, e.g. RTP/UDP H04L65/00
• CPC Class Codes: H04N 21/23614 Multiplexing of additional data and video streams arrangements for simultaneous broadcast of plural pieces of information H04H20/28
• CPC Class Codes: H04N 21/2385 Channel allocation H04N21/266 takes precedence; Bandwidth allocation H04N21/24 takes precedence ; allocation of channels according to the instantaneous demands of the users in time-division multiplex systems H04J3/1682; admission control, resource allocation in open networks H04L12/5692; arrangements for maintenance or administration in data switching networks involving bandwidth and capacity management H04L41/0896; negotiating bandwidth in wireless networks H04W28/16
• CPC Class Codes: H04N 21/24 Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests monitoring of server performance or load G06F11/34; arrangements for observation, testing or troubleshooting for broadcast or for distribution combined with broadcast H04H20/12
• CPC Class Codes: H04N 21/6375 for requesting retransmission , e.g. of data packets lost or corrupted during transmission from serverARQ protocols H04L1/18; implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP] H04L69/16
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56
• CPC Class Codes: H04N 7/002 Special television systems not provided for by H04N7/007 - H04N7/18 still pictures via a television channel H04N1/00098
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

A user device within a communication architecture, the user device comprising: an image capture device configured to determine image data and intrinsic/extrinsic capture device data for the creation of a video channel defining a shared scene; a surface reconstruction entity configured to determine surface reconstruction data associated with the image data from the image capture device; a video channel configured to encode and packetize the image data and intrinsic/extrinsic capture device data; a surface reconstruction channel configured to encode and packetize the surface reconstruction data; a transmitter configured to transmit the video and surface reconstruction channel packets; and a bandwidth controller configured to control the bandwidth allocated to the video channel and the surface reconstruction channel.

Claims

1. A method comprising:

receiving object interaction instructions for interacting with an object in a mixed reality scene;

generating modified object attributes based on the object interaction instructions;

generating an object attribute message based on the modified object attributes;

associating the object attribute message with a sequence number and an object identifier value, the sequence number identifying a position within a sequence of modifications and the object identifier value identifying the object;

determining whether a frame period has ended;

responsive to determining that the frame period has not ended;

generating another object attribute message;

responsive to determining that the frame period has ended;

determining for the frame period a set of object attribute messages corresponding to a specified object identifier value;

determining a latest set of messages from the set of object attribute messages based on associated sequence numbers for the set of object attribute messages; and

deleting in a send path object attribute messages in the set of object attribute messages that are not in the latest set of messages.

2. The method of claim 1, wherein the receiving comprises receiving the object interaction instructions from a participant of the mixed reality scene.

3. The method of claim 2, wherein the participant is not an owner of the mixed reality scene.

4. The method of claim 1, wherein the frame period is a video frame period.

5. The method of claim 1, wherein the method is performed at an intermediate server.

6. The method of claim 1, wherein the method is performed at a user device of a participant of the mixed reality scene.

7. The method of claim 1, wherein the receiving comprises receiving the object interaction instructions from an owner of the mixed reality scene.

8. The method of claim 1, wherein the object interaction instructions involve playing video.

9. A method comprising:

receiving object attribute messages based on object interaction instructions for interacting with the interactive objects in a mixed reality scene;

determining whether a frame period has ended;

responsive to determining that the frame period has not ended;

receiving other object attribute messages;

responsive to determining that the frame period has ended;

estimating a number of processor cycles consumed per update of an interactive object;

determining a current processor load;

determining an object priority for a specified object;

setting a limit on a number of interactive object updates during a frame period, the limit being based on at least one of the number of processor cycles, the current processor load, and the object priority;

determining a set of most recent object attribute messages for the specified object in accordance with the limit on the number of interactive object updates during the frame period; and

deleting in a receive path object attribute messages corresponding to the specified object that are not in set of most recent object attribute messages.

10. The method of claim 9, wherein the method is performed at a user device of a participant of the mixed reality scene who is not an owner of the mixed reality scene.

11. The method of claim 9, wherein the object priority is based on whether the specified object is in view, whether the specified object has been recently viewed, or whether the specified object has been recently interacted with.

12. The method of claim 9, wherein the method is performed at a user device of a participant of the mixed reality scene who is an owner of the mixed reality scene.

13. The method of claim 9, wherein the frame period is a video frame period.

14. The method of claim 9, wherein the determining the set of most recent object attribute messages for the specified object comprises determining a set of most recent sequence numbers associated with the received object attribute messages.

15. A computer program product, the computer program product being embodied on a computer-readable storage medium and configured so as when executed on a processor of a protocol endpoint entity within a shared scene architecture, to:

receive object attribute messages based on object interaction instructions for interacting with the interactive objects in a shared scene;

determine whether a frame period has ended;

responsive to a determination that the frame period has not ended;

receive other object attribute messages;

responsive to a determination that the frame period has ended;

estimate a number of processor cycles consumed per update of an interactive object;

determine a current processor load;

determine an object priority for a specified object;

set a limit on a number of interactive object updates during a frame period, the limit being based on at least one of the number of processor cycles, the current processor load, and the object priority;

determine a set of most recent object attribute messages for the specified object in accordance with the limit on the number of interactive object updates during the frame period; and

delete in a receive path object attribute messages corresponding to the specified object that are not in set of most recent object attribute messages.

16. The computer program product of claim 15, wherein the protocol endpoint entity comprises a user device of a participant of the shared scene who is not an owner of the shared scene.

17. The computer program product of claim 15, wherein the object priority is based on whether the specified object is in view, whether the specified object has been recently viewed, or whether the specified object has been recently interacted with.

18. The computer program product of claim 15, wherein the protocol endpoint entity comprises a user device of a participant of the shared scene who is an owner of the shared scene.

19. The computer program product of claim 15, wherein the frame period is a video frame period.

20. The computer program product of claim 15, wherein to determine the set of most recent object attribute messages for the specified object comprises to determine a set of most recent sequence numbers associated with the received object attribute messages.