Shared scene mesh data synchronization

• US 9,407,865 B1
• Filed: 01/21/2015
• Issued: 08/02/2016
• Est. Priority Date: 01/21/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Reitel, Juri, Ellis, Martin, Birjukov, Andrei, Miao, ZhiCheng, Menezes, Ryan S.
• Primary Examiner(s): Anwah, Olisa
• Application Number:
  US14/602,237
• Publication Number:
  US 20160212375A1
• Time in Patent Office: 559 Days
• Field of Search: 348/14.01, 348/14.03, 348/14, 348/12, 348/14.13
• US Class Current: 1/1
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: H04L 65/1089 by adding media; by removing media
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 21/23406 involving management of server-side video buffer
• CPC Class Codes: H04N 21/236 Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream multiplexing of data packets for data networks, e.g. RTP/UDP H04L65/00
• CPC Class Codes: H04N 21/23614 Multiplexing of additional data and video streams arrangements for simultaneous broadcast of plural pieces of information H04H20/28
• CPC Class Codes: H04N 21/2385 Channel allocation H04N21/266 takes precedence; Bandwidth allocation H04N21/24 takes precedence ; allocation of channels according to the instantaneous demands of the users in time-division multiplex systems H04J3/1682; admission control, resource allocation in open networks H04L12/5692; arrangements for maintenance or administration in data switching networks involving bandwidth and capacity management H04L41/0896; negotiating bandwidth in wireless networks H04W28/16
• CPC Class Codes: H04N 21/24 Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests monitoring of server performance or load G06F11/34; arrangements for observation, testing or troubleshooting for broadcast or for distribution combined with broadcast H04H20/12
• CPC Class Codes: H04N 21/6375 for requesting retransmission , e.g. of data packets lost or corrupted during transmission from serverARQ protocols H04L1/18; implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP] H04L69/16
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56
• CPC Class Codes: H04N 7/002 Special television systems not provided for by H04N7/007 - H04N7/18 still pictures via a television channel H04N1/00098
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

A user device within a communication architecture, the user device comprising: an image capture device configured to determine image data and intrinsic/extrinsic capture device data for the creation of a video channel defining a shared scene; a surface reconstruction entity configured to determine surface reconstruction data associated with the image data from the image capture device; a video channel configured to encode and packetize the image data and intrinsic/extrinsic capture device data; a surface reconstruction channel configured to encode and packetize the surface reconstruction data; a transmitter configured to transmit the video and surface reconstruction channel packets; and a bandwidth controller configured to control the bandwidth allocated to the video channel and the surface reconstruction channel.

Claims

1. A user device within a communication architecture, the user device comprising:

an image capture device configured to determine image data and intrinsic/extrinsic capture device data for the creation of a video channel defining a shared scene;

a surface reconstruction entity configured to determine surface reconstruction data associated with the image data from the image capture device;

a video channel configured to encode and packetize the image data and intrinsic/extrinsic capture device data;

a surface reconstruction channel configured to encode and packetize the surface reconstruction data;

a transmitter configured to transmit the video and surface reconstruction channel packets; and

a bandwidth controller configured to control the bandwidth allocated to the video channel and the surface reconstruction channel.

2. The user device of claim 1, wherein the bandwidth controller is configured to:

determine an initial bandwidth capacity; and

equally divide the initial bandwidth between the surface reconstruction channel and the video channel.

3. The user device of claim 1, wherein the bandwidth controller is configured to:

determine after a predetermined period a short-term bandwidth capacity estimate and a long-term bandwidth capacity estimate for the surface reconstruction channel and the video channel; and

control a level-of-detail parameter within the encoding and packetizing the surface reconstruction data based on the short-term bandwidth capacity estimate and a long-term bandwidth capacity estimate for the surface reconstruction channel and the video channel.

4. The user device of claim 3, wherein the bandwidth controller is configured to allocate a bandwidth to encode and packetize the image data and intrinsic/extrinsic capture device data within the video channel based on the short-term bandwidth capacity estimate and the long-term bandwidth capacity estimate for the surface reconstruction channel and the video channel.

5. The user device of claim 1, wherein the bandwidth controller is further configured to control retransmission of surface reconstruction packet fragments based on an ACK control system.

6. The user device of claim 1, further comprising:

a pending buffer for holding references of any surface reconstruction data packets until they are sent; and

an unacknowledged buffer for holding references of any surface reconstruction data packets once the packets are sent, wherein the bandwidth controller is configured to limit the surface reconstruction packets sent based on the window size of the unacknowledged buffer.

7. The user device of claim 6, further comprising the transmitter configured to send surface reconstruction packets referenced in the pending buffer independently of receiving a feedback message.

8. A communication architecture comprising:

the user device of claim 1; and

a further user device, the further user device comprising;

a receiver configured to receive packets for a video channel defining a shared scene and surface reconstruction channel associated with the image data from the image capture device within the video channel;

a video packet extractor configured to extract the image data and intrinsic/extrinsic capture device data from within the video channel;

a surface reconstruction packet extractor configured to extract surface reconstruction data packets from within a surface reconstruction channel; and

a feedback message generator configured to generate at least on feedback message for controlling the bandwidth allocated to the video channel and the surface reconstruction channel.

9. A method implemented within a communication architecture, the method comprising:

determining image data and intrinsic/extrinsic capture device data from an image capture device for the creation of a video channel defining a shared scene;

determining surface reconstruction data associated with the image data from the image capture device;

encoding and packetizing the image data and intrinsic/extrinsic capture device data within the video channel;

encoding and packetizing the surface reconstruction data within a surface reconstruction channel;

transmitting the video channel and surface reconstruction channel packets; and

controlling the bandwidth allocated to the video channel and the surface reconstruction channel.

10. The method of claim 9, wherein controlling the bandwidth allocated to the video channel and the surface reconstruction channel comprises:

determining an initial bandwidth capacity; and

equally dividing the initial bandwidth between the surface reconstruction channel and the video channel.

11. The method of claim 9, wherein controlling the bandwidth allocated to the video channel and the surface reconstruction channel comprises:

determining after a predetermined period a short-term bandwidth capacity estimate and a long-term bandwidth capacity estimate for the surface reconstruction channel and the video channel;

controlling the bandwidth allocated to the surface reconstruction channel based on the long-term bandwidth capacity estimate for the surface reconstruction comprises controlling a level-of-detail parameter within the encoding and packetizing the surface reconstruction data based on the short-term bandwidth capacity estimate and the long-term bandwidth capacity estimate for the surface reconstruction channel and the video channel.

12. The method of claim 11, wherein controlling the bandwidth allocated to the video channel comprises allocating a bandwidth to encoding and packetizing the image data and intrinsic/extrinsic capture device data within the video channel based on the short-term bandwidth capacity estimate and the long-term bandwidth capacity estimate for the surface reconstruction channel and the video channel.

13. The method of claim 9, wherein controlling the bandwidth allocated to the video channel and the surface reconstruction channel further comprises controlling retransmission of surface reconstruction packet fragments based on an ACK control system.

14. The method of claim 9, further comprising:

maintaining a pending buffer for holding references of any surface reconstruction data packets until they are sent; and

maintaining an unacknowledged buffer for holding references of any surface reconstruction data packets once the packets are sent, wherein controlling the bandwidth allocated to the video channel and the surface reconstruction channel comprises limiting the surface reconstruction packets sent based on the window size of the unacknowledged buffer.

15. The method of claim 14, further comprising sending surface reconstruction packets from the pending buffer independently of receiving a feedback message.

16. The method of claim 9, further comprising:

receiving packets for a video channel defining the shared scene and surface reconstruction channel associated with the image data from the image capture device within the video channel;

extracting the image data and intrinsic/extrinsic capture device data from within the video channel;

extracting surface reconstruction data packets from within a surface reconstruction channel;

generating at least on feedback message for controlling the bandwidth allocated to the video channel and the surface reconstruction channel.

17. A computer program product, the computer program product being embodied on a computer-readable storage medium and configured so as when executed on a processor of a protocol endpoint entity within a shared scene architecture, to:

determine image data and intrinsic/extrinsic capture device data from an image capture device for the creation of a video channel defining a shared scene;

determine surface reconstruction data associated with the image data from the image capture device;

encode and packetize the image data and intrinsic/extrinsic capture device data within the video channel;

encode and packetize the surface reconstruction data within a surface reconstruction channel;

transmit the video channel and surface reconstruction channel packets; and

control the bandwidth allocated to the video channel and the surface reconstruction channel.

18. The computer program product of claim 17, wherein the control of the bandwidth allocated to the video channel and the surface reconstruction channel causes the computer program product to:

determine an initial bandwidth capacity; and

equally divide the initial bandwidth between the surface reconstruction channel and the video channel.

19. The computer program product of claim 18, wherein the control of the bandwidth allocated to the video channel and the surface reconstruction channel causes the computer program product to:

determine after a predetermined period a short-term bandwidth capacity estimate and a long-term bandwidth capacity estimate for the surface reconstruction channel and the video channel;

control a level-of-detail parameter within the encoding and packetizing the surface reconstruction data based on the short-term bandwidth capacity estimate and the long-term bandwidth capacity estimate for the surface reconstruction channel and the video channel.

20. The computer program product of claim 19, wherein the control of the bandwidth allocated to the video channel and the surface reconstruction channel causes the computer program product to controlling the bandwidth allocated to the video channel comprises allocating a bandwidth to encoding and packetizing the image data and intrinsic/extrinsic capture device data within the video channel based on the short-term bandwidth capacity estimate and the long-term bandwidth capacity estimate for the surface reconstruction channel and the video channel.