Shared scene mesh data synchronization

• US 9,924,159 B2
• Filed: 05/15/2017
• Issued: 03/20/2018
• Est. Priority Date: 01/21/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Reitel, Juri, Ellis, Martin, Birjukov, Andrei, Miao, ZhiCheng, Menezes, Ryan S.
• Primary Examiner(s): Anwah, Olisa
• Application Number:
  US15/595,080
• Publication Number:
  US 20170251200A1
• Time in Patent Office: 309 Days
• Field of Search: 348 1401, 348 1403, 348 1412, 348 1413
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: H04L 65/1089 by adding media; by removing media
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 21/23406 involving management of server-side video buffer
• CPC Class Codes: H04N 21/236 Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream multiplexing of data packets for data networks, e.g. RTP/UDP H04L65/00
• CPC Class Codes: H04N 21/23614 Multiplexing of additional data and video streams arrangements for simultaneous broadcast of plural pieces of information H04H20/28
• CPC Class Codes: H04N 21/2385 Channel allocation H04N21/266 takes precedence; Bandwidth allocation H04N21/24 takes precedence ; allocation of channels according to the instantaneous demands of the users in time-division multiplex systems H04J3/1682; admission control, resource allocation in open networks H04L12/5692; arrangements for maintenance or administration in data switching networks involving bandwidth and capacity management H04L41/0896; negotiating bandwidth in wireless networks H04W28/16
• CPC Class Codes: H04N 21/24 Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests monitoring of server performance or load G06F11/34; arrangements for observation, testing or troubleshooting for broadcast or for distribution combined with broadcast H04H20/12
• CPC Class Codes: H04N 21/6375 for requesting retransmission , e.g. of data packets lost or corrupted during transmission from serverARQ protocols H04L1/18; implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP] H04L69/16
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56
• CPC Class Codes: H04N 7/002 Special television systems not provided for by H04N7/007 - H04N7/18 still pictures via a television channel H04N1/00098
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

Within a communication system, packets are received for a video channel defining a shared scene and a surface reconstruction channel associated with surface reconstruction data of image data. The image data and intrinsic and extrinsic capture device data are from an image capture device and are encoded and packetized within the video channel. The surface reconstruction data is encoded and packetized within the surface reconstruction channel. The image data and the intrinsic and extrinsic capture device data are extracted from within the video channel. The surface reconstruction data is extracted from within the surface reconstruction channel. The shared scene is rendered using the image data, the intrinsic and extrinsic capture device data, and the surface reconstruction data.

Claims

1. A method implemented within a communication system, the method comprising:

receiving packets for a video channel defining a shared scene and a surface reconstruction channel associated with surface reconstruction data of image data, the image data as well as intrinsic and extrinsic capture device data being from an image capture device and being encoded and packetized within the video channel, the surface reconstruction data being encoded and packetized within the surface reconstruction channel;

extracting the image data as well as the intrinsic and extrinsic capture device data from within the video channel;

extracting the surface reconstruction data from within the surface reconstruction channel; and

rendering the shared scene using the image data, the intrinsic and extrinsic capture device data, and the surface reconstruction data.

2. The method of claim 1, wherein the rendering the shared scene further comprises displaying a three-dimensional (3D) shared scene.

3. The method of claim 1, wherein surface reconstruction data packets within the surface reconstruction channel are received independently of sending a feedback message.

4. The method of claim 1, further comprising generating at least one feedback message for controlling retransmission of the surface reconstruction data.

5. The method of claim 4, further comprising detecting a content-start event, a content-stop event, or an end-of-file event, and wherein the at least one feedback message is generated responsive to the detecting.

6. The method of claim 4, further comprising:

receiving a new surface reconstruction data packet;

determining a predetermined time period has passed since receiving a previous surface reconstruction data packet; and

wherein, responsive thereto, the at least one feedback message is generated.

7. The method of claim 4, wherein the at least one feedback message is generated responsive to determining a feedback value indicating a last received packet has advanced by a determined number.

8. The method of claim 4, wherein the at least one feedback message is generated responsive to an error being reported when decoding the surface reconstruction data.

9. The method of claim 1, further comprising generating at least one feedback message for controlling bandwidth allocated to the video channel and the surface reconstruction channel.

10. The method of claim 9, further comprising sending the at least one feedback message to a device that communicated the received packets.

11. The method of claim 9, wherein the bandwidth allocated to the video channel and the surface reconstruction channel is determined by equally dividing an initial bandwidth between the surface reconstruction channel and the video channel.

12. A computer program product, the computer program product being embodied on a computer-readable storage medium and configured so as when executed on a processor of a protocol endpoint, to:

receive packets for a video channel defining a shared scene and a surface reconstruction channel associated with surface reconstruction data of image data, the image data as well as intrinsic and extrinsic capture device data being from an image capture device and being encoded and packetized within the video channel, the surface reconstruction data being encoded and packetized within the surface reconstruction channel;

extract the image data as well as the intrinsic and extrinsic capture device data from within the video channel;

extract the surface reconstruction data from within the surface reconstruction channel; and

render the shared scene using the image data, the intrinsic and extrinsic capture device data, and the surface reconstruction data.

13. The computer program product of claim 12, wherein to render the shared scene further comprises to display a three-dimensional (3D) shared scene.

14. The computer program product of claim 12, further comprising to generate at least one feedback message for controlling retransmission of the surface reconstruction data.

15. The computer program product of claim 14, wherein the at least one feedback message is generated responsive to an error being reported when decoding the surface reconstruction data.

16. The computer program product of claim 14, wherein the at least one feedback message is generated responsive to determining a feedback value indicating a last received packet has advanced by a determined number.

17. A system comprising:

a processor of a protocol endpoint; and

at least one computer-readable storage medium storing instructions, which, when executed by the processor, control the processor to perform operations comprising;

receiving packets for a video channel defining a shared scene and a surface reconstruction channel associated with surface reconstruction data of image data, the image data as well as intrinsic and extrinsic capture device data being from an image capture device and being encoded and packetized within the video channel, the surface reconstruction data being encoded and packetized within the surface reconstruction channel;

extracting the image data as well as the intrinsic and extrinsic capture device data from within the video channel;

extracting the surface reconstruction data from within the surface reconstruction channel; and

rendering the shared scene using the image data, the intrinsic and extrinsic capture device data, and the surface reconstruction data.

18. The system of claim 17, wherein the protocol endpoint comprises a user device of a participant of the shared scene who is not an owner of the shared scene.

19. The system of claim 17, wherein the protocol endpoint comprises a user device of a participant of the shared scene who is an owner of the shared scene.

20. The system of claim 17, further comprising generating at least one feedback message for controlling bandwidth allocated to the video channel and the surface reconstruction channel by changing an allocation of an initial bandwidth that equally divides the initial bandwidth between the surface reconstruction channel and the video channel.