Shared Scene Mesh Data Synchronisation

• US 20160212468A1
• Filed: 01/21/2015
• Published: 07/21/2016
• Est. Priority Date: 01/21/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hsu, Bofan, Kozica, Ermin, Lee, Ming-Chieh, Lu, Mei-Hsuan, Aichner, Robert, Menezes, Ryan S., Abhishek, Abhishek
• Granted Patent: US 10,104,415 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04N 13/00 Stereoscopic video systems; Multi-view video systems; Details thereof
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 21/23614 Multiplexing of additional data and video streams arrangements for simultaneous broadcast of plural pieces of information H04H20/28
• CPC Class Codes: H04N 21/2383 Channel coding or modulation of digital bit-stream, e.g. QPSK modulation arrangements for detecting or preventing errors in the information received by adapting the channel coding H04L1/0009; analogue front ends or means for connecting modulators, demodulators or transceivers to a transmission line H04L27/0002
• CPC Class Codes: H04N 21/2668 Creating a channel for a dedicated end-user group, e.g. insertion of targeted commercials based on end-user profiles information retrieval from the Internet by querying with filtering and personalisation G06F16/9535; arrangements for replacing or switching information during the broadcast H04H20/10; push services over packet-switching network H04L12/1859; adaptation of message content in packet-switching networks H04L51/063
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4382 Demodulation or channel decoding, e.g. QPSK demodulation analog front ends or means for connecting modulators, demodulators or transceivers to a transmission line H04L27/0002
• CPC Class Codes: H04N 21/6125 involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 5/44 Receiver circuitry for the reception of television signals according to analogue transmission standards H04N5/14 takes precedence
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

A user device within a communication architecture, the user device comprising: an image capture device configured to determine image data for the creation of a video channel defining the shared scene; an intrinsic/extrinsic data determiner configured to determine intrinsic/extrinsic capture device data associated with the image capture device; and a video encoder configured to encode the image data and intrinsic/extrinsic capture device data within the video channel.

Claims

1. A user device within a communication architecture, the user device comprising:

an image capture device configured to determine image data for the creation of a video channel defining the shared scene;

an intrinsic/extrinsic data determiner configured to determine intrinsic/extrinsic capture device data associated with the image capture device; and

a video encoder configured to encode the image data and intrinsic/extrinsic capture device data within the video channel.

2. The user device of claim 1, wherein the video encoder comprises:

a frame encoder configured to encode the image data as image frames; and

a supplemental enhancement information appender configured to encode the intrinsic/extrinsic capture device data as a supplemental enhancement information message appended to the image frames.

3. The user device of claim 1, wherein the image capture device comprises:

a camera input configured to receive RGB frame data; and

a media capture entity configured to generate image data from the RGB frame data.

4. The user device of claim 3, wherein the media capture entity is further configured to receive the intrinsic/extrinsic capture device data and associate the intrinsic/extrinsic capture device data with the image data.

5. The user device of claim 1, wherein the intrinsic/extrinsic capture device data comprises camera projection matrix data.

6. The user device of claim 1, wherein the intrinsic/extrinsic capture device data comprises camera pose data.

7. A user device within a communication architecture, the user device comprising:

a receiver configured to receive a video channel comprising image data for the shared scene determined from an image capture device and intrinsic/extrinsic capture device data associated with the image capture device; and

a video decoder configured to extract and decode the image data and the intrinsic/extrinsic capture device data associated with the image capture device from the video channel.

8. The user device of claim 7, wherein the video channel comprises a frame encoded video data stream, wherein the video decoder comprises:

a frame decoder configured to frame decode the encoded video data stream; and

a supplemental enhancement information extractor configured to extract and decode supplemental enhancement information messages from within the encoded video data stream.

9. The user device of claim 8, wherein the supplemental enhancement information extractor comprises:

a scanner configured to proxy scan incoming network abstraction layer units; and

an extractor configured to extract intrinsic/extrinsic capture device data from the supplemental enhancement information message appended with each frame.

10. A method implemented at a user device within a communication architecture, the method comprising:

determining image data from an image capture device for the creation of a video channel defining the shared scene;

determining intrinsic/extrinsic capture device data associated with the image capture device; and

encoding the image data and intrinsic/extrinsic capture device data within the video channel.

11. The method of claim 10, wherein encoding the image data and intrinsic/extrinsic capture device data within the video channel comprises:

encoding the image data as image frames; and

encoding the intrinsic/extrinsic capture device data as a supplemental enhancement information message appended to the image frames.

12. The method of claim 10, wherein determining image data from an image capture device for the creation of a video channel defining the shared scene comprises:

receiving RGB frame data; and

generating image data from the RGB frame data.

13. The method of claim 12, wherein determining intrinsic/extrinsic capture device data associated with the image capture device comprises receiving the intrinsic/extrinsic capture device data and associating the intrinsic/extrinsic capture device data with the image data.

14. The method of claim 10, wherein the intrinsic/extrinsic capture device data comprises camera projection matrix data.

15. The method of claim 10, wherein the intrinsic/extrinsic capture device data comprises camera pose data.

16. A method implemented at a user device within a communication architecture, the method comprising:

receiving a video channel comprising image data for the shared scene determined from an image capture device and intrinsic/extrinsic capture device data associated with the image capture device;

extracting and decoding the image data from the video channel; and

extracting the intrinsic/extrinsic capture device data associated with the image capture device.

17. The method of claim 16, wherein the video channel comprises a frame encoded video data stream, wherein extracting and decoding the image data from the video channel comprises frame decoding the encoded video data stream, and extracting the intrinsic/extrinsic capture device data associated with the image capture device comprises extracting and decoding supplemental enhancement information messages from within the encoded video data stream.

18. The method of claim 17, wherein extracting and decoding supplemental enhancement information messages from within the encoded video data stream comprises:

scanning incoming network abstraction layer units; and

extracting intrinsic/extrinsic capture device data from the supplemental enhancement information message appended with each frame.

19. A computer program product, the computer program product being embodied on a computer-readable medium and configured so as when executed on a processor of a user device within a communication architecture, to:

determine image data from an image capture device for the creation of a video channel defining the shared scene;

determine intrinsic/extrinsic capture device data associated with the image capture device; and

encode the image data and intrinsic/extrinsic capture device data within the video channel.

20. A computer program product, the computer program product being embodied on a computer-readable medium and configured so as when executed on a processor of a user device within a communication architecture, to:

receive a video channel comprising image data for the shared scene determined from an image capture device and intrinsic/extrinsic capture device data associated with the image capture device;

extract and decode the image data from the video channel; and

extract the intrinsic/extrinsic capture device data associated with the image capture device.