MULTI-VIEW VIDEO COMPRESSION AND STREAMING

• US 20100329358A1
• Filed: 06/25/2009
• Published: 12/30/2010
• Est. Priority Date: 06/25/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Cha, Florencio, Dinei
• Granted Patent: US 9,648,346 B2
• Time in Patent Office: Days
• US Class Current: 375/240.26
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/187 the unit being a scalable video layer
• CPC Class Codes: H04N 19/54 using feature points or meshes
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

Multi-view video that is being streamed to a remote device in real time may be encoded. Frames of a real-world scene captured by respective video cameras are received for compression. A virtual viewpoint, positioned relative to the video cameras, is used to determine expected contributions of individual portions of the frames to a synthesized image of the scene from the viewpoint position using the frames. For each frame, compression rates for individual blocks of a frame are computed based on the determined contributions of the individual portions of the frame. The frames are compressed by compressing the blocks of the frames according to their respective determined compression rates. The frames are transmitted in compressed form via a network to a remote device, which is configured to render the scene using the compressed frames.

Claims

1. A computer-implemented method of real-time encoding of a multi-view video stream of a scene being captured by a plurality of video cameras, method comprising:

receiving the multi-view video stream, the multi-view video stream comprising a plurality of video streams, each stream comprising a series of video frames of a scene captured by a corresponding different video camera, and a macro-frame comprising a single frame from each video stream, the frames of a macro-frame having been captured substantially concurrently;

estimating a probability distribution of viewpoints;

compressing the multi-view video based on the probability distribution; and

transmitting the compressed multi-view video to a remote device that displays the multi-view video.

2. A computer-implemented method according to claim 1, wherein the probability distribution is based on a past position of a remote viewer of the video, indicia of which was received over the network.

3. A computer-implemented method according to claim 1, wherein the probability distribution comprises a single viewpoint corresponding to a prediction, by a device performing the encoding, of a position of a viewer at viewing time.

4. A computer-implemented method according to claim 1, wherein the viewpoint is based on an average viewer position relative to a display displaying the video.

5. A computer-implemented method according to claim 4, wherein the average viewer position includes content of the video, or the time of day, or the display displaying the video.

6. A computer-implemented method according to claim 1, further comprising:

encoding the multi-view video by, for each frame in a macro-frame of the multi-view video, varying compression rates of corresponding individual segments in the frame based on the probability distribution and based on positions of the video cameras, whereby segments in a same frame are compressed at varying computed compression rates.

7. A computer-implemented method according to claim 6, wherein the compression rates are further based on a depth map of the scene, wherein the compression rate of a segment depends in part upon whether the depth map indicates that the block contains a portion of the scene that is occluded in another frame.

8. One or more computer-readable storage media storing information to enable a computing device to perform a process, the process performed by one or more devices that together capture and transmit multi-view video, the method comprising:

receiving a plurality of multi-view video frames of a real world scene captured by respective cameras;

computing a probability distribution of a viewpoint; and

compressing the frames with compression rates based on the probability distribution.

9. One or more computer-readable storage media according to claim 8, wherein the viewpoint comprises a predicted position of a remote viewer of the frames when the frames are displayed to and viewed by the remote viewer.

10. One or more computer-readable storage media according to claim 9, wherein the predicted position is based on indicia of a past position of the remote viewer.

11. One or more computer-readable storage media according to claim 8, wherein the viewpoint is based on an average viewer position near a display displaying the multi-view video.

12. One or more computer-readable storage media according to claim 8, wherein the viewpoint is based on geometry of a physical area where the multi-view video is being displayed.

13. One or more computer-readable storage media according to claim 8, wherein the viewpoint is computed based on received information about a location of a remote viewer that will view the frames and based on information about time that will pass before the frames are viewed by the remote viewer.

14. One or more computer-readable storage media according to claim 13, wherein the information about the location of the remote viewer includes information indicating movement of the remote viewer.

15. One or more computer-readable storage media according to claim 8, further comprising computing a mesh from a depth map of the scene, using the depth map to identify potentially occluded pixels of the frames, and computing the weights of such pixels accordingly, the weights being used to control the compression ratio of corresponding segments of the frames.

16. A computer-implemented method of encoding multi-view video that is being streamed to a remote device in real time, the method performed by one or more devices that are processing the multi-view video before it is transmitted to the remote device, the method comprising:

receiving the multi-view video, which is comprised of streams of a real-world scene captured by respective video cameras;

computing a probability distribution of a position of a virtual viewpoint relative to the video cameras; and

encoding the streams according to the probability distribution.

17. A computer-implemented method according to claim 16, wherein the virtual viewpoint is based on either a past position of a remote viewer of the, a layout of a room where the multi-view video is being displayed, or a network latency.

18. A computer-implemented method according to claim 16, further comprising determining compression rates for individual portions of frames of the multi-view video based on the probability distribution.

19. A computer-implemented method according to claim 18, wherein the compression is performed using H.264 to compress portions of a frame at different compression rates.

20. A computer-implemented method according to claim 16 wherein given a three-dimensional model of the scene, the encoding is based on a projection of the model to the virtual viewpoint.