Transmission of three-dimensional video

• US 10,547,825 B2
• Filed: 09/16/2015
• Issued: 01/28/2020
• Est. Priority Date: 09/22/2014
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Sadi, Sajid, Perdices-Gonzalez, Sergio, Budhiraja, Rahul, Lee, Brian Dongwoo, Khwaja, Ayesha Mudassir, Mistry, Pranav
• Primary Examiner(s): Suh, Joseph
• Application Number:
  US14/856,511
• Publication Number:
  US 20160088282A1
• Time in Patent Office: 1,595 Days
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G03B 35/08 by simultaneous recording
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 19/54 using feature points or meshes
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/265 Mixing

Abstract

In one embodiment, a method includes receiving, from a client device, a request corresponding to a particular view of a three-dimensional video comprising multiple views. The method also includes accessing a data stream corresponding to the video. The method further includes selecting, based on the request, one or more portions of the accessed data stream, at least one of the selected portions corresponding to the particular view. The method also includes transmitting the selected portion of the accessed data stream to the client device.

Claims

1. A method comprising:

receiving, from a client device, a request to provide for display on the client device during a first time period a particular view of a three-dimensional video comprising a plurality of views;

accessing a data stream corresponding to the video;

selecting, based on the request, a plurality of portions of the accessed data stream to transmit to the client device, wherein the plurality of selected portions comprise;

a high-resolution video of the particular view of the three-dimensional video for display during the first time period; and

a low-resolution video of one or more views of the three-dimensional video that are adjacent to but outside the particular view, wherein the low-resolution video is not for display on the client device during the first time period; and

transmitting to the client device a content container comprising the selected portions of the accessed data stream such that the low-resolution video of the one or more views of the three-dimensional video that are adjacent to but outside the particular view is transmitted to the client device before the high-resolution video of the particular view is displayed on the client device.

2. The method of claim 1, wherein the received request comprises instructions for transmitting one or more of the portions of the accessed data stream at a specified bit rate or omitting one or more portions of the accessed data stream.

3. The method of claim 1, wherein the received request comprises instructions for transforming the accessed data stream.

4. The method of claim 3, wherein the transformation comprises transcoding or resizing the accessed data stream.

5. The method of claim 3, further comprising storing the results of a transformed data stream.

6. The method of claim 1, further comprising:

decoding the accessed data stream; and

extracting each portion of the selected data stream from the decoded data stream.

7. The method of claim 6, wherein:

the accessed data stream comprises a plurality of data streams; and

each of the plurality of data streams comprises a particular one of the extracted portions.

8. The method of claim 1, wherein the video comprises a video stream with multiple resolutions, a video stream from a single camera, a manipulated video stream, or a video stream from a top camera.

9. The method of claim 1, wherein the accessed data stream comprises a 360°

stereoscopic three-dimensional (3-D) video.

10. The method of claim 1, wherein the accessed data stream comprises one or more audio streams that are time synchronized to the video.

11. The method of claim 1, wherein the accessed data stream comprises data to reconstruct the video on the client device.

12. The method of claim 11, wherein the reconstruction data comprises spatial filtering data, distortion data, correction information, or projection parameters of the video.

13. The method of claim 11, wherein the reconstruction data comprises audio spatial reconstruction data.

14. The method of claim 1, wherein when the low-resolution video of one or more views of the three-dimensional video that are adjacent to but outside the particular view is displayed on the client device during a second time period, a resolution of the low-resolution video increases proportionally to an amount of time a view corresponding to the low-resolution video is stationary.

15. The method of claim 1, further comprising

inferring a change in the particular view;

and

selecting, based on the inferred change, one or more portions of the accessed data stream corresponding to the inferred change.

16. The method of claim 1, wherein transmitting the selected portions comprises adjusting a bit rate of the selected portions based on a detected bandwidth of a connection with the client device.

17. The method of claim 1, further comprising selecting one or more of the plurality of portions of the accessed data stream based on the capabilities of the client device.

18. The method of claim 1, further comprising receiving, from the client device, a subsequent request for a high-resolution video of a particular one of the one or more views of the three-dimensional video that are adjacent to but outside the particular view, wherein the request is generated based on a determination that the client device is displaying the particular one of the one or more views of the three-dimensional video that are adjacent to but outside the particular view during a second time period that is after the first time period.

19. The method of claim 1, wherein:

the particular view of the three-dimensional video comprises at least one left-eye image of the view at a particular time and one right-eye image of the view at the particular time;

the data stream comprises the left-eye image or the right-eye image and a description of a plurality of spatial offsets between the left-eye image and the right-eye image; and

transmitting the selected portion of the accessed data stream to the client device comprises transmitting the left-eye image or the right-eye image and the spatial offsets to the client device.

20. The method of claim 19, wherein the particular view of three-dimensional video further comprises at least one left-eye image of the view at a subsequent time and one right-eye image of the view at the subsequent time;

the data stream further comprises a plurality of time-dependent spatial offsets between;

the left-eye image at the particular time and the left-eye image at the subsequent time;

or

the right-eye image at the particular time and the right-eye image at the subsequent time; and

transmitting the selected portion of the accessed data stream to the client device further comprises transmitting the left-eye image at the subsequent time or the right-eye image at the subsequent time and the plurality of time-dependent spatial offsets.

21. One or more non-transitory computer-readable storage media embodying instructions that are operable when executed to:

receive, from a client device, a request to provide for display on the client device during a first time period a particular view of a three-dimensional video comprising a plurality of views;

access a data stream corresponding to the video;

select, based on the request, a plurality of portions of the accessed data stream to transmit to the client device, wherein the plurality of selected portions comprise;

a high-resolution video of the particular view of the three-dimensional video for display during the first time period; and

a low-resolution video of one or more views of the three-dimensional video that are adjacent to but outside the particular view, wherein the low-resolution video is not for display on the client device during the first time period; and

transmit to the client device a content container comprising the selected portions of the accessed data stream such that the low-resolution video of the one or more views of the three-dimensional video that are adjacent to but outside the particular view is transmitted to the client device before the high-resolution video of the particular view is displayed on the client device.

22. The media of claim 21, wherein the received request comprises instructions for transmitting one or more of the portions of the accessed data stream at a specified bit rate or omitting one or more portions of the accessed data stream.

23. The media of claim 21, wherein the received request comprises instructions for transforming the accessed data stream.

24. The media of claim 21, wherein the instructions are further operable when executed to:

decode the accessed data stream; and

extract each portion of the accessed data stream from the decoded data stream.

25. The media of claim 21, wherein the video comprises a video stream with multiple resolutions, a video stream from a single camera, a manipulated video stream, or a video stream from a top camera.

26. The media of claim 21, wherein the instructions are further operable when executed to:

infer a change in the particular view; and

select, based on the inferred change, one or more portions of the accessed data stream corresponding to the inferred change.

27. An apparatus comprising:

one or more non-transitory computer-readable storage media embodying instructions; and

one or more processors coupled to the storage media and configured to execute the instructions to;

receive, from a client device, a request to provide for display on the client device during a first time period a particular view of a three-dimensional video comprising a plurality of views;

access a data stream corresponding to the video;

select, based on the request, a plurality of portions of the accessed data stream to transmit to the client device, wherein the plurality of selected portions comprise;

a high-resolution video of the particular view of the three-dimensional video for display during the first time period; and

a low-resolution video of one or more views of the three-dimensional video that are adjacent to but outside the particular view, wherein the low-resolution video is not for display on the client device during the first time period; and

transmit to the client device a content container comprising the selected portions of the accessed data stream such that the low-resolution video of the one or more views of the three-dimensional video that are adjacent to but outside the particular view is transmitted to the client device before the high-resolution video of the particular view is displayed on the client device.

28. The apparatus of claim 27, wherein the received request comprises instructions for transmitting one or more of the portions of the accessed data stream at a specified bit rate or omitting one or more portions of the accessed data stream.

29. The apparatus of claim 27, wherein the received request comprises instructions for transforming the accessed data stream.

30. The apparatus of claim 27, wherein the processors are further configured to execute the instructions to:

decode the accessed data stream; and

extract each portion of the accessed data stream from the decoded data stream.

31. The apparatus of claim 27, wherein the video comprises a video stream with multiple resolutions, a video stream from a single camera, a manipulated video stream, or a video stream from a top camera.

32. The apparatus of claim 27, wherein the processors are further configured to execute the instructions to:

infer a change in the particular view; and

select, based on the inferred change, one or more portions of the accessed data stream corresponding to the inferred change.