STREAMING SPHERICAL VIDEO

• US 20160352791A1
• Filed: 05/27/2016
• Published: 12/01/2016
• Est. Priority Date: 05/27/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Adams, Riley, Suderman, Rob, Prouty, Jeff, Cower, Dillon Richard, Duvivier, Christian Luc, Scherkus, Andrew
• Granted Patent: US 10,880,346 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04L 65/61 for supporting one-way streaming services, e.g. Internet radio
• CPC Class Codes: H04L 65/764 at the destination reformatting of additional data in video clients H04N21/4355
• CPC Class Codes: H04L 9/40 Network security protocols
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content

Abstract

A method includes receiving a first video stream at a playback device, the first video stream having a first quality, receiving at least two second video streams at the playback device, the at least two second video streams each corresponding to a portion of the first video stream, the at least two second video streams having a second quality, the second quality being a higher quality as compared to the first quality, playing back the first video stream at the playback device, selecting a third video stream from the at least two second video streams based on a view perspective of a user of the playback device, and playing back the third video stream together with the first video stream at the playback device.

Claims

1. A method comprising:

receiving a first video stream at a playback device, the first video stream having a first quality;

receiving at least two second video streams at the playback device, the at least two second video streams each corresponding to a portion of the first video stream, the at least two second video streams having a second quality, the second quality being a higher quality as compared to the first quality;

playing back the first video stream at the playback device;

selecting a third video stream from the at least two second video streams based on a view perspective of a user of the playback device; and

playing back the third video stream together with the first video stream at the playback device.

2. The method of claim 1, further comprising:

determining the view perspective of the user of the playback device has changed;

selecting a fourth video stream from the at least two second video streams based on the changed view perspective; and

playing back the fourth video stream together with the first video stream at the playback device.

3. The method of claim 1, wherein

the first quality is based on bitrate and resolution,

the bitrate is based on a minimum communication channel capability, and

the resolution is based on a minimum pixel density associated with the playback device.

4. The method of claim 1, further comprising:

buffering the first video stream, wherein the playing back of the third video stream together with the first video stream at the playback device includes;

replacing a corresponding portion of the first video stream with the third video stream,

rendering the result on the playback device,

determining an issue exists with the third video stream, and

upon determining an issue exists with the third video stream, the buffered first video stream is rendered on the playback device.

5. The method of claim 1, wherein

the playback device is a head mount display (HMD), and

the playing back of the third video stream together with the first video stream at the playback device includes:

rendering the first video stream on one of a left eye or a right eye display of the HMD, and

rendering the third video stream on another of the left eye or the right eye display of the HMD.

6. The method of claim 1, wherein the at least two second video streams have associated keyframes encoded at different timestamps.

7. The method of claim 1, further comprising:

determining if each of the at least two second video streams is within a viewable threshold of the playback device;

upon determining the at least two second video streams is within the viewable threshold of the playback device, decode the at least two second video streams; and

upon determining the at least two second video streams is not within the viewable threshold of the playback device, skip decoding the at least two second video streams.

8. A method comprising:

encoding a frame of a video at a first quality;

encoding at least two portions of the frame of the video at a second quality, the second quality being a higher quality as compared to the first quality;

storing the frame of the video; and

storing the at least two portions of the frame of the video.

9. The method of claim 8, further comprising:

streaming the frame of the video; and

streaming the at least two portions of the frame of the video.

10. The method of claim 8, wherein the encoding of the at least two portions of the frame of the video at the second quality includes encoding each of the at least two portions of the frame of the video using keyframes having different timestamps.

11. The method of claim 8, further comprising:

receiving an indication of an amount of motion associated with a playback device; and

adapting a size associated with the at least two portions of the frame of the video based on the motion.

12. The method of claim 8, further comprising:

receiving a request to stream a video;

constructing a frame using the frame of the video and one of the at least two portions of the frame of the video; and

streaming the constructed frame.

13. The method of claim 8, wherein

the frame of the video is associated with a spherical video,

encoding the frame of the video includes:

projecting the frame of the video as a cubemap, and

encoding the cubemap using an equiangular sampling algorithm.

14. The method of claim 8, wherein the frame of the video is associated with a spherical video, the method further comprising:

projecting the frame of the video as a cubemap, and

selecting a cubeface of the cubemap, wherein

encoding the frame of the video includes encoding the cubeface at the first quality, and

encoding the at least two portions of the frame of the video includes encoding at least two portions of the the cubeface at the second quality.

15. The method of claim 8, further comprising:

streaming the frame of the video;

streaming the at least two portions of the frame of the video;

receiving an indication that playback on a playback device has paused;

encoding the frame of the video at the second quality; and

streaming the frame of the video encoded at the second quality.

16. The method of claim 8, wherein the first quality is based on at least one of a low frame-rate high resolution stream, a mono stream, a reduced color stream, and a black and white stream.

17. The method of claim 8, further comprising:

streaming the frame of the video as a plurality of encoded frames; and

dropping at least one frame of the plurality of encoded frames during the streaming of the video to achieve a third quality.

18. A streaming server comprising:

an encoder configured to;

encode a frame of a video at a first quality,

encode at least two portions of the frame of the video at a second quality,

the second quality being a higher quality as compared to the first quality;

memory configured to;

store the frame of the video, and

store the at least two portions of the frame of the video; and

a controller configured to;

stream a video including the encoded frame and one or more of the encoded at least two portions of the frame of the video.

19. The streaming server of claim 18, wherein the encoding of the at least two portions of the frame of the video at the second quality includes encoding each of the at least two portions of the frame of the video using keyframes having different timestamps.

20. The streaming server of claim 18, wherein the frame of the video is associated with a spherical video, the encoder further configured to:

project the frame of the video as a cubemap, and

select a cubeface of the cubemap, wherein

encoding the frame of the video includes encoding the cubeface at the first quality, and

encoding the at least two portions of the frame of the video includes encoding at least two portions of the the cubeface at the second quality.