Encoding and decoding virtual reality video

• US 10,469,873 B2
• Filed: 10/11/2017
• Issued: 11/05/2019
• Est. Priority Date: 04/15/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Pang, Derek, Pitts, Colvin, Akeley, Kurt, Htet, Zeyar
• Primary Examiner(s): Du, Haixia
• Application Number:
  US15/730,096
• Publication Number:
  US 20180035134A1
• Time in Patent Office: 755 Days
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/91 Entropy coding, e.g. variable length coding [VLC] or arithmetic coding
• CPC Class Codes: H04N 19/93 Run-length coding
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 23/957 : Light-field or plenoptic cameras or camera modules
• CPC Class Codes: H04N 5/2226 : Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10

Abstract

A virtual reality or augmented reality experience of a scene may be decoded for playback for a viewer through a combination of CPU and GPU processing. A video stream may be retrieved from a data store. A first viewer position and/or orientation may be received from an input device, such as the sensor package on a head-mounted display (HMD). At a processor, the video stream may be partially decoded to generate a partially-decoded bitstream. At a graphics processor, the partially-decoded bitstream may be further decoded to generate viewpoint video of the scene from a first virtual viewpoint corresponding to the first viewer position and/or orientation. The viewpoint video may be displayed on a display device, such as screen of the HMD.

Claims

1. A method for decoding a video stream of a scene for a virtual reality or augmented reality experience, the method comprising:

from a data store, retrieving at least part of the video stream;

at an input device, receiving a first viewer position and/or orientation;

at a processor, partially decoding the video stream to generate a partially-decoded bitstream;

at a graphics processor, further decoding the partially-decoded bitstream to generate viewpoint video of the scene from a first virtual viewpoint corresponding to the first viewer position and/or orientation; and

at a display device, displaying the viewpoint video, wherein a handoff of the partially-decoded bitstream from the processor to the graphics processor is based upon a scanline iteration, the scanline iteration being indicative of a manner in which storage occurs in the graphics processor.

2. The method of claim 1, wherein:

the video stream comprises a plurality of vantages, each of which comprises a view of the scene from one of a plurality of virtual viewpoints within a viewing volume; and

each of the vantages comprises a plurality of tiles, each of which depicts the scene from the virtual viewpoint associated with the vantage, within a limited field of view.

3. The method of claim 2, wherein partially decoding the video stream comprises partially decoding a plurality of the tiles in parallel.

4. The method of claim 2, wherein partially decoding the video stream comprises:

dividing each of the tiles into a plurality of macroblocks; and

removing serial dependencies from each of the tiles and/or macroblocks.

5. The method of claim 1, wherein further decoding the partially-decoded bitstream comprises storing the partially-decoded bitstream in a graphics processor memory of the graphics processor in a manner that reduces access latency for the graphics processor memory.

6. The method of claim 1, wherein:

the graphics processor is configured for multi-threaded operation; and

further decoding the partially-decoded bitstream comprises dividing the partially-decoded bitstream into a plurality of thread-groups, each of which can be processed by a thread of the graphics processor.

7. The method of claim 6, wherein further decoding the partially-decoded bitstream further comprises synchronizing decoding between at least some of the thread-groups.

8. The method of claim 1, wherein:

the video stream comprises a depth channel and one or more color channels;

at least one of partially decoding the video stream and further decoding the partially-decoded bitstream comprises;

decoding one or more color channels; and

decoding the depth channel.

9. The method of claim 1, wherein partially decoding the video stream comprises performing at least one selection from the group consisting of:

performing Huffman decoding on the video stream;

performing inverse run length encoding on the video stream; and

performing de-zigzagging on the video stream.

10. The method of claim 1, wherein further decoding the partially-decoded bitstream comprises performing at least one selection from the group consisting of:

performing de-quantization on the partially-decoded bitstream;

performing inverse discrete cosine transformation on the partially-decoded bitstream; and

performing color conversion on the partially-decoded bitstream.

11. A method for encoding a volumetric video data of a scene for a virtual reality or augmented reality experience, the method comprising:

from a data store, retrieving the volumetric video data; and

at a processor, encoding the volumetric video data to generate a video stream configured to facilitate decoding via a decoding method comprising the steps of;

at a processor, partially decoding the video stream to generate a partially-decoded bitstream; and

at a graphics processor, further decoding the partially-decoded bitstream to generate viewpoint video of the scene from a virtual viewpoint corresponding to a viewer-designated viewer position and/or orientation, wherein a handoff of the partially-decoded bitstream from the processor to the graphics processor is based upon a scanline iteration, the scanline iteration being indicative of a manner in which storage occurs in the graphics processor.

12. The method of claim 11, wherein:

the video stream comprises a plurality of vantages, each of which comprises a view of the scene from one of a plurality of virtual viewpoints within a viewing volume; and

each of the vantages comprises a plurality of tiles, each of which depicts the scene from the virtual viewpoint associated with the vantage, within a limited field of view.

13. A non-transitory computer-readable medium for decoding a video stream of a scene for a virtual reality or augmented reality experience, comprising instructions stored thereon, that when executed by one or more processors, perform the steps of:

causing a data store to retrieve at least part of the video stream;

causing an input device to receive a first viewer position and/or orientation;

partially decoding the video stream to generate a partially-decoded bitstream;

causing a graphics processor to further decode the partially-decoded bitstream to generate viewpoint video of the scene from a first virtual viewpoint corresponding to the first viewer position and/or orientation; and

causing a display device to display the viewpoint video, wherein a handoff of the partially-decoded bitstream from the processor to the graphics processor is based upon a scanline iteration, the scanline iteration being indicative of a manner in which storage occurs in the graphics processor.

14. The non-transitory computer-readable medium of claim 13, wherein:

the video stream comprises a plurality of vantages, each of which comprises a view of the scene from one of a plurality of virtual viewpoints within a viewing volume; and

each of the vantages comprises a plurality of tiles, each of which depicts the scene from the virtual viewpoint associated with the vantage, within a limited field of view.

15. The non-transitory computer-readable medium of claim 13, wherein further decoding the partially-decoded bitstream comprises storing the partially-decoded bitstream in a graphics processor memory of the graphics processor in a manner that reduces access latency for the graphics processor memory.

16. The non-transitory computer-readable medium of claim 13, wherein:

the graphics processor is configured for multi-threaded operation;

further decoding the partially-decoded bitstream comprises dividing the partially-decoded bitstream into a plurality of thread-groups, each of which can be processed by a thread of the graphics processor; and

further decoding the partially-decoded bitstream further comprises synchronizing decoding between at least some of the thread-groups.

17. The non-transitory computer-readable medium of claim 13, wherein:

the video stream comprises a depth channel and one or more color channels;

at least one of partially decoding the video stream and further decoding the partially-decoded bitstream comprises;

decoding one or more color channels; and

decoding the depth channel.

18. The non-transitory computer-readable medium of claim 13, wherein partially decoding the video stream comprises performing at least one selection from the group consisting of:

performing Huffman decoding on the video stream;

performing inverse run length encoding on the video stream; and

performing de-zigzagging on the video stream.

19. The non-transitory computer-readable medium of claim 13, wherein further decoding the partially-decoded bitstream comprises performing at least one selection from the group consisting of:

performing de-quantization on the partially-decoded bitstream;

performing inverse discrete cosine transformation on the partially-decoded bitstream; and

performing color conversion on the partially-decoded bitstream.

20. A non-transitory computer-readable medium for encoding a volumetric video data of a scene for a virtual reality or augmented reality experience, comprising instructions stored thereon, that when executed by one or more processors, perform the steps of:

causing a data store to retrieve the volumetric video data; and

encoding the volumetric video data to generate a video stream configured to facilitate decoding via a decoding method comprising the steps of;

at a processor, partially decoding the video stream to generate a partially-decoded bitstream; and

at a graphics processor, further decoding the partially-decoded bitstream to generate viewpoint video of the scene from a virtual viewpoint corresponding to a viewer-designated viewer position and/or orientation, wherein a handoff of the partially-decoded bitstream from the processor to the graphics processor is based upon a scanline iteration, the scanline iteration being indicative of a manner in which storage occurs in the graphics processor.

21. The non-transitory computer-readable medium of claim 20, wherein:

the video stream comprises a plurality of vantages, each of which comprises a view of the scene from one of a plurality of virtual viewpoints within a viewing volume; and

each of the vantages comprises a plurality of tiles, each of which depicts the scene from the virtual viewpoint associated with the vantage, within a limited field of view.

22. A system for decoding a video stream of a scene for a virtual reality or augmented reality experience, the system comprising:

a data store configured to retrieve at least part of the video stream;

an input device configured to receive a first viewer position and/or orientation;

a processor configured to partially decode the video stream to generate a partially-decoded bitstream;

a graphics processor configured to further decode the partially-decoded bitstream to generate viewpoint video of the scene from a first virtual viewpoint corresponding to the first viewer position and/or orientation; and

a display device configured to display the viewpoint video, wherein a handoff of the partially-decoded bitstream from the processor to the graphics processor is based upon a scanline iteration, the scanline iteration being indicative of a manner in which storage occurs in the graphics processor.

23. The system of claim 22, wherein:

the video stream comprises a plurality of vantages, each of which comprises a view of the scene from one of a plurality of virtual viewpoints within a viewing volume; and

each of the vantages comprises a plurality of tiles, each of which depicts the scene from the virtual viewpoint associated with the vantage, within a limited field of view.

24. The system of claim 22, wherein the graphics processor is configured to further decode the partially-decoded bitstream by storing the partially-decoded bitstream in a graphics processor memory of the graphics processor in a manner that reduces access latency for the graphics processor memory.

25. The system of claim 22, wherein:

the graphics processor is configured for multi-threaded operation;

the graphics processor is configured to further decode the partially-decoded bitstream by;

dividing the partially-decoded bitstream into a plurality of thread-groups, each of which can be processed by a thread of the graphics processor; and

synchronizing decoding between at least some of the thread-groups.

26. The system of claim 22, wherein:

the video stream comprises a depth channel and one or more color channels;

at least one of the processor and the graphics processor is configured to partially decode the video stream or further decode the partially-decoded bitstream by;

decoding one or more color channels; and

decoding the depth channel.

27. The system of claim 22, wherein the processor is configured to partially decode the video stream by performing at least one selection from the group consisting of:

performing Huffman decoding on the video stream;

performing inverse run length encoding on the video stream; and

performing de-zigzagging on the video stream.

28. The system of claim 22, wherein the graphics processor is configured to further decode the partially-decoded bitstream by performing at least one selection from the group consisting of:

performing de-quantization on the partially-decoded bitstream;

performing inverse discrete cosine transformation on the partially-decoded bitstream; and

performing color conversion on the partially-decoded bitstream.

29. A system for encoding a volumetric video data of a scene for a virtual reality or augmented reality experience, the system comprising:

a data store configured to retrieve the volumetric video data; and

a first processor configured to encode the volumetric video data to generate a video stream configured to facilitate decoding via a decoding method comprising the steps of;

at a second processor, partially decoding the video stream to generate a partially-decoded bitstream; and

at a graphics processor, further decoding the partially-decoded bitstream to generate viewpoint video of the scene from a virtual viewpoint corresponding to a viewer-designated viewer position and/or orientation, wherein a handoff of the partially-decoded bitstream from the processor to the graphics processor is based upon a scanline iteration, the scanline iteration being indicative of a manner in which storage occurs in the graphics processor.

30. The system of claim 29, wherein:

the video stream comprises a plurality of vantages, each of which comprises a view of the scene from one of a plurality of virtual viewpoints within a viewing volume; and

each of the vantages comprises a plurality of tiles, each of which depicts the scene from the virtual viewpoint associated with the vantage, within a limited field of view.