Data structures and delivery methods for expediting virtual reality playback

• US 10,419,737 B2
• Filed: 09/15/2017
• Issued: 09/17/2019
• Est. Priority Date: 04/15/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Pang, Derek, Pitts, Colvin, Akeley, Kurt
• Primary Examiner(s): Du, Haixia
• Application Number:
  US15/706,343
• Publication Number:
  US 20180020204A1
• Time in Patent Office: 732 Days
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/178 Metadata, e.g. disparity information
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 19/147 according to rate distortion criteria rate-distortion as a criterion for motion estimation H04N19/567
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/42 characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation H04N19/635 takes precedence
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/895 in combination with error concealment
• CPC Class Codes: H04N 2213/003 Aspects relating to the "2D+depth" image format
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 23/957 : Light-field or plenoptic cameras or camera modules
• CPC Class Codes: H04N 5/2226 : Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10

Abstract

A video stream for a scene for a virtual reality or augmented reality experience may be stored and delivered to a viewer. The video stream may be divided into a plurality of units based on time segmentation, viewpoint segmentation, and/or view orientation segmentation. Each of the units may be divided into a plurality of sub-units based on a different segmentation from the units, via time segmentation, viewpoint segmentation, and/or view orientation segmentation. At least a portion of the video stream may be stored in a file that includes a plurality of the units. Each unit may be a group of pictures that is a sequence of successive frames in time. Each sub-unit may be a vantage defining a viewpoint from which the scene is viewable. Each vantage may be further divided into tiles, each of which is part of the vantage, limited to one or more particular view orientations.

Claims

1. A method for storing a video stream of a scene for a virtual reality or augmented reality experience, the method comprising:

at one or more image capture devices, capturing the video stream;

at a processor, dividing the video stream into a plurality of units based on at least a first selection from the group consisting of time segmentation, viewpoint segmentation, and view orientation segmentation, wherein dividing the video stream into the plurality of units comprises dividing the video stream into the plurality of units based on time segmentation and each of the units comprises a group of pictures comprising a sequence of successive frames in time;

at the processor, dividing each of the units into a plurality of sub-units, based on at least a second selection, different from the first selection, from the group consisting of time segmentation, viewpoint segmentation, and view orientation segmentation, wherein dividing each of the units into the plurality of sub-units comprises dividing each of the groups of pictures into the plurality of sub-units based on viewpoint segmentation and each of the sub-units comprises a vantage defining a viewpoint from which the scene is viewable;

at a data store, storing at least a portion of the video stream in a file comprising a plurality of the units;

at the processor, based on a viewer position and/or orientation, identifying a subset of the file;

from the data store, retrieving the subset of the file without retrieving a remainder of the file;

at the processor, using the subset to generate a view of the scene from the viewer position and/or orientation;

on a display device, displaying the view; and

dividing each of the vantages into a plurality of sub-sub-units, based on at least a third selection, different from the first selection and the second selection, from the group consisting of time segmentation, viewpoint segmentation, and view orientation segmentation.

2. The method of claim 1, further comprising:

at the processor, generating a unit offset table indicating a start address, within the data store, of each of the units; and

at the data store, storing the unit offset table in the file.

3. The method of claim 2, further comprising:

at the processor, generating a sub-unit offset table indicating a start address, within the data store, of each of the sub-units; and

at the data store, storing the sub-unit offset table in the file.

4. The method of claim 1, wherein:

dividing each of the vantages into the plurality of sub-sub-units comprises dividing each of the groups of pictures into the plurality of sub-units based on view orientation segmentation; and

each of the sub-sub-units comprises a tile comprising part of one of the vantages, limited to one or more particular view orientations along which the scene is viewable from the one of the vantages.

5. The method of claim 4, further comprising dividing each of the vantages into frames;

wherein dividing each of the vantages into a plurality of sub-sub-units comprises dividing each of the frames into the tiles.

6. The method of claim 4, further comprising:

at the processor, generating a unit offset table indicating a start address, within the data store, of each of the groups of pictures;

at the processor, generating a sub-unit offset table indicating a start address, within the data store, of each of the vantages;

at the processor, generating a sub-sub-unit offset table indicating a start address, within the data store, of each of the tiles; and

at the data store, storing the unit offset table, the sub-unit offset table, and the sub-sub-unit offset table in the file.

7. The method of claim 4, further comprising identifying at least one duplicate tile of the tiles that is substantially duplicative of a base tile of the tiles;

wherein storing at least the portion of the video stream in the file comprises storing, in place of the duplicate tile, a start address, within the data store, at which the base tile is stored.

8. The method of claim 1, further comprising, at the data store, storing captured scene data comprising at least one of:

scene configuration information pertinent to the scene;

rendering information defining how the video stream is to be rendered for a viewer; and

vantage parameters pertinent to a plurality of vantages, each of which defines a viewpoint from which the scene is viewable.

9. The method of claim 1, further comprising, at the data store, storing content parameters comprising at least one of:

gray room transition defining how transitions in viewpoints are handled during viewing of the video stream;

viewing volume behavior of a viewing volume from which the scene is viewable; and

viewing volume rotation and/or offset defining rotation and/or position of the viewing volume.

10. The method of claim 1, further comprising:

at the data store, storing at least a second portion of the video stream in a second file comprising a second plurality of the units; and

at the processor, combining the file with the second file to generate a combined file comprising the portion and the second portion of the video stream.

11. A non-transitory computer-readable medium for storing a video stream of a scene for a virtual reality or augmented reality experience, comprising instructions stored thereon, that when executed by one or more processors, perform the steps of:

causing a data store to retrieve a video stream captured by one or more image capture devices;

dividing the video stream into a plurality of units based on at least a first selection from the group consisting of time segmentation, viewpoint segmentation, and view orientation segmentation, wherein dividing the video stream into the plurality of units comprises dividing the video stream into the plurality of units based on time segmentation and each of the units comprises a group of pictures comprising a sequence of successive frames in time;

dividing each of the units into a plurality of sub-units, based on at least a second selection, different from the first selection, from the group consisting of time segmentation, viewpoint segmentation, and view orientation segmentation, wherein dividing each of the units into the plurality of sub-units comprises dividing each of the groups of pictures into the plurality of sub-units based on viewpoint segmentation, and each of the sub-units comprises a vantage defining a viewpoint from which the scene is viewable;

causing the data store to store at least a portion of the video stream in a file comprising a plurality of the units;

based on a viewer position and/or orientation, identifying a subset of the file;

retrieving the subset of the file without retrieving a remainder of the file;

using the subset to generate a view of the scene from the viewer position and/or orientation; and

displaying the view on a display device, wherein;

the non-transitory computer-readable medium further comprises instructions stored thereon, that when executed by one or more processors, divide each of the vantages into a plurality of sub-sub-units, based on at least a third selection, different from the first selection and the second selection, from the group consisting of time segmentation, viewpoint segmentation, and view orientation segmentation;

dividing each of the vantages into the plurality of sub-sub-units comprises dividing each of the groups of pictures into the plurality of sub-units based on view orientation segmentation; and

each of the sub-sub-units comprises a tile comprising part of one of the vantages, limited to one or more particular view orientations along which the scene is viewable from the one of the vantages.

12. The non-transitory computer-readable medium of claim 11, further comprising instructions stored thereon, that when executed by one or more processors, perform the steps of:

generating a unit offset table indicating a start address, within the data store, of each of the units;

causing the data store to store the unit offset table in the file;

generating a sub-unit offset table indicating a start address, within the data store, of each of the sub-units; and

causing the data store to store the sub-unit offset table in the file.

13. The non-transitory computer-readable medium of claim 11, further comprising instructions stored thereon, that when executed by one or more processors, divide each of the vantages into frames;

wherein dividing each of the vantages into a plurality of sub-sub-units comprises dividing each of the frames into the tiles.

14. The non-transitory computer-readable medium of claim 11, further comprising instructions stored thereon, that when executed by one or more processors, perform the steps of:

generating a unit offset table indicating a start address, within the data store, of each of the groups of pictures;

generating a sub-unit offset table indicating a start address, within the data store, of each of the vantages; and

generating a sub-sub-unit offset table indicating a start address, within the data store, of each of the tiles; and

causing the data store to store the unit offset table, the sub-unit offset table, and the sub-sub-unit offset table in the file.

15. The non-transitory computer-readable medium of claim 11, further comprising instructions stored thereon, that when executed by one or more processors, identify at least one duplicate tile of the tiles that is substantially duplicative of a base tile of the tiles;

wherein storing at least the portion of the video stream in the file comprises storing, in place of the duplicate tile, a start address, within the data store, at which the base tile is stored.

16. The non-transitory computer-readable medium of claim 11, further comprising instructions stored thereon, that when executed by one or more processors, perform the steps of:

causing the data store to store at least a second portion of the video stream in a second file comprising a second plurality of the units; and

combining the file with the second file to generate a combined file comprising the portion and the second portion of the video stream.

17. A system for storing a video stream of a scene for a virtual reality or augmented reality experience, the system comprising:

a processor configured to;

divide a video stream captured by one or more image capture devices into a plurality of units based on at least a first selection from the group consisting of time segmentation, viewpoint segmentation, and view orientation segmentation; and

divide each of the units into a plurality of sub-units, based on at least a second selection, different from the first selection, from the group consisting of time segmentation, viewpoint segmentation, and view orientation segmentation;

a data store configured to store at least a portion of the video stream in a file comprising a plurality of the units; and

a display device;

wherein the processor is further configured to;

based on a viewer position and/or orientation, identify a subset of the file;

retrieve the subset of the file without retrieving a remainder of the file; and

use the subset to generate a view of the scene from the viewer position and/or orientation;

wherein the processor is further configured to divide the video stream into the plurality of units by dividing the video stream into the plurality of units based on time segmentation, and each of the units comprises a group of pictures comprising a sequence of successive frames in time;

the processor is further configured to divide each of the units into the plurality of sub-units by dividing each of the groups of pictures into the plurality of sub-units based on viewpoint segmentation, and each of the sub-units comprises a vantage defining a viewpoint from which the scene is viewable;

wherein the processor is further configured to divide each of the vantages into a plurality of sub-sub-units, based on at least a third selection, different from the first selection and the second selection, from the group consisting of time segmentation, viewpoint segmentation, and view orientation segmentation;

wherein the processor is further configured to divide each of the vantages into the plurality of sub-sub-units by dividing each of the groups of pictures into the plurality of sub-units based on view orientation segmentation;

each of the sub-sub-units comprises a tile comprising part of one of the vantages, limited to one or more particular view orientations along which the scene is viewable from the one of the vantages; and

wherein the display device is further configured to display the view.

18. The system of claim 17, wherein:

the processor is further configured to generate a unit offset table indicating a start address, within the data store, of each of the units;

the data store is further configured to store the unit offset table in the file

the processor is further configured to generate a sub-unit offset table indicating a start address, within the data store, of each of the sub-units; and

the data store is further configured to store the sub-unit offset table in the file.

19. The system of claim 17, wherein:

the processor is further configured to divide each of the vantages into frames; and

the processor is further configured to divide each of the vantages into a plurality of sub-sub-units by dividing each of the frames into the tiles.

20. The system of claim 17, wherein:

the processor is further configured to;

generate a unit offset table indicating a start address, within the data store, of each of the groups of pictures;

generate a sub-unit offset table indicating a start address, within the data store, of each of the vantages; and

generate a sub-sub-unit offset table indicating a start address, within the data store, of each of the tiles; and

the data store is further configured to store the unit offset table, the sub-unit offset table, and the sub-sub-unit offset table in the file.

21. The system of claim 17, wherein:

the processor is further configured to identify at least one duplicate tile of the tiles that is substantially duplicative of a base tile of the tiles; and

the data store is further configured to store at least the portion of the video stream in the file by storing, in place of the duplicate tile, a start address, within the data store, at which the base tile is stored.

22. The system of claim 17, wherein:

the data store is further configured to store at least a second portion of the video stream in a second file comprising a second plurality of the units; and

the processor is further configured to combine the file with the second file to generate a combined file comprising the portion and the second portion of the video stream.

23. The system of claim 17, further comprising the one or more image capture devices.