Layered content delivery for virtual and augmented reality experiences

• US 10,546,424 B2
• Filed: 10/11/2017
• Issued: 01/28/2020
• Est. Priority Date: 04/15/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Pang, Derek, Pitts, Colvin, Akeley, Kurt
• Primary Examiner(s): Du, Haixia
• Application Number:
  US15/729,918
• Publication Number:
  US 20180089903A1
• Time in Patent Office: 839 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 19/31 in the temporal domain
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 23/957 Light-field or plenoptic cameras or camera modules
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10

Abstract

A virtual reality or augmented reality experience of a scene may be presented to a viewer using layered data retrieval and/or processing. A first layer of a video stream may be retrieved, and a first viewer position and/or orientation may be received. The first layer may be processed to generate first viewpoint video of the scene from a first virtual viewpoint corresponding to the first viewer position and/or orientation. The first viewpoint video may be displayed for the viewer. Then, a second layer of the video stream may be retrieved, and a second viewer position and/or orientation may be received. The second layer may be processed to generate second viewpoint video of the scene from a second virtual viewpoint corresponding to the second viewer position and/or orientation, with higher quality than the first viewpoint video. The second viewpoint video may be displayed for the viewer.

Claims

1. A method for presenting a virtual reality or augmented reality experience of a scene, the method comprising:

from a data store, retrieving a first layer of a video stream;

at an input device, receiving a first viewer position and/or orientation;

at a processor, processing the first layer of the video stream to generate first viewpoint video of the scene from a first virtual viewpoint corresponding to the first viewer position and/or orientation, the first viewpoint video having a first quality level;

at a display device, displaying the first viewpoint video;

from the data store, after displaying the first viewpoint video, retrieving a second layer of the video stream;

at the input device, receiving a second viewer position and/or orientation;

at the processor, processing the second layer of the video stream to generate second viewpoint video of the scene from a second virtual viewpoint corresponding to the second viewer position and/or orientation, the second viewpoint video having a second quality level higher than the first quality level; and

at the display device, displaying the second viewpoint video, wherein;

the video stream comprises a plurality of vantages, each of which comprises a view of the scene from one of a plurality of virtual viewpoints within a viewing volume;

each of the vantages comprises a plurality of tiles, each of which depicts the scene from the virtual viewpoint associated with the vantage, within a limited field of view;

the first layer of the video stream comprises a first set of the tiles; and

the second layer of the video stream comprises a second set of the tiles, each of which corresponds to one of the tiles of the first set of the tiles and comprises a level of view-dependent lighting greater than that of the corresponding tile of the first set of the tiles.

2. A non-transitory computer-readable medium for presenting a virtual reality or augmented reality experience of a scene, comprising instructions stored thereon, that when executed by one or more processors, perform the steps of:

causing a data store to retrieve a first layer of a video stream;

causing an input device to receive a first viewer position and/or orientation;

processing the first layer of the video stream to generate first viewpoint video of the scene from a first virtual viewpoint corresponding to the first viewer position and/or orientation, the first viewpoint video having a first quality level;

causing a display device, displaying the first viewpoint video;

after displaying the first viewpoint video, causing the data store to retrieve a second layer of the video stream;

causing the input device to receive a second viewer position and/or orientation;

processing the second layer of the video stream to generate second viewpoint video of the scene from a second virtual viewpoint corresponding to the second viewer position and/or orientation, the second viewpoint video having a second quality level higher than the first quality level; and

causing the display device to display the second viewpoint video, wherein;

the video stream comprises a plurality of vantages, each of which comprises a view of the scene from one of a plurality of virtual viewpoints within a viewing volume;

each of the vantages comprises a plurality of tiles, each of which depicts the scene from the virtual viewpoint associated with the vantage, within a limited field of view;

the first layer of the video stream comprises a first set of the tiles; and

the second layer of the video stream comprises a second set of the tiles, each of which corresponds to one of the tiles of the first set of the tiles and comprises a level of view-dependent lighting greater than that of the corresponding tile of the first set of the tiles.

3. A system for presenting a virtual reality or augmented reality experience of a scene, the system comprising:

a data store configured to retrieve a first layer of a video stream;

an input device configured to receive a first viewer position and/or orientation;

a processor configured to process the first layer of the video stream to generate first viewpoint video of the scene from a first virtual viewpoint corresponding to the first viewer position and/or orientation, the first viewpoint video having a first quality level; and

a display device configured to display the first viewpoint video;

wherein;

the data store is further configured, after the first viewpoint video has been displayed, to retrieve a second layer of the video stream;

the input device is further configured to receive a second viewer position and/or orientation;

the processor is further configured to process the second layer of the video stream to generate second viewpoint video of the scene from a second virtual viewpoint corresponding to the second viewer position and/or orientation, the second viewpoint video having a second quality level higher than the first quality level; and

the display device is further configured to display the second viewpoint video, wherein;

the video stream comprises a plurality of vantages, each of which comprises a view of the scene from one of a plurality of virtual viewpoints within a viewing volume;

each of the vantages comprises a plurality of tiles, each of which depicts the scene from the virtual viewpoint associated with the vantage, within a limited field of view;

the first layer of the video stream comprises a first set of the tiles; and

the second layer of the video stream comprises a second set of the tiles, each of which corresponds to one of the tiles of the first set of the tiles and comprises a level of view-dependent lighting greater than that of the corresponding tile of the first set of the tiles.