Spatial random access enabled video system with a three-dimensional viewing volume

• US 10,341,632 B2
• Filed: 05/09/2017
• Issued: 07/02/2019
• Est. Priority Date: 04/15/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Pang, Derek, Pitts, Colvin, Akeley, Kurt
• Primary Examiner(s): Du, Haixia
• Application Number:
  US15/590,877
• Publication Number:
  US 20170244948A1
• Time in Patent Office: 784 Days
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 23/957 Light-field or plenoptic cameras or camera modules
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10

Abstract

An environment may be displayed from a viewpoint. According to one method, volumetric video data may be acquired depicting the environment, for example, using a tiled camera array. A plurality of vantages may be distributed throughout a viewing volume from which the environment is to be viewed. The volumetric video data may be used to generate video data for each vantage, representing the view of the environment from that vantage. User input may be received designating a viewpoint within the viewing volume. From among the plurality of vantages, a subset nearest to the viewpoint may be identified. The video data from the subset may be retrieved and combined to generate viewpoint video data depicting the environment from the viewpoint. The viewpoint video data may be displayed for the viewer to display a view of the environment from the viewpoint selected by the user.

Claims

1. A method for displaying an environment from a viewpoint, the method comprising:

at an input device, receiving user input designating the viewpoint within a viewing volume;

at one or more processors, identifying, from among a plurality of vantages within the viewing volume, a subset of the vantages nearest to the viewpoint comprising at least two of the vantages, each of which has associated video data;

at a data store, retrieving the video data from the subset of the vantages;

at the one or more processors, combining the video data from the subset of the vantages to generate viewpoint video data depicting the environment from the viewpoint; and

at a display device, displaying the viewpoint video data, wherein identifying the subset comprises identifying four vantages of the plurality of vantages that define a tetrahedral shape around the viewpoint.

2. The method of claim 1, further comprising, prior to receiving the user input:

with one or more cameras, capturing volumetric video data depicting the environment from within the viewing volume;

at the one or more processors, distributing the plurality of vantages throughout the viewing volume; and

at the one or more processors, using the volumetric video data to generate the video data for each of the vantages such that the video data for each vantage comprises a subset of the volumetric video data that depicts the environment as viewed from the vantage.

3. The method of claim 1, wherein:

the video data for each of the vantages depicts an omnidirectional view from the vantage;

the user input further designates a view direction; and

the viewpoint video data further depicts the environment from along the view direction.

4. The method of claim 3, wherein:

the video data for each of the vantages comprises a plurality of tiles;

each of the tiles comprises a sectoral portion of the video data; and

combining the video data from the subset of the vantages to generate the viewpoint video data comprises combining the video data from only the tiles of the video data that are oriented proximate the view direction.

5. The method of claim 4, further comprising, prior to displaying the viewpoint video data on the display device, transmitting the viewpoint video data to a client computing device having the display device exclusive of video data for any of the tiles that are not oriented proximate the view direction.

6. The method of claim 1, further comprising, prior to displaying the viewpoint video data on the display device, transmitting the viewpoint video data to a client computing device having the display device exclusive of video data for any of the vantages outside the subset.

7. The method of claim 1, further comprising using the video data from the subset of the vantages to predict at least a portion of additional viewpoint video data from one of the viewpoint and an additional viewpoint proximate to the viewpoint.

8. The method of claim 1, further comprising using a first portion of the video data from the subset of the vantages, pertaining to a first portion of the viewpoint video data, to predict a second portion of the viewpoint video data to which the first portion of the video data does not directly pertain.

9. The method of claim 1, further comprising:

at the one or more processors, compressing at least one selection from a group consisting of the viewpoint video data and the video data from the subset of the vantages;

transmitting the selection, in compressed form, to a client computing device having the display device; and

decompressing the at least one selection from the group consisting of the viewpoint video data and the video data from the subset of the vantages at the client computing device.

10. A method for displaying an environment from a viewpoint, the method comprising:

at an input device, receiving user input designating the viewpoint within a viewing volume;

at one or more processors, identifying, from among a plurality of vantages within the viewing volume, a subset of the vantages nearest to the viewpoint comprising at least two of the vantages, each of which has associated video data;

at a data store, retrieving the video data from the subset of the vantages;

at the one or more processors, combining the video data from the subset of the vantages to generate viewpoint video data depicting the environment from the viewpoint; and

at a display device, displaying the viewpoint video data, wherein the video data for each of the vantages comprises at least a first layer having a first resolution, and a second layer having a second resolution higher than the first resolution and further comprising, at the one or more processors;

identifying a shortage of at least one of bandwidth, storage space, and processing power; and

in response to identification of the shortage, selecting the first layer;

wherein, combining the video data from the subset of the vantages to generate the viewpoint video data comprises combining the first layers of the video data from the subset.

11. The method of claim 10, wherein the first resolution is selected from a group consisting of spatial resolution, bit rate, and temporal resolution.

12. The method of claim 10, further comprising, prior to displaying the viewpoint video data on the display device, transmitting the viewpoint video data to a client computing device having the display device;

wherein the viewpoint video data excludes data from the second layers of the video data from the subset.

13. The method of claim 10, further comprising, at the one or more processors, after displaying the viewpoint video data:

identifying cessation of the shortage; and

in response to identification of cessation of the shortage, selecting the second layer;

wherein the method further comprises;

combining the video data from the subset of the vantages to generate additional viewpoint video data by combining the second layers of the video data from the subset; and

at the display device, displaying the additional viewpoint video data.

14. A method for displaying an environment from a viewpoint, the method comprising:

at an input device, receiving user input designating the viewpoint within a viewing volume;

at one or more processors, identifying, from among a plurality of vantages within the viewing volume, a subset of the vantages nearest to the viewpoint comprising at least two of the vantages, each of which has associated video data;

at a data store, retrieving the video data from the subset of the vantages;

at the one or more processors, combining the video data from the subset of the vantages to generate viewpoint video data depicting the environment from the viewpoint; and

at a display device, displaying the viewpoint video data, wherein;

the user input further designates a view direction;

the viewpoint video data further depicts the environment from along the view direction; and

the method further comprises using the video data from the subset of the vantages to predict from the viewpoint at one of the view direction and an additional view direction different from the view direction.

15. A method for preparing viewpoint video data for display for a user, the method comprising:

with one or more cameras, capturing volumetric video data depicting an environment from within a viewing volume;

at one or more processors, distributing a plurality of vantages throughout the viewing volume; and

at the one or more processors, using the volumetric video data to generate video data for each of the vantages such that the video data for each vantage comprises a subset of the volumetric video data that depicts the environment as viewed from the vantage, wherein generating the video data for each of the vantages comprises generating, within the video data for each of the vantages, at least a first layer having a first resolution and a second layer having a second resolution higher than the first resolution, and further comprising transmitting the first layer, but not the second layer, of the video data for a subset of the vantages to a client computing device having a display device.

16. The method of claim 15, wherein the video data for each of the vantages depicts an omnidirectional view from the vantage.

17. The method of claim 16, wherein:

the video data for each of the vantages comprises a plurality of tiles; and

each of the tiles comprises a sectoral portion of the video data.

18. The method of claim 15, further comprising transmitting the video data for a subset of the vantages to a client computing device having a display device without transmitting the video data for any of the vantages outside the subset to the client computing device.

19. The method of claim 15, further comprising:

at the one or more processors, compressing at least a portion of the video data; and

transmitting at least the portion of the video data, in compressed form, to a client computing device having a display device.

20. A non-transitory computer-readable medium for displaying an environment from a viewpoint, comprising instructions stored thereon, that when executed by one or more processors, perform the steps of:

causing an input device to receive user input designating the viewpoint within a viewing volume;

identifying, from among a plurality of vantages within the viewing volume, a subset of the vantages nearest to the viewpoint comprising at least two of the vantages, each of which has associated video data;

causing a data store to retrieve the video data from the subset of the vantages;

combining the video data from the subset of the vantages to generate viewpoint video data depicting the environment from the viewpoint; and

causing a display device to display the viewpoint video data, wherein the video data for each of the vantages comprises at least a first layer having a first resolution, and a second layer having a second resolution higher than the first resolution and further comprising instructions stored thereon, that when executed by one or more processors, perform the steps of;

identifying a shortage of at least one of bandwidth, storage space, and processing power; and

in response to identification of the shortage, selecting the first layer;

wherein, combining the video data from the subset of the vantages to generate the viewpoint video data comprises combining the first layers of the video data from the subset.

21. The non-transitory computer-readable medium of claim 20, further comprising instructions stored thereon, that when executed by one or more processors, perform the steps of:

causing one or more cameras to capture volumetric video data depicting the environment from within the viewing volume;

distributing the plurality of vantages throughout the viewing volume; and

using the volumetric video data to generate the video data for each of the vantages such that the video data for each vantage comprises a subset of the volumetric video data that depicts the environment as viewed from the vantage.

22. The non-transitory computer-readable medium of claim 20, wherein:

the video data for each of the vantages depicts an omnidirectional view from the vantage;

the user input further designates a view direction; and

the viewpoint video data further depicts the environment from along the view direction.

23. The non-transitory computer-readable medium of claim 22, wherein:

the video data for each of the vantages comprises a plurality of tiles;

each of the tiles comprises a sectoral portion of the video data; and

combining the video data from the subset of the vantages to generate the viewpoint video data comprises combining the video data from only the tiles of the video data that are oriented proximate the view direction.

24. The non-transitory computer-readable medium of claim 20, further comprising additional instructions stored thereon, that when executed by one or more processors, prior to causing the display device to display the viewpoint video data, initiate transmission the viewpoint video data to a client computing device having the display device exclusive of video data for any of the vantages outside the subset.

25. The non-transitory computer-readable medium of claim 20, further comprising additional instructions stored thereon, that when executed by the one or more processors, use the video data from the subset of the vantages to predict at least a portion of additional viewpoint video data from one of the viewpoint and an additional viewpoint proximate to the viewpoint.

26. The non-transitory computer-readable medium of claim 20, further comprising additional instructions stored thereon, that when executed by the one or more processors, perform the steps of:

compressing at least one selection from a group consisting of the viewpoint video data and the video data from the subset of the vantages;

initiating transmission of the selection, in compressed form, to a client computing device having the display device; and

decompressing the at least one selection from the group consisting of the viewpoint video data and the video data from the subset of the vantages at the client computing device.

27. A non-transitory computer-readable medium for preparing viewpoint video data for display for a user, comprising instructions stored thereon, that when executed by one or more processors, perform the steps of:

causing one or more cameras to capture volumetric video data depicting an environment from within a viewing volume;

distributing a plurality of vantages throughout the viewing volume; and

using the volumetric video data to generate video data for each of the vantages such that the video data for each vantage comprises a subset of the volumetric video data that depicts the environment as viewed from the vantage, wherein generating the video data for each of the vantages comprises generating, within the video data for each of the vantages, at least a first layer having a first resolution and a second layer having a second resolution higher than the first resolution;

wherein the non-transitory computer-readable medium further comprises instructions stored thereon, that when executed by one or more processors, initiate transmission of the first layer, but not the second layer, of the video data for a subset of the vantages to a client computing device having a display device.

28. The non-transitory computer-readable medium of claim claim 27, wherein:

the video data for each of the vantages depicts an omnidirectional view from the vantage;

the video data for each of the vantages comprises a plurality of tiles; and

each of the tiles comprises a sectoral portion of the video data.

29. The non-transitory computer-readable medium of claim 27, further comprising additional instructions stored thereon, that when executed by the one or more processors, initiate transmission of the video data for a subset of the vantages to a client computing device having a display device without transmitting the video data for any of the vantages outside the subset to the client computing device.

30. The non-transitory computer-readable medium of claim 27, further comprising additional instructions stored thereon, that when executed by the one or more processors, perform the steps of:

compressing at least a portion of the video data; and

initiating transmission of at least the portion of the video data, in compressed form, to a client computing device having a display device.

31. A system for displaying an environment from a viewpoint, the system comprising:

an input device configured to receive user input designating the viewpoint within a viewing volume;

one or more processors, communicatively coupled to the input device configured to identify, from among a plurality of vantages within the viewing volume, a subset of the vantages nearest to the viewpoint comprising at least two of the vantages, each of which has associated video data;

a data store, communicatively coupled to the processor, configured to retrieve the video data from the subset of the vantages; and

a display device communicatively coupled to the processor;

wherein;

the one or more processors are further configured to combine the video data from the subset of the vantages to generate viewpoint video data depicting the environment from the viewpoint; and

the display device is configured to display the viewpoint video data, wherein the video data for each of the vantages comprises at least a first layer having a first resolution, and a second layer having a second resolution higher than the first resolution and wherein the one or more processors are further configured to;

identify a shortage of at least one of bandwidth, storage space, and processing power; and

in response to identification of the shortage, select the first layer;

wherein, the one or more processors are further configured to combine the video data from the subset of the vantages to generate the viewpoint video data by combining the first layers of the video data from the subset.

32. The system of claim 31, further comprising one or more cameras configured to capture volumetric video data depicting the environment from within the viewing volume;

wherein the one or more processors are further configured to;

distribute the plurality of vantages throughout the viewing volume; and

use the volumetric video data to generate the video data for each of the vantages such that the video data for each vantage comprises a subset of the volumetric video data that depicts the environment as viewed from the vantage.

33. The system of claim 31, wherein:

the video data for each of the vantages depicts an omnidirectional view from the vantage;

the user input further designates a view direction; and

the viewpoint video data further depicts the environment from along the view direction.

34. The system of claim 33, wherein:

the video data for each of the vantages comprises a plurality of tiles;

each of the tiles comprises a sectoral portion of the video data; and

the one or more processors are further configured to combine the video data from the subset of the vantages to generate the viewpoint video data by combining the video data from only the tiles of the video data that are oriented proximate the view direction.

35. The system of claim 31, wherein the one or more processors are further configured, prior to display of the viewpoint video data on the display device, to initiate transmission of the viewpoint video data to a client computing device having the display device exclusive of video data for any of the vantages outside the subset.

36. The system of claim 31, wherein the one or more processors are further configured to use the video data from the subset of the vantages to predict at least a portion of additional viewpoint video data from one of the viewpoint and an additional viewpoint proximate to the viewpoint.

37. The system of claim 31, wherein the one or more processors are further configured to:

compress at least one selection from a group consisting of the viewpoint video data and the video data from the subset of the vantages;

initiate transmission of the selection, in compressed form, to a client computing device having the display device; and

decompress the at least one selection from the group consisting of the viewpoint video data and the video data from the subset of the vantages at the client computing device.

38. A system for preparing viewpoint video data for display for a user, the system comprising:

one or more cameras configured to capture volumetric video data depicting an environment from within a viewing volume; and

one or more processors, communicatively coupled to the one or more cameras, configured to;

distribute a plurality of vantages throughout the viewing volume; and

use the volumetric video data to generate video data for each of the vantages such that the video data for each vantage comprises a subset of the volumetric video data that depicts the environment as viewed from the vantage, wherein the one or more processors are further configured to generate the video data for each of the vantages by generating, within the video data for each of the vantages, at least a first layer having a first resolution and a second layer having a second resolution higher than the first resolution;

wherein the one or more processors are further configured to initiate transmission of the first layer, but not the second layer, of the video data for a subset of the vantages to a client computing device having a display device.

39. The system of claim 38, wherein:

the video data for each of the vantages depicts an omnidirectional view from the vantage;

the video data for each of the vantages comprises a plurality of tiles; and

each of the tiles comprises a sectoral portion of the video data.

40. The system of claim 38, wherein the one or more processors are further configured to initiate transmission of the video data for a subset of the vantages to a client computing device having a display device without transmitting the video data for any of the vantages outside the subset to the client computing device.

41. The system of claim 38, wherein the one or more processors are further configured to:

compress at least a portion of the video data; and

initiate transmission of at least the portion of the video data, in compressed form, to a client computing device having a display device.