Vantage generation and interactive playback

• US 10,444,931 B2
• Filed: 05/09/2017
• Issued: 10/15/2019
• Est. Priority Date: 05/09/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Akeley, Kurt
• Primary Examiner(s): Broome, Said
• Application Number:
  US15/590,841
• Publication Number:
  US 20180329602A1
• Time in Patent Office: 889 Days
• Field of Search: None
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Video data of an environment may be prepared for presentation to a user in a virtual reality or augmented reality experience. According to one method, a plurality of locations distributed throughout a viewing volume may be designated, at which a plurality of vantages are to be positioned to facilitate viewing of the environment from proximate the locations. For each location, a plurality of images of the environment, captured from viewpoints proximate the location, may be retrieved. For each location, the images may be reprojected to a three-dimensional shape and combined to generate a combined image. The combined image may be applied to one or more surfaces of the three-dimensional shape to generate a vantage. The vantages may be stored such that the vantages can be used to generate viewpoint video of the scene, as viewed from a virtual viewpoint corresponding to an actual viewer'"'"'s viewpoint within the viewing volume.

Claims

1. A method for preparing video data of an environment for presentation to a viewer in an experience selected from virtual reality and augmented reality experiences, the method comprising:

at a processor, designating a plurality of locations, distributed throughout a viewing volume, at which a plurality of vantages are to be positioned to facilitate viewing of the environment from proximate the locations in the experience;

at the processor, for each location of the plurality of the locations;

retrieving a plurality of images of the environment captured from viewpoints proximate the location; and

combining the images to generate a combined image to form a vantage, wherein combining the images comprises, for each subject pixel of the combined image, using linear interpolation to select color and/or depth data for the subject pixel based on corresponding color and/or depth data for each candidate pixel of each of the plurality of images that corresponds to the subject pixel; and

at a data store, storing each of the vantages such that the vantages can be used to generate viewpoint video of the environment, as viewed from a virtual viewpoint corresponding to an actual viewer'"'"'s viewpoint within the viewing volume.

2. A method for preparing video data of an environment for presentation to a viewer in an experience selected from virtual reality and augmented reality experiences, the method comprising:

at a processor, designating a plurality of locations, distributed throughout a viewing volume, at which a plurality of vantages are to be positioned to facilitate viewing of the environment from proximate the locations in the experience;

at the processor, for each location of the plurality of the locations;

retrieving a plurality of images of the environment captured from viewpoints proximate the location; and

combining the images to generate a combined image to form a vantage; and

at a data store, storing each of the vantages such that the vantages can be used to generate viewpoint video of the environment, as viewed from a virtual viewpoint corresponding to an actual viewer'"'"'s viewpoint within the viewing volume, and further comprising, at the processor;

receiving a designation of the virtual viewpoint; and

using the vantages to generate the viewpoint video.

3. The method of claim 2, wherein using the vantages to generate the viewpoint video comprises combining the combined images from a plurality of the vantages.

4. The method of claim 3, wherein combining the combined images from a plurality of the vantages comprises:

assigning a fitness value to each candidate pixel of each of the combined images that corresponds to the subject pixel; and

setting the subject pixel to be the same as the candidate pixel that corresponds to the subject pixel and has the highest fitness value;

wherein the fitness value for each of the candidate pixels represents a degree to which it is desirable to include the subject pixel in the experience.

5. The method of claim 3, wherein combining the combined images from a plurality of the vantages comprises, for each subject pixel of the viewpoint video, using linear interpolation to select color and/or depth data for the subject pixel based on corresponding color and/or depth data for each candidate pixel of each of the combined images that corresponds to the subject pixel.

6. The method of claim 3, wherein using the vantages to generate the viewpoint video comprises:

identifying a subset of the vantages that are proximate the virtual viewpoint; and

reprojecting at least portions of the combined images of the subset of the vantages to the virtual viewpoint.

7. The method of claim 6, wherein identifying the subset of the vantages comprises identifying vertices of a polyhedron containing the virtual viewpoint.

8. The method of claim 7, wherein:

the polyhedron comprises a tetrahedron; and

identifying the vertices of the polyhedron comprises identifying four of the vantages that define corners of the tetrahedron.

9. A method for preparing video data of an environment for presentation to a viewer in an experience selected from virtual reality and augmented reality experiences, the method comprising:

at a processor, designating a plurality of locations, distributed throughout a viewing volume, at which a plurality of vantages are to be positioned to facilitate viewing of the environment from proximate the locations in the experience;

at the processor, for each location of the plurality of the locations;

retrieving a plurality of images of the environment captured from viewpoints proximate the location; and

combining the images to generate a combined image to form a vantage; and

at a data store, storing each of the vantages such that the vantages can be used to generate viewpoint video of the environment, as viewed from a virtual viewpoint corresponding to an actual viewer'"'"'s viewpoint within the viewing volume, wherein designating a plurality of locations comprises spacing the plurality of locations apart at irregular intervals.

10. A non-transitory computer-readable medium for preparing video data of an environment for presentation to a viewer in an experience selected from virtual reality and augmented reality experiences, comprising instructions stored thereon, that when executed by a processor, cause the processor to perform the steps of:

designating a plurality of locations, distributed throughout a viewing volume, at which a plurality of vantages are to be positioned to facilitate viewing of the environment from proximate the locations in the experience;

for each location of the plurality of the locations;

retrieving a plurality of images of the environment captured from viewpoints proximate the location; and

combining the images to generate a combined image to form a vantage; and

causing a data store to store each of the vantages such that the vantages can be used to generate viewpoint video of the environment, as viewed from a virtual viewpoint corresponding to an actual viewer'"'"'s viewpoint within the viewing volume,

receiving designation of the virtual viewpoint; and

using the vantages to generate the viewpoint video.

11. The non-transitory computer-readable medium of claim 10, wherein:

using the vantages to generate the viewpoint video comprises combining the combined images from a plurality of the vantages; and

combining the combined images from a plurality of the vantages comprises;

assigning a fitness value to each candidate pixel of each of the combined images that corresponds to the subject pixel; and

setting the subject pixel to be the same as the candidate pixel that corresponds to the subject pixel and has the highest fitness value;

wherein the fitness value for each of the candidate pixels represents a degree to which it is desirable to include the subject pixel in the experience.

12. The non-transitory computer-readable medium of claim 10, wherein:

using the vantages to generate the viewpoint video comprises combining the combined images from a plurality of the vantages; and

combining the combined images from a plurality of the vantages comprises, for each subject pixel of the viewpoint video, using linear interpolation to select color and/or depth data for the subject pixel based on corresponding color and/or depth data for each candidate pixel of each of the combined images that corresponds to the subject pixel.

13. The non-transitory computer-readable medium of claim 10, wherein using the vantages to generate the viewpoint video comprises:

identifying a subset of the vantages that are proximate the virtual viewpoint by identifying vertices of a polyhedron containing the virtual viewpoint; and

combining the combined images from the subset of the vantages by reprojecting at least portions of the combined images of the subset of the vantages to the virtual viewpoint.

14. The non-transitory computer-readable medium of claim 13, wherein:

the polyhedron comprises a tetrahedron; and

identifying the vertices of the polyhedron comprises identifying four of the vantages that define corners of the tetrahedron.

15. A system for preparing video data of an environment for presentation to a viewer in an experience selected from virtual reality and augmented reality experiences, the system comprising:

a processor configured to;

designate a plurality of locations, distributed throughout a viewing volume, at which a plurality of vantages are to be positioned to facilitate viewing of the environment from proximate the locations in the experience; and

for each location of the plurality of the locations;

retrieve a plurality of images of the environment captured from viewpoints proximate the location; and

combine the images to generate a combined image to form a vantage; and

a data store communicatively coupled to the processor, configured to store each of the vantages such that the vantages can be used to generate viewpoint video of the environment, as viewed from a virtual viewpoint corresponding to an actual viewer'"'"'s viewpoint within the viewing volume, wherein the processor is further configured to combine the images by, for each subject pixel of the combined image, using linear interpolation to select color and/or depth data for the subject pixel based on corresponding color and/or depth data for each candidate pixel of each of the plurality of images that corresponds to the subject pixel.

16. A system for preparing video data of an environment for presentation to a viewer in an experience selected from virtual reality and augmented reality experiences, the system comprising:

a processor configured to;

designate a plurality of locations, distributed throughout a viewing volume, at which a plurality of vantages are to be positioned to facilitate viewing of the environment from proximate the locations in the experience; and

for each location of the plurality of the locations;

retrieve a plurality of images of the environment captured from viewpoints proximate the location; and

combine the images to generate a combined image to form a vantage; and

a data store communicatively coupled to the processor, configured to store each of the vantages such that the vantages can be used to generate viewpoint video of the environment, as viewed from a virtual viewpoint corresponding to an actual viewer'"'"'s viewpoint within the viewing volume, wherein the processor is further configured to;

receive designation of the virtual viewpoint; and

use the vantages to generate the viewpoint video.

17. The system of claim 16, wherein the processor is further configured to use the vantages to generate the viewpoint video by combining the combined images from a plurality of the vantages by, for each subject pixel of the viewpoint video:

assigning a fitness value to each candidate pixel of each of the combined images that corresponds to the subject pixel; and

setting the subject pixel to be the same as the candidate pixel that corresponds to the subject pixel and has the highest fitness value;

wherein the fitness value for each of the candidate pixels represents a degree to which it is desirable to include the subject pixel in the experience.

18. The system of claim 16, wherein the processor is further configured to use the vantages to generate the viewpoint video by combining the combined images from a plurality of the vantages by, for each subject pixel of the viewpoint video, using linear interpolation to select color and/or depth data for the subject pixel based on corresponding color and/or depth data for each candidate pixel of each of the combined images that corresponds to the subject pixel.

19. The system of claim 16, wherein the processor is further configured to use the vantages to generate the viewpoint video by:

identifying a subset of the vantages that are proximate the virtual viewpoint by identifying vertices of a polyhedron containing the virtual viewpoint; and

combining the combined images from the subset of the vantages by reprojecting at least portions of the combined images of the subset of the vantages to the virtual viewpoint.

20. The system of claim 19, wherein:

the polyhedron comprises a tetrahedron; and

the processor is further configured to identify the vertices of the polyhedron by identifying four of the vantages that define corners of the tetrahedron.

21. A system for preparing video data of an environment for presentation to a viewer in an experience selected from virtual reality and augmented reality experiences, the system comprising:

a processor configured to;

designate a plurality of locations, distributed throughout a viewing volume, at which a plurality of vantages are to be positioned to facilitate viewing of the environment from proximate the locations in the experience; and

for each location of the plurality of the locations;

retrieve a plurality of images of the environment captured from viewpoints proximate the location; and

combine the images to generate a combined image to form a vantage; and

a data store communicatively coupled to the processor, configured to store each of the vantages such that the vantages can be used to generate viewpoint video of the environment, as viewed from a virtual viewpoint corresponding to an actual viewer'"'"'s viewpoint within the viewing volume, wherein the processor is further configured to designate the plurality of locations by spacing the plurality of locations apart at irregular intervals.