Generation of virtual reality with 6 degrees of freedom from limited viewer data

• US 10,474,227 B2
• Filed: 02/15/2018
• Issued: 11/12/2019
• Est. Priority Date: 05/09/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Carothers, Trevor, Akeley, Kurt
• Primary Examiner(s): Broome, Said
• Application Number:
  US15/897,994
• Publication Number:
  US 20180329485A1
• Time in Patent Office: 635 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking

Abstract

A virtual reality or augmented reality experience may be presented for a viewer through the use of input including only three degrees of freedom. The input may include orientation data indicative of a viewer orientation at which a head of the viewer is oriented. The viewer orientation may be mapped to an estimated viewer location. Viewpoint video may be generated of a scene as viewed from a virtual viewpoint with a virtual location corresponding to the estimated viewer location, from along the viewer orientation. The viewpoint video may be displayed for the viewer. In some embodiments, mapping may be carried out by defining a ray at the viewer orientation, locating an intersection of the ray with a three-dimensional shape, and, based on a location of the intersection, generating the estimated viewer location. The shape may be generated via calibration with a device that receives input including six degrees of freedom.

Claims

1. A method for providing a virtual reality or augmented reality experience for a viewer, the method comprising:

at a first input device, receiving orientation data indicative of a viewer orientation at which a head of a viewer is oriented;

at a processor, mapping the viewer orientation to an estimated viewer location by;

defining a ray at the viewer orientation;

locating an intersection of the ray with a three-dimensional shape; and

based on a three-dimensional location of the intersection, generating the estimated viewer location;

at the processor, generating viewpoint video of a scene as viewed from a virtual viewpoint with a virtual location corresponding to the estimated viewer location, from along the viewer orientation; and

at a display device, displaying the viewpoint video for the viewer.

2. The method of claim 1, wherein:

the first input device is incorporated into a head-mounted display, and is incapable of providing an actual viewer location of the head; and

receiving the orientation data comprises receiving a three-dimensional measurement of the viewer orientation.

3. The method of claim 1, wherein the three-dimensional shape is generally spherical.

4. The method of claim 1, further comprising, prior to receiving the orientation data:

at a second input device, receiving calibration data for each of a plurality of calibration orientations of the head of the viewer, indicative of;

a calibration viewer orientation at which the head is oriented; and

a calibration viewer position at which the head is positioned;

for each of the calibration orientations, using the calibration viewer orientation and the calibration viewer position to project a point; and

defining the three-dimensional shape based on locations of the points.

5. The method of claim 4, further comprising, prior to receiving the orientation data, storing the three-dimensional shape in connection with an identity of the viewer.

6. The method of claim 1, wherein:

the method further comprises, at a storage device, prior to generating the viewpoint video, retrieving at least part of a video stream captured by an image capture device; and

generating the viewpoint video comprises using at least part of the video stream.

7. The method of claim 6, wherein:

the method further comprises, prior to generating the viewpoint video;

at the processor, designating a plurality of locations, distributed throughout a viewing volume, at which a plurality of vantages are to be positioned to facilitate viewing of the scene from proximate the locations;

at the processor, for each location of the plurality of the locations;

retrieving a plurality of images of the scene captured from viewpoints proximate the location; and

combining the images to generate a combined image to generate a vantage; and

at a data store, storing each of the vantages;

retrieving at least part of the video stream comprises retrieving at least a subset of the vantages; and

generating the viewpoint video comprises using the subset to generate the viewpoint video.

8. The method of claim 7, wherein:

the method further comprises, prior to retrieving the subset of the vantages, identifying the subset of the vantages based on proximity of the vantages of the subset to the virtual viewpoint; and

using the vantages to generate the viewpoint video comprises reprojecting at least portions of the combined images of the subset of the vantages to the virtual viewpoint.

9. A non-transitory computer-readable medium for providing a virtual reality or augmented reality experience for a viewer, comprising instructions stored thereon, that when executed by a processor, perform the steps of:

causing a first input device to receive orientation data indicative of a viewer orientation at which a head of a viewer is oriented;

mapping the viewer orientation to an estimated viewer location by;

defining a ray at the viewer orientation;

locating an intersection of the ray with a three-dimensional shape; and

based on a three-dimensional location of the intersection, generating the estimated viewer location;

generating viewpoint video of a scene as viewed from a virtual viewpoint with a virtual location corresponding to the estimated viewer location, from along the viewer orientation; and

causing a display device to display the viewpoint video for the viewer.

10. The non-transitory computer-readable medium of claim 9, wherein:

the first input device is incorporated into a head-mounted display, and is incapable of providing an actual viewer location of the head; and

receiving the orientation data comprises receiving a three-dimensional measurement of the viewer orientation.

11. The non-transitory computer-readable medium of claim 9, wherein the three-dimensional shape is generally spherical.

12. The non-transitory computer-readable medium of claim 9, further comprising instructions stored thereon, that when executed by a processor, perform the steps of, prior to receiving the orientation data:

causing a second input device, receiving calibration data for each of a plurality of calibration orientations of the head of the viewer, indicative of;

a calibration viewer orientation at which the head is oriented; and

a calibration viewer position at which the head is positioned;

for each of the calibration orientations, using the calibration viewer orientation and the calibration viewer position to project a point; and

defining the three-dimensional shape based on locations of the points.

13. The non-transitory computer-readable medium of claim 12, further comprising instructions stored thereon, that when executed by a processor, store the three-dimensional shape in connection with an identity of the viewer prior to receipt of the orientation data.

14. The non-transitory computer-readable medium of claim 9, wherein:

the non-transitory computer-readable medium further comprises instructions stored thereon, that when executed by a processor, cause a storage device to retrieve at least part of a video stream captured by an image capture device prior to generating the viewpoint video; and

generating the viewpoint video comprises using at least part of the video stream.

15. The non-transitory computer-readable medium of claim 14, wherein:

the non-transitory computer-readable medium further comprises instructions stored thereon, that when executed by a processor, perform the steps of, prior to generating the viewpoint video;

designating a plurality of locations, distributed throughout a viewing volume, at which a plurality of vantages are to be positioned to facilitate viewing of the scene from proximate the locations;

for each location of the plurality of the locations;

retrieving a plurality of images of the scene captured from viewpoints proximate the location; and

combining the images to generate a combined image to generate a vantage; and

causing a data store, storing each of the vantages;

retrieving at least part of the video stream comprises retrieving at least a sub-set of the vantages; and

generating the viewpoint video comprises using the subset to generate the viewpoint video.

16. The non-transitory computer-readable medium of claim 15, wherein:

the non-transitory computer-readable medium further comprises instructions stored thereon, that when executed by a processor, identifies the subset of the vantages based on proximity of the vantages of the subset to the virtual viewpoint prior to retrieval of the subset of the vantages; and

using the vantages to generate the viewpoint video comprises reprojecting at least portions of the combined images of the subset of the vantages to the virtual viewpoint.

17. A system for providing a virtual reality or augmented reality experience for a viewer, the system comprising:

a first input device configured to receive orientation data indicative of a viewer orientation at which a head of a viewer is oriented;

a processor configured to;

map the viewer orientation to an estimated viewer location by;

defining a ray at the viewer orientation;

locating an intersection of the ray with a three-dimensional shape; and

based on a three-dimensional location of the intersection, generating the estimated viewer location; and

generate viewpoint video of a scene as viewed from a virtual view-point with a virtual location corresponding to the estimated viewer location, from along the viewer orientation; and

a display device configured to display the viewpoint video for the viewer.

18. The system of claim 17, wherein:

the first input device is incorporated into a head-mounted display, and is incapable of providing an actual viewer location of the head; and

the first input device is configured to receive the orientation data by receiving a three-dimensional measurement of the viewer orientation.

19. The system of claim 17, wherein the three-dimensional shape is generally spherical.

20. The system of claim 17, further comprising a second input device configured to receive calibration data for each of a plurality of calibration orientations of the head of the viewer, indicative of:

a calibration viewer orientation at which the head is oriented; and

a calibration viewer position at which the head is positioned;

and wherein the processor is further configured to;

for each of the calibration orientations, use the calibration viewer orientation and the calibration viewer position to project a point; and

define the three-dimensional shape based on locations of the points.

21. The system of claim 20, wherein the processor is further configured to store the three-dimensional shape in connection with an identity of the viewer prior to receipt of the orientation data.

22. The system of claim 17, further comprising a storage device configured to retrieve, prior to generation of the viewpoint video, at least part of a video stream captured by an image capture device;

and wherein generating the viewpoint video comprises using at least part of the video stream.

23. The system of claim 22, wherein the processor is further configured to, prior to generation of the viewpoint video:

designate a plurality of locations, distributed throughout a viewing volume, at which a plurality of vantages are to be positioned to facilitate viewing of the scene from proximate the locations; and

for each location of the plurality of the locations;

retrieving a plurality of images of the scene captured from viewpoints proximate the location; and

combining the images to generate a combined image to generate a vantage;

and wherein;

the system further comprises a data store configured to store each of the vantages;

the processor is further configured to retrieve at least part of the video stream by retrieving at least a subset of the vantages; and

the processor is further configured to generate the viewpoint video by using the subset to generate the viewpoint video.

24. The system of claim 23, wherein:

the processor is further configured to, prior to retrieval of the subset of the vantages, identify the subset of the vantages based on proximity of the vantages of the subset to a virtual viewpoint; and

the processor is further configured to use the vantages to generate the viewpoint video by reprojecting at least portions of the combined images of the subset of the vantages to the virtual viewpoint.