Omnistereo capture and render of panoramic virtual reality content

• US 10,375,381 B2
• Filed: 12/15/2017
• Issued: 08/06/2019
• Est. Priority Date: 05/27/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Esteban, Carlos Hernandez, Anderson, Robert, Wu, Changchang, Krainin, Michael
• Primary Examiner(s): Senfi, Behrooz M
• Application Number:
  US15/843,366
• Publication Number:
  US 20180160106A1
• Time in Patent Office: 599 Days
• CPC Class Codes: G02B 2027/011 comprising device for correcting geometrical aberrations, distortion
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/211 using temporal multiplexing
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking

Abstract

Systems and methods are described include defining, at a computing device, a set of images based on captured images, projecting, at the computing device, a portion of the set of images from a planar perspective image plane onto a spherical image plane by recasting a plurality of viewing rays associated with the portion of the set of images from a plurality of viewpoints arranged around a curved path to a viewpoint, determining, at the computing device, a periphery boundary corresponding to the viewpoint and generating updated images by removing pixels that are outside of the periphery boundary, and providing, for display, the updated images within the bounds of the periphery boundary.

Claims

1. A system comprising:

at least one processor;

memory storing instructions that, when executed by the at least one processor, cause the system to perform operations including;

receiving a set of images based on captured video streams collected from at least one stereo pair of cameras;

calculating optical flow between images from the set of images to generate a plurality of image frames that are not part of the set of images, the calculating of the optical flow including analyzing image intensity fields for selected columns of pixels associated with the set of images;

interleaving the plurality of image frames into the set of images at the respective selected columns of pixels and stitching together a portion of the plurality of image frames and the set of images based at least in part on the optical flow; and

generating, using the portion of the plurality of image frames and the set of images, an omnistereo panorama.

2. The system of claim 1, wherein the omnistereo panorama is formatted for display in a head mounted display device.

3. The system of claim 1, wherein the set of images are based on captured video streams collected from 6 to 8 stereo pairs of cameras.

4. The system of claim 1, further comprising:

obtaining pose information associated with the at least one stereo pair of cameras; and

determining the portion of the plurality of image frames to stitch together with the set of images based on the pose information.

5. The system of claim 1, wherein the set of images correspond to rendered computer graphics (CG) content.

6. The system of claim 1, wherein the stitching is performed using a stitching ratio selected based at least in part on a diameter of a circular camera rig housing the at least one stereo pair of cameras used to obtain the captured images.

13. The system of claim 1, wherein the set of images include content captured using a rotatable camera rig, and the operations further comprise:

selecting, with the processor, portions of the content, the portions of the content corresponding to the plurality of image frames captured at a distance of about one meter from an outward edge of a base of the camera rig to about five meters from the outward edge of the base of the camera rig; and

stitching together the portions of the content.

14. The system of claim 13, wherein the operations further comprise:

providing the omnistereo panorama to a head-mounted display device.

15. The system of claim 13, wherein the stitching together of the portions of content is based on matching one or more image aspects included in at least two of the plurality of image frames.

16. The system of claim 13, wherein the stitching of the portions of the content is performed using a stitching ratio selected based at least in part on a diameter of the rotatable camera rig that captured the content.

17. The system of claim 13, the operations further comprising:

interpolating additional image frames that are not part of the portions of the content, and interleaving the additional image frames into the portions of the content;

stitching together the portions of content and the additional image frames; and

generating an updated stereoscopic panoramic view including the portions of the content and the additional image frames.

18. The system of claim 17, wherein stitching together the portions of the content and the additional image frames includes:

matching a portion of a first column of pixels in a first image frame to a first portion of a second column of pixels in a second image frame,

matching a second portion of the second column of pixels to a portion of a third column of pixels in a third image frame; and

generating a cohesive scene portion by;

combining the first column of pixels with the first portion of the second column of pixels; and

combining the second portion of the second column of pixels with the third column of pixels.

7. A computer program product tangibly embodied on a non-transitory computer-readable storage medium and comprising instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to:

receive a set of images based on captured video streams collected from at least one stereo pair of cameras;

generate a plurality of image frames based at least in part on a calculated optical flow between images from the set of images, the calculated optical flow being based on analyzing image intensity fields for selected columns of pixels associated with the set of images;

interleave the plurality of image frames into the set of images at the respective selected columns of pixels and stitching together a portion of the plurality of image frames and the set of images based at least in part on the optical flow; and

generate, using the portion of the plurality of image frames and the set of images, an omnistereo panorama.

8. The computer program product of claim 7, wherein the omnistereo panorama is formatted for display in a head mounted display device.

9. The computer program product of claim 7, wherein the set of images are based on captured video streams collected from 6 stereo pairs of cameras.

10. The computer program product of claim 7, further comprising:

obtaining pose information associated with the at least one stereo pair of cameras; and

determining the portion of the plurality of image frames to stitch together with the set of images based on the pose information.

11. The computer program product of claim 7, wherein the set of images correspond to rendered computer graphics (CG) content.

12. The computer program product of claim 7, wherein the stitching is performed using a stitching ratio selected based at least in part on a diameter of a circular camera rig housing the at least one stereo pair of cameras used to obtain the captured images.