CAPTURE AND RENDER OF PANORAMIC VIRTUAL REALITY CONTENT

• US 20160353089A1
• Filed: 05/27/2015
• Published: 12/01/2016
• Est. Priority Date: 05/27/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Gallup, David, Anderson, Robert, Esteban, Carlos Hernandez, Seitz, Steven Maxwell, Adams, Riley
• Granted Patent: US 10,038,887 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/011 comprising device for correcting geometrical aberrations, distortion
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/211 using temporal multiplexing
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking

Abstract

Systems and methods are described for defining a set of images based on captured images, receiving a viewing direction associated with a user of a virtual reality (VR) head mounted display, receiving an indication of a change in the viewing direction. The methods further include configuring, a re-projection of a portion of the set of images, the re-projection based at least in part on the changed viewing direction and a field of view associated with the captured images, and converting the portion from a spherical perspective projection into a planar perspective projection, rendering by the computing device and for display in the VR head mounted display, an updated view based on the re-projection, the updated view configured to correct distortion and provide stereo parallax in the portion, and providing, to the head mounted display, the updated view including a stereo panoramic scene corresponding to the changed viewing direction.

Claims

1. A computer-implemented method comprising:

defining, at a computing device, a set of images based on captured images;

receiving, at the computing device, a viewing direction associated with a user of a virtual reality (VR) head mounted display;

receiving, at the computing device, an indication of a change in the viewing direction;

in response to receiving the indication, configuring, by the computing device, a re-projection of a portion of the set of images, the re-projection based at least in part on the changed viewing direction and a field of view associated with the captured images, and converting the portion, using the re-projection, from a spherical perspective projection into a planar perspective projection;

rendering by the computing device and for display in the VR head mounted display, an updated view based on the re-projection, the updated view configured to correct distortion and provide stereo parallax in the portion; and

providing, to the head mounted display, the updated view including a stereo panoramic scene corresponding to the changed viewing direction.

2. The method of claim 1, wherein the re-projection further comprises recasting a portion of viewing rays associated with the set of images from a plurality of viewpoints arranged around a circular shape.

3. The method of claim 1, wherein the set of images are captured using a camera rig that is circular and configured to move, during capture, in an arcing motion parallel to a rotatable base of the camera rig, the camera rig including the base, a mount plate, and at least one stereo pair of cameras, the at least one stereo pair including:

a first camera, placed on the mount plate, and configured to point in a viewing direction that is tangential to an edge of the mount plate and arranged to point toward a leftward direction; and

a second camera, placed on the mount plate in a side-by-side fashion to the first camera and placed at an interpupillary distance from the first camera, the second camera arranged to point in a viewing direction that is tangential to an edge of the mount plate and arranged to point toward a rightward direction.

4. The method of claim 3, wherein the camera rig is configured with a diameter of about 10 centimeters to about 75 centimeters.

5. The method of claim 3, wherein the interpupillary distance includes about 5 centimeters to about 6.5 centimeters.

6. The method of claim 1, wherein the field of view associated with the captured images is about one-hundred eighty degrees.

7. The method of claim 1, wherein the set of images correspond to rendered computer graphics (CG) content.

8. A system comprising:

at least one processor;

memory storing instructions that, when executed by the at least one processor, cause the system to perform operations including;

defining a set of images based on captured video streams collected from at least one stereo pair of cameras;

calculating optical flow in the set of images to interpolate image frames that are not part of the set of images;

interleaving the image frames into the set of images and stitching together the image frames and the set of images based at least in part on the optical flow; and

generating, using the image frames and the set of images, an omnistereo panorama for display in a VR head mounted display.

9. The system of claim 8, wherein the set of images are based on captured video streams collected from about 6 to about 8 stereo pairs of cameras.

10. The system of claim 8, further comprising, using pose information associated, with the at least one stereo pair, to pre-stitch a portion of the set of images before performing the interleaving.

11. The system of claim 8, wherein calculating optical flow in the set of images includes analyzing image intensity fields for a portion of columns of pixels associated with the set of images and performing optical flow techniques on the portion of columns of pixels.

12. The system of claim 8, wherein the set of images correspond to rendered computer graphics (CG) content.

13. A computer-implemented method comprising:

rotating a camera rig in a first direction while simultaneously capturing, using a stereo pair of cameras arranged on the camera rig, images associated with a scene surrounding the camera rig; and

rotating the camera rig in a second direction while simultaneously capturing, using the stereo pair, additional images associated with the scene, the camera rig being circular and configured to move, during capture, in an arcing motion parallel to a base of the camera rig, the stereo pair placed with a viewing direction that is offset at opposite sides of a center of the base of the camera rig, the stereo pair arranged at approximate human interpupillary distance apart.

14. The method of claim 13, wherein the scene surrounding the camera rig comprises a three-hundred sixty degree field and the images and the additional images depict different portions of degrees within the three-hundred sixty degree field.

15. The method of claim 13, wherein the camera rig is circular and includes a rotatable base and a mount plate, and the stereo pair of cameras includes a first camera, placed on the mount plate, and configured to point in a viewing direction that is tangential to an edge of the mount plate and arranged to point toward a leftward direction, and a second camera, placed on the mount plate in a side-by-side fashion to the first camera and placed at an interpupillary distance from the first camera, the second camera arranged to point in a viewing direction that is tangential to an edge of the mount plate and arranged to point toward a rightward direction.

16. The method of claim 13, wherein the interpupillary distance comprises about 5 centimeters to about 6.5 centimeters.

17. The method of claim 13, wherein the stereo pair comprises at least one pair of a video camera, an image sensor, a stereoscopic camera, an infrared camera, or a mobile device.

18. A method comprising:

receiving, at a computing device, a set of images depicting captured content from a rotatable camera rig;

selecting, with the computing device, portions of image frames in the images, the image frames including content captured with the camera rig at a distance of about one radial meter from an outward edge of a base of the camera rig to about five radial meters from the outward edge of the base of the camera rig;

stitching together the portions of image frames to generate a stereoscopic panoramic view, the stitching based at least in part on matching the portions to at least one other image frame in the portions, the stitching performed using a stitching ratio selected based at least in part on a diameter of the camera rig; and

providing the view to a head-mounted display.

19. The method of claim 18, further comprising interpolating additional image frames that are not part of the portions of image frames, and interleaving the additional image frames into the portions of image frames to generate virtual content for the view;

stitching together the portions of image frames and the additional image frames; and

providing an updated view to the head-mounted display, the updated view based at least in part on the portions of image frames and the additional image frames.

20. The method of claim 19, wherein stitching together the portions of image frames and the additional image frames includes matching a first column of pixels in a first image frame to a second column of pixels in a second image frame, and matching the second column of pixels to a third column of pixels in a third image frame, the matching providing a cohesive scene portion.