Capture and render of panoramic virtual reality content

• US 10,038,887 B2
• Filed: 05/27/2015
• Issued: 07/31/2018
• Est. Priority Date: 05/27/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Gallup, David, Anderson, Robert, Esteban, Carlos Hernandez, Seitz, Steven Maxwell, Adams, Riley
• Primary Examiner(s): Senfi, Behrooz
• Application Number:
  US14/723,151
• Publication Number:
  US 20160353089A1
• Time in Patent Office: 1,161 Days
• CPC Class Codes: G02B 2027/011 comprising device for correcting geometrical aberrations, distortion
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/211 using temporal multiplexing
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking

Abstract

Systems and methods are described for defining a set of images based on captured images, receiving a viewing direction associated with a user of a virtual reality (VR) head mounted display, receiving an indication of a change in the viewing direction. The methods further include configuring, a re-projection of a portion of the set of images, the re-projection based at least in part on the changed viewing direction and a field of view associated with the captured images, and converting the portion from a spherical perspective projection into a planar perspective projection, rendering by the computing device and for display in the VR head mounted display, an updated view based on the re-projection, the updated view configured to correct distortion and provide stereo parallax in the portion, and providing, to the head mounted display, the updated view including a stereo panoramic scene corresponding to the changed viewing direction.

Claims

1. A computer-implemented method comprising:

defining, at a computing device, a set of images based on captured images;

receiving, at the computing device, a viewing direction associated with a user of a virtual reality (VR) head mounted display;

receiving, at the computing device, an indication of a change in the viewing direction;

in response to receiving the indication;

configuring, by the computing device, a re-projection of a portion of the set of images, the re-projection based at least in part on the changed viewing direction and a field of view associated with the captured images,

converting the portion, using the re-projection, from a spherical perspective projection into a planar perspective projection to reduce vertical parallax in the portion, and

generating, by the computing device, an updated view based on the re-projection, the updated view configured to correct distortion and provide, in the portion, stereo effects for the user of the head mounted display; and

triggering for display, in the head mounted display, the updated view including a stereo panoramic scene corresponding to the changed viewing direction.

2. The method of claim 1, wherein the re-projection further comprises recasting a portion of viewing rays associated with the set of images from a plurality of viewpoints arranged around a circular shape.

3. The method of claim 1, wherein the set of images are captured using a camera rig that is circular and configured to move, during capture, in an arcing motion parallel to a rotatable base of the camera rig, the camera rig including the base, a mount plate, and at least one stereo pair of cameras, the at least one stereo pair including:

a first camera, placed on the mount plate, and configured to point in a viewing direction that is tangential to an edge of the mount plate and arranged to point toward a leftward direction; and

a second camera, placed on the mount plate in a side-by-side fashion to the first camera and placed at an interpupillary distance from the first camera, the second camera arranged to point in a viewing direction that is tangential to an edge of the mount plate and arranged to point toward a rightward direction.

4. The method of claim 3, wherein the camera rig is configured with a diameter of about 10 centimeters to about 75 centimeters.

5. The method of claim 3, wherein the interpupillary distance includes about 5 centimeters to about 6.5 centimeters.

6. The method of claim 1, wherein the field of view associated with the captured images is about one-hundred eighty degrees.

7. The method of claim 1, wherein the set of images correspond to rendered computer graphics (CG) content.

8. A computer-implemented method for capturing and generating stereo panoramic content, the method comprising:

rotating a camera rig in a first direction while simultaneously capturing, using a stereo pair of cameras arranged on the camera rig, a first set of images associated with a scene surrounding the camera rig;

rotating the camera rig in a second direction while simultaneously capturing, using the stereo pair, a second set of images associated with the scene, the camera rig being circular and configured to move, during capture, in an arcing motion parallel to a base of the camera rig, the stereo pair placed with a viewing direction that is offset at opposite sides of a center of the base of the camera rig, the stereo pair being arranged on the camera rig at approximate human interpupillary distance apart;

generating a combined set of images from the first set of images and the second set of images;

performing a re-projection of a portion of the combined set of images, the re-projection based at least in part on a field of view associated with the first set of images or the second set of images; and

generating, using the re-projected portion, at least one stereoscopic panorama of the scene.

9. The method of claim 8, wherein the scene surrounding the camera rig comprises a three-hundred sixty degree field and the first set of images and the second set of images depict different portions of degrees within the three-hundred sixty degree field.

10. The method of claim 8, wherein the camera rig is circular and includes a rotatable base and a mount plate, and the stereo pair of cameras includes a first camera, placed on the mount plate, and configured to point in a viewing direction that is tangential to an edge of the mount plate and arranged to point toward a leftward direction, and a second camera, placed on the mount plate in a side-by-side fashion to the first camera and placed at an interpupillary distance from the first camera, the second camera arranged to point in a viewing direction that is tangential to an edge of the mount plate and arranged to point toward a rightward direction.

11. The method of claim 8, wherein the re-projection is performed to correct distortion in the portion.

12. The method of claim 8, wherein the stereo pair comprises at least one pair of a video camera, an image sensor, a stereoscopic camera, an infrared camera, or a mobile device.

13. A computer-implemented system comprising:

at least one processor;

memory storing instructions that, when executed by the at least one processor, cause the system to perform operations including;

defining, at a computing device, a set of images based on captured images;

receiving, at the computing device, a viewing direction associated with a user of a virtual reality (VR) head mounted display;

receiving, at the computing device, an indication of a change in the viewing direction;

in response to receiving the indication;

configuring, by the computing device, a re-projection of a portion of the set of images, the re-projection based at least in part on the changed viewing direction and a field of view associated with the captured images,

converting the portion, using the re-projection, from a spherical perspective projection into a planar perspective projection to reduce vertical parallax in the portion, and

generating, by the computing device, an updated view based on the re-projection, the updated view configured to correct distortion and provide, in the portion, stereo effects for the user of the head mounted display; and

triggering for display, in the head mounted display, the updated view including a stereo panoramic scene corresponding to the changed viewing direction.

14. The system claim 13, wherein the re-projection further comprises recasting a portion of viewing rays associated with the set of images from a plurality of viewpoints arranged around a circular shape.

15. The system claim 13, wherein the set of images are captured using a camera rig that is circular and configured to move, during capture, in an arcing motion parallel to a rotatable base of the camera rig, the camera rig including the base, a mount plate, and at least one stereo pair of cameras, the at least one stereo pair including:

a first camera, placed on the mount plate, and configured to point in a viewing direction that is tangential to an edge of the mount plate and arranged to point toward a leftward direction; and

a second camera, placed on the mount plate in a side-by-side fashion to the first camera and placed at an interpupillary distance from the first camera, the second camera arranged to point in a viewing direction that is tangential to an edge of the mount plate and arranged to point toward a rightward direction.

16. The system claim 15, wherein the camera rig is configured with a diameter of about 10 centimeters to about 75 centimeters.

17. The system claim 15, wherein the interpupillary distance includes about 5 centimeters to about 6.5 centimeters.

18. The system claim 13, wherein the field of view associated with the captured images is about one-hundred eighty degrees.

19. The system claim 13, wherein the set of images correspond to rendered computer graphics (CG) content.