Stereoscopic 3D camera for virtual reality experience

• US 9,948,919 B2
• Filed: 06/10/2016
• Issued: 04/17/2018
• Est. Priority Date: 04/29/2015
• Status: Active Grant

• Current Assignee: Lucid VR, Inc.
• Original Assignee: Lucid VR, Inc.
• Inventors: Jin, Han Xiong, Rowell, Adam
• Primary Examiner(s): Dastouri, Mehrdad
• Assistant Examiner(s): CATTUNGAL, ROWINA J
• Application Number:
  US15/179,056
• Publication Number:
  US 20160323561A1
• Time in Patent Office: 676 Days
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/178 Metadata, e.g. disparity information
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 5/907 using static stores, e.g. storage tubes or semiconductor memories H04N5/91 takes precedence
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal
• CPC Class Codes: H04N 9/8227 the additional signal being at least another television signal
• CPC Class Codes: H04N 9/8715 involving the mixing of the reproduced video signal with a non-recorded signal, e.g. a text signal

Abstract

Embodiments are disclosed for embedding calibration metadata for a stereoscopic video capturing device. The device captures a sequence of stereoscopic images by a plurality of image sensors and combines the captured sequence of stereoscopic images into a stereoscopic video sequence. The device further embeds calibration information into the stereoscopic video sequence in a real time as the sequence of stereoscopic images is being recorded. The calibration information can be used to correct distortion caused by hardware variances of individual video capturing devices. The corrected stereoscopic videos can be used to provide a virtual reality (VR) experience by immersing a user in a simulated environment.

Claims

1. A method for rendering a stereoscopic video sequence using calibration metadata, comprising:

receiving, by a processor, the stereoscopic video sequence including a sequence of stereoscopic images by a stereoscopic video playback device;

extracting, by a processor, calibration information from the stereoscopic video sequence, the calibration information comprising a distortion center of a lens wherein the distortion center is different from a geometric image center when the lens lacks circular symmetry, and a distortion coefficient of the lens comprising a function of a radius from a center of a stereoscopic image in the sequence of stereoscopic images to an edge of the stereoscopic image;

for each display pixel of the stereoscopic video playback device, identifying a corresponding image pixel of the stereoscopic video sequence using the calibration information, by determining a fisheye distortion parameter for a pixel comprising θ

_d equals θ

(1+k₁θ

²+k₂6⁴+k₃θ

⁶+k₄θ

⁸), and determining a distorted point coordinates x′ and

y′

, wherein x′

equals (θ

_d/r)a, and y′

equals (θ

_d/r)b; and

rendering, by a processor, a stereoscopic video on a stereoscopic display of the stereoscopic video playback device by projecting contents of corresponding image pixels of the stereoscopic video sequence to the display pixels of the stereoscopic video playback device.

2. The method of claim 1, wherein the step of extracting comprises:

extracting calibration information from a metadata header of a stereoscopic video file containing the stereoscopic video sequence.

3. The method of claim 1, wherein the step of extracting comprises:

extracting calibration information from a subtitle channel or a closed caption channel of a stereoscopic video file containing the stereoscopic video sequence.

4. The method of claim 1, wherein the step of extracting comprises:

decoding calibration information via an image steganography process from at least some of the stereoscopic images of the stereoscopic video sequence.

5. The method of claim 4, wherein the decoding of the calibration information via the image steganography process is performed by a graphics processing unit (GPU) of the stereoscopic video playback device.

6. The method of claim 1, wherein the stereoscopic video playback device includes a stereoscopic display component for rendering left and right channels of the stereoscopic video separately.

7. The method of claim 1, wherein the stereoscopic video sequence is captured by a plurality of wide-angle lenses of a stereoscopic recording device, and wherein for each wide-angle lens, the calibration information includes a plurality of focal lengths of a the wide-angle lens, the focal lengths measures ability of the wide-angle lens to bend light at different directions.

8. The method of claim 1, wherein the stereoscopic video sequence is captured by a plurality of wide-angle lenses of a stereoscopic recording device, and wherein for each wide-angle lens, the calibration information includes coordinates of a distortion center of images captured by the wide-angle lens.

9. The method of claim 1, wherein the stereoscopic video sequence is captured by a plurality of wide-angle lenses of a stereoscopic recording device, and wherein for each wide-angle lens, the calibration information includes a plurality of distortion coefficients that describe levels of fisheye distortion as the function of a radius from a center of a captured image to an edge of the captured image.

10. The method of claim 1, wherein the calibration information includes a rotation matrix that describes a rotational relationship between images of a left channel of the stereoscopic video sequence and images of a right channel of the stereoscopic video sequence, and a projection matrix vertically aligning the images of the left channel with the images of the right channel.

11. A method for generating a virtual reality (VR) using 360-degree stereoscopic videos, comprising:

receiving, by a processor, a plurality of stereoscopic video sequences, each of the stereoscopic video sequences including a wide-angle left channel video and a wide-angle right channel video;

extracting, by a processor, calibration information from the stereoscopic video sequences, the calibration information comprising a distortion center of a lens wherein the distortion center is different from a geometric image center when the lens lacks circular symmetry, and a distortion coefficient of the lens comprising a function of a radius from a center of a stereoscopic image in the sequence of stereoscopic images to an edge of the stereoscopic image;

applying, by a processor, the calibration information to frames of the stereoscopic video sequences to correct distortion caused by capturing devices of the stereoscopic video sequences, by determining a fisheye distortion parameter for a pixel comprising θ

_d equals θ

(1+k₁θ

²+k₂6⁴+k₃θ

⁶+k₄θ

⁸), and determining a distorted point coordinates x′ and

v′

, wherein x′

equals (θ

_d/r)a, and y′

equals (θ

_d/r)b;

projecting, by a processor, frames of the stereoscopic video sequences from spherical forms to rectangular forms; and

stitching, by a processor, frames from wide-angle left channel videos into 360-degree frames of a left channel of a 360-degree VR stereoscopic video, and stitching frames from wide-angle right channel videos into 360-degree frames of a right channel of the 360-degree VR stereoscopic video.

12. The method of claim 11, wherein the step of projecting comprises:

projecting frames of the stereoscopic video sequences using equirectangular projection to generate frames of a rectangular form.

13. The method of claim 11, wherein the calibration information is stored in a metadata header, a subtitle channel, or a closed caption channel of stereoscopic files containing the stereoscopic video sequences.

14. The method of claim 11, wherein the calibration information for each capturing device capturing a particular stereoscopic video sequence is embedded in frames of the particular stereoscopic video sequence using a steganography process.

15. The method of claim 11, wherein the calibration information for each capturing device capturing a particular stereoscopic video sequence includes focal lengths of lenses of the capturing device, coordinates of distortion centers of images captured by the lenses of the capturing device, or distortion coefficients that describe levels of fisheye distortion as function of a radius from a center of images captured by the lenses of the capturing device to an edge of the captured images.

16. The method of claim 11, wherein the calibration information for each capturing device capturing a particular stereoscopic video sequence includes a rotation matrix that describes a rotational relationship between images of a left channel of the particular stereoscopic video sequence and images of a right channel of the particular stereoscopic video sequence, and a projection matrix vertically aligning the images of the left channel with the images of the right channel.

17. A method to identify a mapping between a display pixel of a stereoscopic video playback device, and a corresponding image pixel of a stereoscopic video sequence, the method comprising:

retrieving, by a processor, a distortion coefficient of a lens, a focal length of the lens, a distortion center parameter of the lens, a stereoscopic calibration matrix of a capturing device, and a raw fisheye image captured by the lens;

given the display pixel in the stereoscopic video playback device, determining, by a processor, a point in the world coordinate system that maps to the display pixel in the stereoscopic video playback device by using an orientation and parameters of the stereoscopic video playback device;

applying, by a processor, the stereoscopic calibration matrix to map the raw fisheye image into a rectified image;

intersecting, by a processor, a line from the lens to the point in the world coordinate system with the rectified image to obtain a second pixel coordinates in normalized units;

based on the second pixel coordinates in normalized units, determining, by a processor, a radius and a circular angle of the second pixel;

based on the distortion coefficient of the lens, the radius of the second pixel and the circular angle of the second pixel, determining, by a processor, a distorted coordinate by determining a fisheye distortion parameter for a pixel comprising θ

_d equals θ

_d(1+k₁θ

²+k₂θ

⁴+k₃θ

⁶+k₄θ

⁸), and determining the distorted coordinates x′ and

y′

, wherein x′

equals (θ

_d/r)a, and y′

equals (θ

_d/r)b; and

based on the focal length of the lens, and the distortion center parameter of the lens, converting, by a processor, the distorted coordinate into the corresponding image pixel of the stereoscopic video sequence.