STEREOSCOPIC 3D CAMERA FOR VIRTUAL REALITY EXPERIENCE

• US 20160323561A1
• Filed: 06/10/2016
• Published: 11/03/2016
• Est. Priority Date: 04/29/2015
• Status: Active Grant

• Current Assignee: Lucid VR, Inc.
• Original Assignee: Lucid VR, Inc.
• Inventors: JIN, Han Xiong, Rowell, Adam
• Granted Patent: US 9,948,919 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/178 Metadata, e.g. disparity information
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 5/907 using static stores, e.g. storage tubes or semiconductor memories H04N5/91 takes precedence
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal
• CPC Class Codes: H04N 9/8227 the additional signal being at least another television signal
• CPC Class Codes: H04N 9/8715 involving the mixing of the reproduced video signal with a non-recorded signal, e.g. a text signal

Abstract

Embodiments are disclosed for embedding calibration metadata for a stereoscopic video capturing device. The device captures a sequence of stereoscopic images by a plurality of image sensors and combines the captured sequence of stereoscopic images into a stereoscopic video sequence. The device further embeds calibration information into the stereoscopic video sequence in a real time as the sequence of stereoscopic images is being recorded. The calibration information can be used to correct distortion caused by hardware variances of individual video capturing devices. The corrected stereoscopic videos can be used to provide a virtual reality (VR) experience by immersing a user in a simulated environment.

Claims

1. A method for embedding calibration metadata, comprising:

capturing a sequence of stereoscopic images by a plurality of image sensors of a stereoscopic video rerecording device;

combining the captured sequence of stereoscopic images into a stereoscopic video sequence; and

embedding calibration information into the stereoscopic video sequence in a real time as the sequence of stereoscopic images is being recorded.

2. The method of claim 1, wherein the step of embedding comprises:

storing the calibration information in a metadata header of a stereoscopic video file containing the stereoscopic video sequence in a real time as the sequence of stereoscopic images is being recorded and stored into the stereoscopic video file.

3. The method of claim 1, where the step of embedding comprises:

storing the calibration information in a subtitle channel or a closed caption channel of a stereoscopic video file containing the stereoscopic video sequence in a real time as the sequence of stereoscopic images is being recorded and stored into the stereoscopic video file.

4. The method of claim 1, where the step of embedding comprises:

embedding the calibration information into at least some of the stereoscopic images via an image steganography process in a real time as the sequence of stereoscopic images is being captured.

5. The method of claim 1, wherein the calibration information includes at least two sets of camera intrinsic calibration metadata, each set of camera intrinsic calibration metadata corresponds to a camera module of the stereoscopic video rerecording device including a wide-angle lens and an image processor.

6. The method of claim 5, wherein the set of camera intrinsic calibration metadata includes a plurality of focal lengths of a corresponding camera module, the focal lengths measures ability of the wide-angle lens of the corresponding camera module to bend light at different directions.

7. The method of claim 5, wherein the set of camera intrinsic calibration metadata includes coordinates of a distortion center of images captured by the wide-angle lens of the corresponding camera module.

8. The method of claim 5, wherein the set of camera intrinsic calibration metadata includes a plurality of distortion coefficients that describe levels of fisheye distortion as a function of a radius from a center of the captured image to an edge of the captured image.

9. The method of claim 1, wherein the calibration information includes a set of stereoscopic calibration metadata, which relates to relevant relationship between camera modules of the stereoscopic video rerecording device including wide-angle lenses and image processors.

10. The method of claim 9, wherein the set of stereoscopic calibration metadata is used to rectify the captured images for a left channel and a right channel such that rectified images of the left and right channels are in forms as though the rectified images were captured from the same image plane.

11. The method of claim 9, wherein the set of stereoscopic calibration metadata includes a rotation matrix that describes a rotational correction to align an image captured by one fisheye lens of the stereoscopic video rerecording device to another image captured by another fisheye lens of the stereoscopic video rerecording device.

12. The method of claim 9, wherein the set of stereoscopic calibration metadata includes a projection matrix vertically aligning an image captured by one fisheye lens of the stereoscopic video rerecording device to another image captured by another fisheye lens of the stereoscopic video rerecording device.

13. The method of claim 1, wherein the calibration information is embedded into the video sequence in a real time by a graphics processing unit (GPU).

14. The method of claim 1, wherein the calibration information includes inertial measurement data, location data, image sensor information, or lens distortion profile.

30. The method of claim 1, wherein the calibration information for each capturing device capturing a particular stereoscopic video sequence includes a rotation matrix that describes a rotational relationship between images of a left channel of the particular stereoscopic video sequence and images of a right channel of the particular stereoscopic video sequence, and a projection matrix vertically aligning the images of the left channel with the images of the right channel.

15. A method for rendering a stereoscopic video sequence using calibration metadata, comprising:

receiving a stereoscopic video sequence including a sequence of stereoscopic images by a stereoscopic video playback device;

extracting calibration information from the stereoscopic video sequence;

for each display pixel of the stereoscopic video playback device, identifying a corresponding image pixel of the stereoscopic video sequence using the calibration information; and

rendering a stereoscopic video on a stereoscopic display of the stereoscopic video playback device by projecting contents of corresponding image pixels of the stereoscopic video sequence to the display pixels of the stereoscopic video playback device.

16. The method of claim 15, wherein the step of extracting comprises:

extracting calibration information from a metadata header of a stereoscopic video file containing the stereoscopic video sequence.

17. The method of claim 15, wherein the step of extracting comprises:

extracting calibration information from a subtitle channel or a closed caption channel of a stereoscopic video file containing the stereoscopic video sequence.

18. The method of claim 15, wherein the step of extracting comprises:

decoding calibration information via an image steganography process from at least some of the stereoscopic images of the stereoscopic video sequence.

19. The method of claim 18, wherein the decoding of the calibration information via the image steganography process is performed by a graphics processing unit (GPU) of the stereoscopic video playback device.

20. The method of claim 15, wherein the stereoscopic video playback device includes a stereoscopic display component for rendering left and right channels of the stereoscopic video separately.

21. The method of claim 15, wherein the stereoscopic video sequence is captured by a plurality of wide-angle lenses of a stereoscopic recording device, and wherein for each wide-angle lens, the calibration information includes a plurality of focal lengths of a the wide-angle lens, the focal lengths measures ability of the wide-angle lens to bend light at different directions.

22. The method of claim 15, wherein the stereoscopic video sequence is captured by a plurality of wide-angle lenses of a stereoscopic recording device, and wherein for each wide-angle lens, the calibration information includes coordinates of a distortion center of images captured by the wide-angle lens.

23. The method of claim 15, wherein the stereoscopic video sequence is captured by a plurality of wide-angle lenses of a stereoscopic recording device, and wherein for each wide-angle lens, the calibration information includes a plurality of distortion coefficients that describe levels of fisheye distortion as a function of a radius from a center of a captured image to an edge of the captured image.

24. The method of claim 15, wherein the calibration information includes a rotation matrix that describes a rotational relationship between images of a left channel of the stereoscopic video sequence and images of a right channel of the stereoscopic video sequence, and a projection matrix vertically aligning the images of the left channel with the images of the right channel.

25. A method for generating a virtual reality (VR) using 360-degree stereoscopic videos, comprising:

receiving a plurality of stereoscopic video sequences, each of the stereoscopic video sequences including a wide-angle left channel video and a wide-angle right channel video;

extracting calibration information from the stereoscopic video sequences;

applying the calibration information to frames of the stereoscopic video sequences to correct distortion caused by capturing devices of the stereoscopic video sequences;

projecting frames of the stereoscopic video sequences from spherical forms to rectangular forms; and

stitching frames from wide-angle left channel videos into 360-degree frames of a left channel of a 360-degree VR stereoscopic video, and stitching frames from wide-angle right channel videos into 360-degree frames of a right channel of the 360-degree VR stereoscopic video.

26. The method of claim 25, wherein the step of projecting comprises:

projecting frames of the stereoscopic video sequences using equirectangular projection to generate frames of a rectangular form.

27. The method of claim 25, wherein the calibration information is stored in a metadata header, a subtitle channel, or a closed caption channel of stereoscopic files containing the stereoscopic video sequences.

28. The method of claim 25, wherein the calibration information for each capturing device capturing a particular stereoscopic video sequence is embedded in frames of the particular stereoscopic video sequence using a steganography process.

29. The method of claim 25, wherein the calibration information for each capturing device capturing a particular stereoscopic video sequence includes focal lengths of lenses of the capturing device, coordinates of distortion centers of images captured by the lenses of the capturing device, or distortion coefficients that describe levels of fisheye distortion as a function of a radius from a center of images captured by the lenses of the capturing device to an edge of the captured images.