Stereoscopic 3D camera for virtual reality experience

• US 9,930,315 B2
• Filed: 04/29/2016
• Issued: 03/27/2018
• Est. Priority Date: 04/29/2015
• Status: Active Grant

• Current Assignee: Lucid VR, Inc.
• Original Assignee: Lucid VR, Inc.
• Inventors: Jin, Han Xiong, Rowell, Adam
• Primary Examiner(s): Dastouri, Mehrdad
• Assistant Examiner(s): CATTUNGAL, ROWINA J
• Application Number:
  US15/143,443
• Publication Number:
  US 20160323560A1
• Time in Patent Office: 697 Days
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/178 Metadata, e.g. disparity information
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 5/907 using static stores, e.g. storage tubes or semiconductor memories H04N5/91 takes precedence
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal
• CPC Class Codes: H04N 9/8227 the additional signal being at least another television signal
• CPC Class Codes: H04N 9/8715 involving the mixing of the reproduced video signal with a non-recorded signal, e.g. a text signal

Abstract

Embodiments are disclosed for a stereoscopic device (also referred to simply as the “device”) that captures three-dimensional (3D) images and videos with a wide field of view and provides a virtual reality (VR) experience by immersing a user in a simulated environment using the captured 3D images or videos.

Claims

1. A method for recording stereoscopic 3D video, comprising:

recording sequences of stereoscopic images by multiple image sensors of a stereoscopic video recording device;

combining the recorded sequences of stereoscopic images into a stereoscopic video sequence; and

embedding calibration information into the stereoscopic video sequence in a real time as the sequences of stereoscopic images are being recorded, said embedding comprising;

receiving the stereoscopic video sequence;

embedding calibration information in at least two ways into the stereoscopic video sequence, a first way comprising embedding a static calibration information once per the stereoscopic video sequence, the static calibration information comprising a lens distortion including a distance between a lens and an image sensor, a second way comprising embedding a time varying calibration information once per frame of the stereoscopic video sequence, the time varying calibration information comprising inertial measurement data, the calibration information relating to the multiple image sensors of the stereoscopic video recording device and embedded into a portion of an aggregated stereoscopic video frame that stores the information that relates to the multiple image sensors;

combining multiple stereoscopic video sequences recorded by multiple stereoscopic recording devices;

embedding calibration information from the multiple stereoscopic video sequences into each frame of a combined stereoscopic video sequence using a video steganography process, wherein at least some of the calibration information relates to the multiple stereoscopic recording devices that captured the multiple stereoscopic video sequences;

embedding a portion of calibration information into the stereoscopic video sequence in the real time by a graphical processing unit (GPU) via an image steganography process, as the sequences of stereoscopic images are being recorded;

embedding another portion of the calibration information into a metadata section of a video file for storing the stereoscopic video sequence; and

sending the stereoscopic video sequence comprising the embedded calibration information to a playback device.

2. The method of claim 1, wherein the calibration information includes inertial measurement data, location data, or image sensor information.

3. The method of claim 1, wherein the step of embedding comprises:

embedding calibration information into the stereoscopic video sequence in the real time via an image steganography process, as the sequences of stereoscopic images are being recorded.

4. The method of claim 3, wherein the calibration information is embedded into each frame of the stereoscopic video sequence in the real time via the image steganography process.

5. The method of claim 3, wherein the calibration information is embedded into the stereoscopic video sequence in the real time by a graphical processing unit (GPU) via the image steganography process.

6. The method of claim 1, further comprising:

processing a video sequence of stereoscopic images during a playback process, wherein the stereoscopic images were captured by multiple stereoscopic video recording devices;

extracting calibration information embedded in a calibration metadata from the video sequence of stereoscopic images, wherein the extracted calibration metadata relates to the multiple stereoscopic video recording devices; and

generating a video sequence for playback using the extracted calibration metadata.

7. A method for processing a stereoscopic video, comprising:

receiving a stereoscopic 3D video sequence comprising an embedded calibration information;

processing video frames of the stereoscopic 3D video in a real time, when the stereoscopic 3D video sequence is being played back, to identify data for stabilizing the stereoscopic 3D video sequence based on a comparison of a portion of the video frames to other frames in the stereoscopic 3D video sequence;

extracting, when the stereoscopic 3D video sequences being playback, the embedded calibration information comprising;

extracting calibration information in at least two ways from the stereoscopic video sequence, a first way comprising extracting a static calibration information once per the stereoscopic video sequence, the static calibration information comprising a lens distortion profile including a distance between a lens and an image sensor, a second way comprising extracting a time varying calibration information once per frame of the stereoscopic video sequence, the time varying calibration information comprising inertial measurement data, the calibration information relating to multiple image sensors of the stereoscopic video recording device and extracted from a portion of an aggregated stereoscopic video frame that stores the information that relates to the multiple image sensors;

extracting calibration information from multiple stereoscopic video sequences into each frame of a combined stereoscopic video sequence using a video steganography process, wherein at least some of the calibration information relates to the multiple image sensors that captured the multiple stereoscopic video sequences;

extracting a portion of calibration information from the stereoscopic video sequence in the real time by a graphical processing unit (GPU) via an image steganography process;

extracting another portion of the calibration information from a metadata section of a video file for storing the stereoscopic video sequence;

generating a video stabilization operation based on a left channel and a right channel of the stereoscopic 3D video sequence; and

applying the video stabilization operation on the portion of the video frames for both the left and right channels.

8. The method of claim 7, further comprising:

stabilizing a rotation of the stereoscopic 3D video sequence on the captured images;

unwarping the stereoscopic 3D video sequence into a rectilinear mapping; and

further stabilizing the stereoscopic video 3D sequence in a rectilinear space by stabilizing translation and additional rotational components.

9. The method of claim 7, wherein the video stabilization operation is applied by a graphical processing unit (GPU).

10. The method of claim 7, wherein the stereoscopic 3D video sequence is recorded by a 3D stereoscopic camera having a field of view of approximately 180 degrees.

11. The method of claim 7, further comprising:

playing back the left and right channels of the video frames using a virtual reality headset.

12. The method of claim 11, wherein the video stabilization operation focuses on a center portion of an image window that the virtual reality headset is oriented at.

13. The method of claim 7, wherein the stereoscopic 3D video sequence is captured by fisheye lenses of a 3D stereoscopic camera.

14. The method of claim 13, wherein the video stabilization operation includes:

stabilizing a rotation of the stereoscopic 3D video sequence;

unwarping the stereoscopic 3D video sequence into a rectilinear mapping; and

stabilizing the stereoscopic 3D video sequence in a rectilinear space by stabilizing translational and rotational components.

15. The method of claim 7, further comprising:

extracting metadata from the stereoscopic 3D video sequence, wherein the metadata is captured by a 3D stereoscopic camera and is transmitted to a 3D stereoscopic playback device; and

stabilizing the stereoscopic 3D video sequence using the extracted metadata to improve precision of the video stabilization operation.

16. The method of claim 15, wherein the extracted metadata includes inertial measurement data collected by an accelerometer or a gyroscope of the 3D stereoscopic camera.

17. The method of claim 7, further comprising:

cropping each frame for right and left channels of the stereoscopic 3D video sequence after stabilization.