3D Camera Calibration for Adjustable Camera Settings

• US 20190058870A1
• Filed: 10/19/2018
• Published: 02/21/2019
• Est. Priority Date: 06/10/2016
• Status: Active Application

• Current Assignee: Lucid VR, Inc.
• Original Assignee: Lucid VR, Inc.
• Inventors: Rowell, Adam, Fernandes, Sheldon, Jin, Han
• Granted Patent: US 10,848,743 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 13/06 Panoramic objectives; So-called "sky lenses" including panoramic objectives having reflecting surfaces
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/178 Metadata, e.g. disparity information
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 5/907 using static stores, e.g. storage tubes or semiconductor memories H04N5/91 takes precedence
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal
• CPC Class Codes: H04N 9/8227 the additional signal being at least another television signal
• CPC Class Codes: H04N 9/8715 involving the mixing of the reproduced video signal with a non-recorded signal, e.g. a text signal

Abstract

Described are calibration and rectification systems and methods for 3D cameras. The calibration methods described herein derive calibration parameters in real time for multiple camera settings. Calibration parameters are used to rectify stereo images as part of a process for creating 3D images, videos, and VR or AR experiences. Additionally multi-camera systems for implementing the calibration and rectification methods are disclosed. 3D camera calibration methods involving a limited number of calibration points reduce the cost and complexity of conventional calibration methods while also allowing for higher performing 3D cameras having many possible configurations of camera settings.

Claims

1. A method of interpolating camera calibration parameters comprising:

providing a calibration file having a plurality of calibration points, each calibration point associated with unique camera setting values and mapped to unique calibration metadata, the calibration metadata arranged as a matrix of calibration parameters;

detecting real time camera settings at a real time camera position and locating, within the calibration file, calibration points having camera setting values most proximate to the real time camera settings;

storing calibration metadata for the calibration points having camera settings most proximate to the real time camera settings in memory as proximate calibration metadata and extracting, from proximate calibration metadata, intrinsic calibration parameters and extrinsic calibration parameters; and

applying an interpolation process to the intrinsic calibration parameters and extrinsic calibration parameters to calculate real time intrinsic calibration parameters and real time extrinsic calibration parameters.

2. The method of claim 1, wherein the interpolation process is a one dimensional interpolation process comprising:

applying an interpolation function to the intrinsic calibration parameters to generate the real time intrinsic calibration parameters for a first camera module and a second camera module having the real time camera settings; and

applying the interpolation function to the extrinsic calibration parameters to generate the real time extrinsic calibration parameters for a camera device having the real time camera settings.

3. The method of claim 1, wherein the interpolation process is a two dimensional interpolation process comprising:

applying an interpolation function to the intrinsic calibration parameters to generate intermediate intrinsic calibration parameters for a first camera module and a second camera module;

applying an interpolation function to the intermediate intrinsic calibration parameters to generate real time intrinsic calibration parameters for the first camera module and the second camera module having the real time camera setting,

applying an interpolation function to the extrinsic calibration parameters to generate intermediate extrinsic calibration parameters for the first camera module and the second camera module; and

applying an interpolation function to the intermediate extrinsic calibration parameters to generate real time extrinsic calibration parameters for the camera device having the real time camera setting.

4. The method of claim 1, wherein intrinsic calibration parameters comprise focal length of a first camera module, focal length of a second camera module, distortion center of a first camera module, distortion center of a second camera module, distortion coefficients of a first camera module, and distortion coefficients of a second camera module.

5. The method of claim 4, wherein distortion coefficients of the first camera module and distortion coefficients of the second camera module include a fisheye distortion parameter comprising θ

_d equals θ

(1+k₁θ

²+k₂6⁴+k₃θ

⁶+k₄θ

⁸).

6. The method of claim 1, wherein stereoscopic calibration parameters comprise three Euler angles describing a rotational offset of a first camera module relative to a second camera module and three translation distances describing a vertical offset in three dimensions between the first camera module and the second camera module.

7. The method of claim 1, wherein the camera settings are selected from the group comprising baseline, focus, zoom, aspect ratio, and relative camera rotation.

8. The method of claim 2, wherein the interpolation function is an interpolation method selected from the group comprising one dimensional nearest neighbor interpolation, linear interpolation, and cubic interpolation.

9. The method of claim 2, wherein the interpolation function is a series of linear interpolation functions.

10. The method of claim 2, wherein the interpolation function is a multi-dimension interpolation method selected from the group comprising two dimensional nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, trilinear interpolation, and tricubic interpolation.

11. A method of calculating real time calibration parameters for a camera device having a camera setting comprising:

providing a library of calibration files, each calibration file including a calibration point associated with a unique value for a camera setting, each calibration file further having a calibration parameter matrix including extrinsic calibration parameters for a camera device having a left camera module and a right camera module, the calibration parameter matrix further having intrinsic calibration parameters for the left and right camera modules;

detecting a real time value for the camera setting from the camera device and locating, within the library of calibration files, two calibration files associated with values for the camera setting most proximate to the real time value for the camera setting,

extracting the intrinsic calibration parameters and the extrinsic calibration parameters from the two calibration files;

applying an interpolation function to the intrinsic calibration parameters and receiving as an output of the interpolation function real time intrinsic calibration parameters for the first camera module and the second camera module having the real time camera setting; and

applying the interpolation function to the extrinsic calibration parameters and receiving as an output of the interpolation function real time extrinsic calibration parameters for the camera device having the real time camera setting.

12. The method of claim 11, further comprising storing the real time extrinsic calibration parameters and the real time intrinsic calibration parameters in a real time calibration file.

13. The method of claim 12, further comprising calculating rectification and projection matrices for the first camera module and the second camera module using the real time calibration file and assembling the rectification and projection matrices in a rectification file.

14. The method of claim 13, further comprising receiving a raw image frame captured by the camera device and embedding the real time calibration file and the rectification file in a header or subtitle channel of the raw image file.

15. The method of claim 13, further comprising receiving a raw image frame captured by the camera device and encoding the real time calibration file and the rectification file with the raw image frame as an image file.

16. The method of claim 11, wherein the library of calibration files includes at least 16 calibration files.

17. The method of claim 11, further comprising before application of the interpolation function normalizing one or more of the intrinsic calibration parameters by dividing a value for an intrinsic calibration parameter by a pixel dimension corresponding to pixel dimensions of an image frame captured by the camera device.

18. A method of calculating real time calibration parameters for a camera device having a first camera setting and a second camera setting, the method comprising:

providing a library of calibration files, each calibration file associated with a unique value for the first camera setting and a unique value for the second camera setting, each calibration file further having a calibration parameter matrix including extrinsic calibration parameters for the camera device, the calibration parameter matrix further having intrinsic calibration parameters for the first and second camera modules;

detecting a real time value for the first camera setting from the camera device and locating, within the library of calibration files, two calibration files associated with values for the first camera setting most proximate to the real time value for the first camera setting,

detecting a real time value for the second camera setting from the camera device and locating, within the library of calibration files, two calibration files associated with values for the second camera setting most proximate to the real time value for the second camera setting,

extracting the intrinsic calibration parameters and the extrinsic calibration parameters from the two calibration files associated with the first camera setting;

applying an interpolation function to the intrinsic calibration parameters and receiving as an output of the interpolation function a first set of intermediate intrinsic calibration parameters for the first camera module and the second camera module having the real time value for the first camera setting;

applying the interpolation function to the extrinsic calibration parameters and receiving as an output of the interpolation function a first set of intermediate extrinsic calibration parameters for the camera device having the real time value for the first camera setting,

extracting the intrinsic calibration parameters and the extrinsic calibration parameters from the two calibration files associated with the second camera setting;

applying the interpolation function to the intrinsic calibration parameters and receiving as an output of the interpolation function a second set of intermediate intrinsic calibration parameters for the first camera module and the second camera module having the real time value for the second camera setting;

applying the interpolation function to the extrinsic calibration parameters and receiving as an output of the interpolation function a second set of intermediate extrinsic calibration parameters for the camera device having the real time value for the second camera setting,

applying the interpolation function to the first set of intermediate intrinsic calibration parameters and the second set of intermediate intrinsic calibration parameters and receiving as an output of the interpolation function real time intrinsic calibration parameters for the camera device having the real time value for the first camera setting and the real time value for the second camera setting, and

applying the interpolation function to the first set of intermediate extrinsic calibration parameters and the second set of intermediate extrinsic calibration parameters and receiving as an output of the interpolation function real time extrinsic calibration parameters for the camera device having the first real time value for the first camera setting and the real time value for the second camera setting.

19. The method of claim 18, wherein the extrinsic calibration parameters include a rotation matrix formed by multiplying three elemental rotation matrices, each elemental rotation matrix including trigonometric functions of an Euler angle.

20. The method of claim 19, wherein the extrinsic calibration parameters are extracted from the two calibration files associated with the first camera setting and the two calibration files associated with the second camera setting by, within each calibration file, separating the elemental rotation matrices from the rotation matrix and deriving an x dimension Euler angle from an x dimension elemental rotation matrix, deriving a y dimension Euler angle from an y dimension elemental rotation matrix, and a z dimension Euler angle from a z dimension elemental rotation matrix.