CALIBRATION OF MULTI-CAMERA DEVICES USING REFLECTIONS THEREOF

• US 20150341618A1
• Filed: 05/23/2014
• Published: 11/26/2015
• Est. Priority Date: 05/23/2014
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: HOLZ, David S., HE, Hongyuan Jimmy
• Granted Patent: US 9,648,300 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/20 Image signal generators
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 17/002 for television cameras

Abstract

The technology disclosed can provide capabilities such as calibrating an imaging device based on images taken by device cameras of reflections of the device itself. Implementations exploit device components that are easily recognizable in the images, such as one or more light-emitting devices (LEDs) or other light sources to eliminate the need for specialized calibration hardware and can be accomplished, instead, with hardware readily available to a user of the device—the device itself and a reflecting surface, such as a computer screen. The user may hold the device near the screen under varying orientations and capture a series of images of the reflection with the device'"'"'s cameras. These images are analyzed to determine camera parameters based on the known positions of the light sources. If the positions of the light sources themselves are subject to errors requiring calibration, they may be solved for as unknowns in the analysis.

Claims

1. A method of calibrating an imaging device including a plurality of cameras, the method including:

capturing reflection images of an imaging device using at least two cameras;

analyzing the reflection images to locate at least one feature of the imaging device therein and an error thereof, using a current calibration parameter set; and

determining an improved current calibration parameter set for the imaging device based at least in part on location of the at least one feature of the imaging device.

2. The method of claim 1, wherein the capturing includes capturing reflection images of the imaging device including at least one light source, and wherein the at least one feature of the imaging device is located in the reflection images and corresponds to the at least one light source.

3. The method of claim 1, wherein the analyzing includes:

reconstructing a set of three-dimensional (3D) positions for one or more features of the imaging device captured in the reflection images by applying stereo matching techniques.

4. The method of claim 3, wherein the analyzing includes computing the error by finding a low value for a cost function indicative of a deviation of an expected location of the at least one feature from an observed location of the at least one feature.

5. The method of claim 1, wherein the capturing includes capturing reflection images of the imaging device including three light sources positioned substantially along a straight line, and wherein the at least one feature of the imaging device is located in the reflection images and corresponds to the three light sources.

6. The method of claim 5, wherein the analyzing includes:

reconstructing a set of three-dimensional (3D) positions for the reflected light sources of the imaging device captured in the reflection images using the current calibration parameters by applying stereo matching techniques; and

computing the error from the 3D positions of the reflected light sources.

7. The method of claim 6, wherein the computing the error includes:

determining an extent to which the 3D positions are coplanar.

8. The method of claim 7, wherein determining an extent includes:

computing bisecting planes between an origin (actual) location of a light source and an expected location for the reflected light sources as reconstructed;

computing a set of dot products, each of a combination of a normal corresponding to a bisecting plane and a bisecting point, wherein the plane and the point are between the origin location of a particular light source and the expected location for the reflected particular light source;

computing a variance of the set of all dot products; and

providing the variance as the error.

9. The method of claim 8, wherein determining improved current calibration parameter set includes:

finding a calibration parameter set corresponding to a low value of the error.

10. The method of claim 1, wherein the capturing includes capturing reflection images of the imaging device includes capturing reflection images of an imaging device disposed in front of a reflective screen, the reflection images captured from reflections of the device in the reflective screen.

11. The method of claim 1, wherein the capturing includes capturing reflection images at varying orientations of the imaging device relative to a reflecting surface.

12. The method of claim 1, wherein the calibration parameters include an intrinsic parameter corresponding to a physical arrangement of portions of the camera.

13. The method of claim 12, wherein the intrinsic parameter includes at least one of a distance between a lens and a sensor of at least one of the cameras, a translational displacement between a lens and a sensor of at least one of the cameras, a relative rotation between a lens and a sensor of at least one of the cameras, and a parameter describing a curvature of a lens of at least one of the cameras.

14. The method of claim 1, wherein the calibration parameters include an extrinsic parameter corresponding to a physical arrangement of one or more cameras in the device.

15. The method of claim 14, wherein the extrinsic parameter includes at least one of a distance between any two of the cameras, a translational displacement between any two of the cameras, a relative rotation between any two of the cameras, and a position of at least one light source of the imaging device.

16. A method of determining calibration parameters for an imaging device including three light sources disposed before a reflective surface, including:

capturing includes capturing reflection images of the imaging device including three light sources positioned substantially along a straight line;

reconstructing a set of three-dimensional (3D) positions for the three light sources of the imaging device captured in the reflection images using stereo matching;

computing a set of bisecting planes, each between an origin (actual) location of one of the three light sources and an expected 3D position for the reflected light sources as reconstructed;

computing a set of dot products, each of a combination of a normal corresponding to a bisecting plane and a bisecting point, wherein the plane and the point are between the origin location of a particular light source and expected location for the reflected particular light source;

computing an error comprised of a variance of the set of all dot products; and

finding a calibration parameter set corresponding to a value of the error less than a threshold.

17. The method of claim 16, further including:

computing a score for a user based at least in part upon a set of error values determined from a repetitive execution of calibration;

comparing the score for the user to a set of scores determined for a plurality of users; and

providing to the user an indication of success based at least in part upon a favorable outcome of the comparison.

18. A computer system for calibrating an imaging device including a plurality of cameras, the system including:

an interface to receive images from at least two cameras;

a memory to store the images and instructions for execution by a processor; and

a processor to execute the instructions to analyze reflection images of the imaging device captured with the at least two cameras to thereby locate at least one feature of the imaging device therein, and to determine calibration parameters of the imaging device based on the at least one located feature.

19. The system of claim 18, further including a mirror with a screen having a reflective surface.

20. The system of claim 18 further configured to reconstruct a set of three-dimensional (3D) positions for one or more features of the imaging device captured in the reflection images by applying stereo matching techniques.