Calibration of multi-camera devices using reflections thereof

• US 9,648,300 B2
• Filed: 05/23/2014
• Issued: 05/09/2017
• Est. Priority Date: 05/23/2014
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: Holz, David S., He, Hongyuan (Jimmy)
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): Pham, Nam
• Application Number:
  US14/286,165
• Publication Number:
  US 20150341618A1
• Time in Patent Office: 1,082 Days
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/20 Image signal generators
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 17/002 for television cameras

Abstract

The technology disclosed can provide capabilities such as calibrating an imaging device based on images taken by device cameras of reflections of the device itself. Implementations exploit device components that are easily recognizable in the images, such as one or more light-emitting devices (LEDs) or other light sources to eliminate the need for specialized calibration hardware and can be accomplished, instead, with hardware readily available to a user of the device—the device itself and a reflecting surface, such as a computer screen. The user may hold the device near the screen under varying orientations and capture a series of images of the reflection with the device'"'"'s cameras. These images are analyzed to determine camera parameters based on the known positions of the light sources. If the positions of the light sources themselves are subject to errors requiring calibration, they may be solved for as unknowns in the analysis.

Claims

1. A method of calibrating an imaging device, the imaging device including a plurality of cameras, the method including:

capturing reflection images of the imaging device using at least two of the plurality of cameras;

analyzing the reflection images to (i) locate at least one feature of the imaging device wherein the at least one feature is located in the reflection images and (ii) compute an error based upon a deviation of an expected location of the at least one feature from an observed location of the at least one feature, using a current calibration parameter set; and

determining an improved current calibration parameter set for the imaging device based at least in part on the error computed from the deviation between an expected location of the at least one feature and an observed location of the at least one feature.

2. The method of claim 1, wherein the capturing includes capturing reflection images of the imaging device including at least one light source, and wherein the at least one feature of the imaging device is located in the reflection images and corresponds to the at least one light source.

3. The method of claim 1, wherein the analyzing includes:

reconstructing a set of three-dimensional (3D) positions for one or more features of the imaging device captured in the reflection images by applying stereo matching techniques.

4. The method of claim 3, wherein the analyzing includes computing the error by finding a low value for a cost function indicative of a deviation of an expected location of the at least one feature from an observed location of the at least one feature.

5. The method of claim 1, wherein the capturing includes capturing reflection images of the imaging device including three light sources positioned substantially along a straight line, and wherein the at least one feature of the imaging device is located in the reflection images and corresponds to the three light sources.

6. The method of claim 5, wherein the analyzing includes:

reconstructing a set of three-dimensional (3D) positions for the reflected light sources of the imaging device captured in the reflection images using the current calibration parameters by applying stereo matching techniques; and

computing the error from the 3D positions of the reflected light sources.

7. The method of claim 6, wherein the computing the error includes:

determining an extent to which the 3D positions are coplanar.

8. The method of claim 7, wherein determining an extent includes:

computing a set of bisecting planes between an origin (actual) location of a light source and an expected location for the reflected light sources as reconstructed;

computing a set of dot products, each of a combination of a normal corresponding to a bisecting plane and a bisecting point, wherein the plane and the point are between the origin location of a particular light source and the expected location for the reflected particular light source;

computing a variance of the set of all dot products; and

providing the variance as the error.

9. The method of claim 8, wherein determining improved current calibration parameter set includes:

finding a calibration parameter set corresponding to a low value of the error.

10. The method of claim 1, wherein the capturing includes capturing reflection images of the imaging device includes capturing reflection images of an imaging device disposed in front of a reflective screen, the reflection images captured from reflections of the device in the reflective screen.

11. The method of claim 1, wherein the capturing includes capturing reflection images at varying orientations of the imaging device relative to a reflecting surface.

12. The method of claim 1, wherein the calibration parameters include an intrinsic parameter corresponding to a physical arrangement of portions of the camera.

13. The method of claim 12, wherein the intrinsic parameter includes at least one of a distance between a lens and a sensor of at least one of the cameras, a translational displacement between a lens and a sensor of at least one of the cameras, a relative rotation between a lens and a sensor of at least one of the cameras, and a parameter describing a curvature of a lens of at least one of the cameras.

14. The method of claim 1, wherein the calibration parameters include an extrinsic parameter corresponding to a physical arrangement of one or more cameras in the device.

15. The method of claim 14, wherein the extrinsic parameter includes at least one of a distance between any two of the cameras, a translational displacement between any two of the cameras, a relative rotation between any two of the cameras, and a position of at least one light source of the imaging device.

16. The method of claim 9, wherein the finding further comprises:

finding a calibration parameter set corresponding to a value of the error less than a threshold.

17. The method of claim 1, further comprising:

computing a score for a user based at least in part upon a set of error values determined from a repetitive execution of calibration;

comparing the score for the user to a set of scores determined for a plurality of users; and

providing to the user an indication of success based at least in part upon a favorable outcome of the comparison.

18. A computer system for calibrating an imaging device including a plurality of cameras, the system including:

an interface to receive reflection images of the imaging device, the reflection images captured using at least two cameras of the plurality of cameras;

a memory to store the reflection images and instructions for execution by a processor; and

a processor to execute the instructions to analyze the reflection images to (i) locate at least one feature of the imaging device wherein the at least one feature is located in the reflection images and (ii) compute an error based upon a deviation of an expected location of the at least one feature from an observed location of the at least one feature, using a current calibration parameter set, and to determine an improved calibration parameter set for the imaging device based on the at least in part on the error computed from the deviation of the expected location of the at least one feature from the observed location of the at least one feature.

19. The system of claim 18, further including a mirror with a screen having a reflective surface.

20. The system of claim 18 further configured to reconstruct a set of three-dimensional (3D) positions for one or more features of the imaging device captured in the reflection images by applying stereo matching techniques.