MULTIPLE CAMERA SYSTEM WITH AUTO RECALIBRATION

• US 20160212418A1
• Filed: 04/02/2015
• Published: 07/21/2016
• Est. Priority Date: 01/19/2015
• Status: Active Grant

• Current Assignee: Packsize LLC (Packsize International, LLC)
• Original Assignee: Aquifi, Inc.
• Inventors: Salvagnini, Pietro, Demirdjian, David, Hummel, Britta Silke, Kamal, Ahmed Tashrif, Stoppa, Michele, Pasqualotto, Giuliano
• Granted Patent: US 11,019,330 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 7/00 Methods or arrangements for processing data by operating upon the order or content of the data handled logic circuits H03K19/00
• CPC Class Codes: H04N 17/002 for television cameras

Abstract

A method for a computing device to recalibrate a multiple camera system includes collecting calibration data from one or more pictures captured by the multiple camera system of the computing device. The multiple camera system includes two or more cameras that are each physically separated by a distance from one another. The method further includes detecting decalibration of the camera system. The method further includes, when the camera system is decalibrated, generating recalibration parameters based on the calibration data. The method further includes determining whether the recalibration parameters are valid parameters and, when they are, updating the multiple camera system based on the recalibration parameters.

Claims

1. A method for a computing device to recalibrate a multiple camera system, the method comprises:

generate calibration data from one or more pictures captured by the multiple camera system of the computing device, wherein the multiple camera system includes two or more cameras, and wherein each camera of the two or more cameras is physically separated by a distance from another camera of the two or more cameras;

detecting decalibration; and

when the decalibration is detected;

generating recalibration parameters based on the calibration data;

determining whether the recalibration parameters are valid parameters; and

when the recalibration parameters are valid parameters, updating the multiple camera system based on the recalibration parameters.

2. The method of claim 1, wherein the generating the calibration data comprises:

for a picture of the one or more pictures;

comparing image data of a first camera of the multiple camera system with image data of a second camera of the multiple camera system to identify a plurality of interest point tuples;

removing erroneous interest point tuples from the plurality of interest point tuples based on coordinate system mapping inconsistencies of the multiple camera system to produce a set of matching interest point tuples; and

updating the calibration data to include the set of matching interest point tuples for the picture.

3. The method of claim 2, wherein the removing erroneous interest point tuples comprises at least one of:

identifying an erroneous interest point tuple of the erroneous interest point tuples when a first interest point of the erroneous interest point tuple of the first camera is outside of an epipolar band of a second interest point of the erroneous interest point tuple of the second camera;

identifying the erroneous interest point tuple when the first interest point matches more than one interest point of the second camera;

identifying the erroneous interest point tuple when a distance metric between the first interest point and the second interest point is greater than a distance threshold; and

utilizing an essential matrix equation on the plurality of interest point tuples to identify the erroneous interest point tuples.

4. The method of claim 1, wherein the detecting the decalibration comprises:

detecting occurrence of one or more physical events from a set of physical events occurring to the device, wherein the set of physical event includes dropping, rapid acceleration, rapid deceleration, flexing, a temperature increase, a temperature decrease, a use milestone, and an aging milestone.

5. The method of claim 1, wherein the detecting decalibration comprises:

for a picture of the one or more pictures, calculating a set of epipolar liner errors for at least some matching interest point tuples of a set of matching interest point tuples of the picture;

determining a decalibration metric based on the set of epipolar line errors;

when the decalibration metric exceeds a decalibration threshold;

determining whether the picture includes a sufficient level of textural diversity to render a reliable indication of the recalibration event; and

when the picture includes the sufficient level of textural diversity, indicating the decalibration.

6. The method of claim 5 further comprises:

when the picture does not include the sufficient level of textural diversity;

for a second picture of the one or more pictures, calculating a second set of epipolar line errors for at least some matching interest point tuples of a second set of matching interest point tuples of the second picture;

determining a second decalibration metric based on the second set of epipolar line errors;

when the decalibration metric exceeds a decalibration threshold;

determining whether the picture and the second picture collectively include the sufficient level of textural diversity to render the reliable indication of the recalibration event; and

when the picture and the second picture collectively include the sufficient level of textural diversity, indicating the decalibration.

7. The method of claim 1, wherein the generating the recalibration parameters based on the calibration data comprises:

determining a number of pictures of the one or more pictures to use for recalibration parameter calculations; and

calculating extrinsic parameters for the multiple camera system using an estimated essential matrix and the calibration data.

8. The method of claim 7, wherein the calculating the extrinsic parameters comprises:

generating the estimated essential matrix based on a set of linear equations, wherein a linear equation of the set of linear equations includes a first coordinate point of a first camera of the two or more cameras, a second coordinate point of a second camera of the two or more cameras, a first matrix of the first camera, a second matrix of the second camera, and a first essential matrix of a first interest point tuple of a set of matching interest point tuples, wherein the first interest point tuple includes the first and second coordinates points;

decomposing the estimated essential matrix into a set of solutions, wherein each solution of the set of solutions includes a rotation matrix and a translation vector;

for each solution of the set of solutions, triangulating interest point tuples of the calibration data into a three-dimensional coordinate system to produce triangulated points;

identifying the solution of the set of solution that provides valid results for the triangulated interest points to produce an identified solution; and

establishing the extrinsic parameters as the rotation matrix and translation vector of the identified solution.

9. The method of claim 7, wherein the calculating the extrinsic parameters comprises:

generating the estimated essential matrix based on a set of linear equations, wherein a linear equation of the set of linear equations includes a fundamental matrix, a first matrix of the first camera, a second matrix of the second camera, and a first essential matrix of a first point tuple of the set of matching interest point tuples, wherein an interest point tuple includes the point and the corresponding point;

decomposing the estimated essential matrix into a set of solutions, wherein each solution of the set of solutions includes a rotation matrix and a translation vector;

for each solution of the set of solutions, triangulating interest point tuples of the calibration data into a three-dimensional coordinate system to produce triangulated interest points;

identifying the solution of the set of solution that provides valid results for the triangulated interest points to produce an identified solution; and

establishing the extrinsic parameters as the rotation matrix and translation vector of the identified solution.

10. The method of claim 7 further comprises:

obtaining camera parameter constraints regarding the two or more cameras; and

calculating intrinsic parameters and extrinsic parameters for each camera of the two or more cameras using a Bayesian formulation, the camera parameter constraints, and the calibration data.

11. The method of claim 1, wherein the determining whether the recalibration parameters are valid parameters comprises:

determining that the recalibration parameters converge to values that are within an expected range of parameter values.

12. The method of claim 1 further comprises:

when the decalibration is detected, providing a message for display on the computing device, wherein the message indicates decalibration of the multiple camera system.

13. A computing device comprises:

a multiple camera system that includes two or more cameras, wherein each camera of the two or more cameras is physically separated by a distance from another camera of the two or more cameras;

memory; and

a processing module operably coupled to the two or more cameras and the memory, wherein the processing module is operable to;

generate calibration data from one or more pictures captured by the multiple camera system;

store the calibration data in the memory;

detect decalibration; and

when the decalibration is detected;

generate recalibration parameters based on the calibration data;

determine whether the recalibration parameters are valid parameters; and

when the recalibration parameters are valid parameters, update the multiple camera system based on the recalibration parameters.

14. The computing device of claim 13, wherein the processing module is further operable to generate the calibration data by:

for a picture of the one or more pictures;

comparing image data of a first camera of the multiple camera system with image data of a second camera of the multiple camera system to identify a plurality of interest point tuples;

removing erroneous interest point tuples from the plurality of interest point tuples based on coordinate system mapping inconsistencies of the multiple camera system to produce a set of matching interest point tuples; and

updating the calibration data to include the set of matching interest point tuples for the picture.

15. The computing device of claim 14, wherein the processing module is further operable to remove the erroneous interest point tuples by at least one of:

identifying an erroneous interest point tuple of the erroneous interest point tuples when a first interest point of the erroneous interest point tuple of the first camera is outside of an epipolar band of a second interest point of the erroneous interest point tuple of the second camera;

identifying the erroneous interest point tuple when the first interest point matches more than one interest point of the second camera;

identifying the erroneous interest point tuple when a distance metric between the first interest point and the second point of interest is greater than a distance threshold; and

utilizing an essential matrix equation on the plurality of interest point tuples to identify the erroneous interest point tuples.

16. The computing device of claim 13, wherein the processing module is further operable to detect the decalibration by:

detecting occurrence of one or more physical events from a set of physical events occurring to the device, wherein the set of physical event includes dropping, rapid acceleration, rapid deceleration, flexing, a temperature increase, a temperature decrease, a use milestone, and an aging milestone.

17. The computing device of claim 13, wherein the processing module is further operable to detect the decalibration by:

for a picture of the one or more pictures, calculating a set of epipolar line errors for at least some matching interest point tuples of a set of matching interest point tuples of the picture;

determining a decalibration metric based on the set of epipolar line errors;

when the decalibration metric exceeds a decalibration threshold;

determining whether the picture includes a sufficient level of textural diversity to render a reliable indication of the recalibration event; and

when the picture includes the sufficient level of textural diversity, indicating the decalibration.

18. The computing device of claim 17, wherein the processing module is further operable to:

when the picture does not include the sufficient level of textural diversity;

for a second picture of the one or more pictures, calculate a second set of epipolar liner errors for at least some matching interest point tuples of a second set of matching interest point tuples of the second picture;

determine a second decalibration metric based on the second set of epipolar line errors;

when the decalibration metric exceeds a decalibration threshold;

determine whether the picture and the second picture collectively include the sufficient level of textural diversity to render the reliable indication of the recalibration event; and

when the picture and the second picture collectively include the sufficient level of textural diversity, indicate the decalibration.

19. The computing device of claim 13, wherein the processing module is further operable to generate the recalibration parameters based on the calibration data by:

determine a number of pictures of the one or more pictures to use for recalibration parameter calculations; and

calculating extrinsic parameters for each camera of the two or more cameras using an estimated essential matrix and the calibration data.

20. The computing device of claim 19, wherein the processing module is further operable to calculate the extrinsic parameters by:

generating the estimated essential matrix based on a set of linear equations, wherein a linear equation of the set of linear equations includes a first coordinate point of a first camera of the two or more cameras, a second coordinate point of a second camera of the two or more cameras, a first matrix of the first camera, a second matrix of the second camera, and a first essential matrix of a first interest point tuple of a set of matching interest point tuples, wherein the first interest point tuple includes the first and second coordinates points;

decomposing the estimated essential matrix into a set of solutions, wherein each solution of the set of solutions includes a rotation matrix and a translation vector;

for each solution of the set of solutions, triangulating interest point tuples of the calibration data into a three-dimensional coordinate system to produce triangulated interest points;

identifying the solution of the set of solution that provides valid results for the triangulated interest points to produce an identified solution; and

establishing the extrinsic parameters as the rotation matrix and translation vector of the identified solution.

21. The computing device of claim 19, wherein the processing module is further operable to calculate the extrinsic parameters by:

generating the estimated essential matrix based on a set of linear equations, wherein a linear equation of the set of linear equations includes a fundamental matrix, a first matrix of the first camera, a second matrix of the second camera, and a first essential matrix of a first interest point tuple of the set of matching interest point tuples, wherein an interest point tuple includes the interest point and the corresponding interest point;

decomposing the estimated essential matrix into a set of solutions, wherein each solution of the set of solutions includes a rotation matrix and a translation vector;

for each solution of the set of solutions, triangulating interest point tuples of the calibration data into a three-dimensional coordinate system to produce triangulated interest points;

identifying the solution of the set of solution that provides valid results for the triangulated interest points to produce an identified solution; and

establishing the extrinsic parameters as the rotation matrix and translation vector of the identified solution.

22. The computing device of claim 19, wherein the processing module is further operable to:

obtain camera parameter constraints regarding the two or more cameras; and

calculate intrinsic parameters and extrinsic parameters for each camera of the two or more cameras using a Bayesian formulation, the camera parameter constraints, and the calibration data.

23. The computing device of claim 13, wherein the processing module is further operable to determine whether the recalibration parameters are valid parameters by:

determining that the recalibration parameters converge to values that are within an expected range of parameter values.

24. A computer readable storage medium comprises:

a first memory section that stores operational instructions that, when executed by a computing device, causes the computing device to;

generate calibration data from one or more pictures captured by the multiple camera system; and

store the calibration data in memory of the computing device;

a second memory section that stores operational instructions that, when executed by the computing device, causes the computing device to;

detect decalibration; and

a third memory section that stores operational instructions that, when executed by the computing device, causes the computing device to;

when the decalibration is detected;

generate recalibration parameters based on the calibration data;

determine whether the recalibration parameters are valid parameters; and

when the recalibration parameters are valid parameters, update the multiple camera system based on the recalibration parameters.

25. The computer readable storage medium of claim 24, wherein the first memory section further stores operational instructions that, when executed by the computing device, causes the computing device to generate the calibration data by:

for a picture of the one or more pictures;

comparing image data of a first camera of the multiple camera system with image data of a second camera of the multiple camera system to identify a plurality of interest point tuples;

removing erroneous interest point tuples from the plurality of interest point tuples based on coordinate system mapping inconsistencies of the multiple camera system to produce a set of matching interest point tuples; and

updating the calibration data to include the set of matching interest point tuples for the picture.

26. The computer readable storage medium of claim 25, wherein the first memory section further stores operational instructions that, when executed by the computing device, causes the computing device to remove the erroneous interest point tuples by at least one of:

identifying an erroneous interest point tuple of the erroneous interest point tuples when a first interest point of the erroneous interest point tuple of the first camera is outside of an epipolar band of a second interest point of the erroneous interest point tuple of the second camera;

identifying the erroneous interest point tuple when the first interest point matches more than one interest point of the second camera;

identifying the erroneous interest point tuple when a distance metric between the first interest point and the second interest point is greater than a distance threshold; and

utilizing an essential matrix equation on the plurality of interest point tuples to identify the erroneous interest point tuples.

27. The computer readable storage medium of claim 24, wherein the second memory section further stores operational instructions that, when executed by the computing device, causes the computing device to detect the decalibration by:

detecting occurrence of one or more physical events from a set of physical events occurring to the device, wherein the set of physical event includes dropping, rapid acceleration, rapid deceleration, flexing, a temperature increase, a temperature decrease, a use milestone, and an aging milestone.

28. The computer readable storage medium of claim 24, wherein the second memory section further stores operational instructions that, when executed by the computing device, causes the computing device to detect the decalibration by:

for a picture of the one or more pictures, calculating a set of epipolar liner errors for at least some matching interest points of a set of matching interest point tuples of the picture;

determining a decalibration metric based on the set of epipolar line errors;

when the decalibration metric exceeds a decalibration threshold;

determining whether the picture includes a sufficient level of textural diversity to render a reliable indication of the recalibration event; and

when the picture includes the sufficient level of textural diversity, indicating the decalibration.

29. The computer readable storage medium of claim 28, wherein the first memory section further stores operational instructions that, when executed by the computing device, causes the computing device to:

when the picture does not include the sufficient level of textural diversity;

for a second picture of the one or more pictures, calculate a second set of epipolar liner errors for at least some matching interest points of a second set of matching interest point tuples of the second picture;

determine a second decalibration metric based on the second set of epipolar line errors;

when the decalibration metric exceeds a decalibration threshold;

determine whether the picture and the second picture collectively include the sufficient level of textural diversity to render the reliable indication of the recalibration event; and

when the picture and the second picture collectively include the sufficient level of textural diversity, indicate the decalibration.

30. The computer readable storage medium of claim 24, wherein the third memory section further stores operational instructions that, when executed by the computing device, causes the computing device to generate the recalibration parameters based on the calibration data by:

determine a number of pictures of the one or more pictures to use for recalibration parameter calculations; and

calculating extrinsic parameters for each camera of the two or more cameras using an estimated essential matrix and the calibration data.

31. The computer readable storage medium of claim 30, wherein the third memory section further stores operational instructions that, when executed by the computing device, causes the computing device to calculate the extrinsic parameters by:

generating the estimated essential matrix based on a set of linear equations, wherein a linear equation of the set of linear equations includes a first coordinate point of a first camera of the two or more cameras, a second coordinate point of a second camera of the two or more cameras, a first matrix of the first camera, a second matrix of the second camera, and a first essential matrix of a first interest point tuple of a set of matching interest point tuples, wherein the first interest point tuple includes the first and second coordinates points;

decomposing the estimated essential matrix into a set of solutions, wherein each solution of the set of solutions includes a rotation matrix and a translation vector;

for each solution of the set of solutions, triangulating interest point tuples of the calibration data into a three-dimensional coordinate system to produce triangulated interest points;

identifying the solution of the set of solution that provides valid results for the triangulated interest points to produce an identified solution; and

establishing the extrinsic parameters as the rotation matrix and translation vector of the identified solution.

32. The computer readable storage medium of claim 30, wherein the third memory section further stores operational instructions that, when executed by the computing device, causes the computing device to calculate the extrinsic parameters by:

generating the estimated essential matrix based on a set of linear equations, wherein a linear equation of the set of linear equations includes a fundamental matrix, a first matrix of the first camera, a second matrix of the second camera, and a first essential matrix of a first interest point tuple of the set of matching interest point tuples, wherein an interest point tuple includes the interest point and the corresponding interest point;

decomposing the estimated essential matrix into a set of solutions, wherein each solution of the set of solutions includes a rotation matrix and a translation vector;

for each solution of the set of solutions, triangulating interest point tuples of the calibration data into a three-dimensional coordinate system to produce triangulated interest points;

identifying the solution of the set of solution that provides valid results for the triangulated interest points to produce an identified solution; and

establishing the extrinsic parameters as the rotation matrix and translation vector of the identified solution.

33. The computer readable storage medium of claim 30, wherein the third memory section further stores operational instructions that, when executed by the computing device, causes the computing device to:

obtain camera parameter constraints regarding the two or more cameras; and

calculate intrinsic parameters and extrinsic parameters for each camera of the two or more cameras using a Bayesian formulation, the camera parameter constraints, and the calibration data.

34. The computer readable storage medium of claim 24, wherein the third memory section further stores operational instructions that, when executed by the computing device, causes the computing device to determine whether the recalibration parameters are valid parameters by:

determining that the recalibration parameters converge to values that are within an expected range of parameter values.

35. The computer readable storage medium of claim 24, wherein the third memory section further stores operational instructions that, when executed by the computing device, causes the computing device to:

when the decalibration is detected, display a message that indicates decalibration of the multiple camera system.