Camera pair calibration using non-standard calibration objects

• US 9,686,539 B1
• Filed: 06/12/2016
• Issued: 06/20/2017
• Est. Priority Date: 06/12/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Molgaard, Claus, Zuliani, Marco, Hubel, Paul M.
• Primary Examiner(s): Tran, Trang U
• Application Number:
  US15/180,048
• Time in Patent Office: 373 Days
• Field of Search: 348187, 348180, 348188, 348 47
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras

Abstract

Systems, methods, and computer readable media for calibrating two cameras (image capture units) using a non-standard, and initially unknown, calibration object are described. More particularly, an iterative approach to determine the structure and pose of an target object in an unconstrained environment are disclosed. The target object may be any of a number of predetermined objects such as a specific three dimensional (3D) shape, a specific type of animal (e.g., dogs), or the face of an arbitrary human. Virtually any object whose structure may be expressed in terms of a relatively low dimensional parametrized model may be used as a target object. The identified object (i.e., its pose and shape) may be used as input to a bundle adjustment operation resulting in camera calibration.

Claims

1. A method to calibrate cameras, comprising:

obtaining a first image of an object from a first image capture unit and, concurrently, a second image of the object from a second image capture unit;

identifying a first set of landmark points based on the first image and a second set of landmark points based on the second image;

determining a first pose of the object based on the first set of landmark points and a putative structure of the object and a second pose of the object based on the second set of landmark points and the putative structure of the object;

determining an estimated structure of the object based on a model of the object and the first and second poses;

determining a projection error based on the first pose, the second pose and the estimated structure of the object;

evaluating whether the projection error meets a threshold value;

calibrating the first and second image capture units based on the estimated structure when the projection error meets the threshold value; and

repeating each of the determining actions and the evaluating action when the projection error does not meet the threshold value, wherein the putative structure of the object is replaced by the estimated structure of the object.

2. The method of claim 1, wherein the object comprises a human face.

3. The method of claim 2, wherein the putative structure of the object comprises an affine space of face shapes.

4. The method of claim 1, wherein the first and second image capture units are in a single electronic device.

5. The method of claim 1, wherein determining a first pose comprises applying the first set of landmark points to a POSIT operation.

6. The method of claim 1, wherein calibrating comprises applying the estimated structure of the object to a bundle adjust operation to calibrate the first and second image capture units.

7. The method of claim 6 further comprising applying at least one of the first pose and second pose to the bundle adjust operation.

8. An electronic device, comprising:

a first image capture unit;

a second image capture unit;

a memory communicatively coupled to the first and second image capture units; and

one or more processors communicatively coupled to the first and second image capture units and the memory, the one or more processors configured to execute instructions stored in the memory to cause the electronic device to —

obtain a first image of an object from the first image capture unit and, concurrently, a second image of the object from the second image capture unit;

identify a first set of landmark points based on the first image and a second set of landmark points based on the second image;

determine a first pose of the object based on the first set of landmark points and a putative structure of the object and a second pose of the object based on the second set of landmark points and the putative structure of the object;

determine an estimated structure of the object based on a model of the object and the first and second poses;

determine a projection error based on the first pose, the second pose and the estimated structure of the object;

evaluate whether the projection error meets a threshold value;

calibrate the first and second image capture units based on the estimated structure when the projection error meets the threshold value; and

repeat each of the instructions to determine and the instructions to evaluate when the projection error does not meet the threshold value, wherein the putative structure of the object is replaced by the estimated structure of the object.

9. The electronic device of claim 8, wherein the object comprises a human face.

10. The electronic device of claim 9, wherein the putative structure of the object comprises an affine space of face shapes.

11. The electronic device of claim 8, wherein the instructions to cause the electronic device to obtain first and second images comprise instructions to obtain the first and second images from the memory.

12. The electronic device of claim 8, wherein the instructions to cause the electronic device to determine a first pose comprise instructions to apply the first set of landmark points to a POSIT operation.

13. The electronic device of claim 10, wherein the the instructions to calibrate comprise instructions to apply the estimated structure of the object to a bundle adjust operation to calibrate the first and second image capture units.

14. The electronic device of claim 13, wherein the instructions to cause the electronic device to calibrate further comprise instructions to apply at least one of the first pose and second pose to the bundle adjust operation.

15. A non-transitory program storage device, readable by one or more processors and comprising instructions stored thereon to cause the one or more processors to:

obtain a first image of an object from a first image capture unit and, concurrently, a second image of the object from a second image capture unit;

identify a first set of landmark points based on the first image and a second set of landmark points based on the second image;

determine a first pose of the object based on the first set of landmark points and a putative structure of the object and a second pose of the object based on the second set of landmark points and the putative structure of the object;

determine an estimated structure of the object based on a model of the object and the first and second poses;

determine a projection error based on the first pose, the second pose and the estimated structure of the object;

evaluate whether the projection error meets a threshold value;

calibrate the first and second image capture units based on the estimated structure when the projection error meets the threshold value; and

repeat each of the instructions to determine and the instructions to evaluate when the projection error does not meet the threshold value, wherein the putative structure of the object is replaced by the estimated structure of the object.

16. The non-transitory program storage device of claim 15, wherein the instructions to determine first and second poses based on a putative structure of the object comprise instructions to determine first and second poses based on an affine space of face shapes.

17. The non-transitory program storage device of claim 15, wherein the instructions to cause the one or more processors to determine a first pose comprise instructions to apply the first set of landmark points to a POSIT operation.

18. The non-transitory program storage device of claim 15, wherein the the instructions to calibrate comprise instructions to apply the estimated structure of the object to a bundle adjust operation to calibrate the first and second image capture units.

19. The non-transitory program storage device of claim 18, wherein the instructions to cause the electronic device to calibrate further comprise instructions to apply at least one of the first pose and second pose to the bundle adjust operation.