METHOD FOR DETERMINING THE POSE OF A CAMERA WITH RESPECT TO AT LEAST ONE REAL OBJECT

• US 20120120199A1
• Filed: 07/29/2009
• Published: 05/17/2012
• Est. Priority Date: 07/29/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Metaio GmbH (Apple Inc.)
• Inventors: Ben Himane, Mohammed Selim
• Granted Patent: US 9,111,347 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/75 involving models

Abstract

A method for determining the pose of a camera with respect to at least one real object, the method comprises the following steps: operating the camera (1) for capturing a 2-dimensional (or 3-dimensional) image (4) including at least a part of the real object (3), providing a transformation matrix (T) which includes information regarding a correspondence between 3-dimensional points (Pi*) associated with the real object (3) and corresponding 2-dimensional points (or 3-dimensional points) (p,) of the real object (5) as included in the 2-dimensional (or 3-dimensional) image (4), and determining an initial estimate of the transformation matrix (Tl) as an initial basis for an iterative minimization process used for iteratively refining the transformation matrix, determining a Jacobian matrix (J) which includes information regarding the initial estimate of the transformation matrix (Tl) and reference values of 3-dimensional points (Pi*) associated with the real object (3). Further, in the iterative minimization process, in each one of multiple iteration loops determining a respective updated version of the transformation matrix (T) based on a respective previous version of the transformation matrix (T) and based on the Jacobian matrix (J), wherein the Jacobian matrix is not updated during the iterative minimization process, and determining the pose of the camera (1) with respect to the real object (3) using the transformation matrix (T) determined at the end of the iterative minimization process. As a result, the camera pose can be calculated with rather low computational time.

Claims

1. A method for determining the pose of a camera with respect to at least one real object, the method comprising the steps of:

operating the camera for capturing a 2-dimensional image including at least a part of the real object;

providing a transformation matrix (T) which includes information regarding a correspondence between 3-dimensional points (Pi*) associated with the real object and corresponding 2-dimensional points (p_i) of the real object as included in the 2-dimensional image;

determining an initial estimate of the transformation matrix (Tl) as an initial basis for an iterative minimization process used for iteratively refining the transformation matrix;

determining a Jacobian matrix (J) which includes information regarding the initial estimate of the transformation matrix (Tl) and reference values of 3-dimensional points (Pi*) associated with the real object;

in the iterative minimization process, in each one of multiple iteration loops determining a respective updated version of the transformation matrix (T) based on a respective previous version of the transformation matrix (T) and based on the Jacobian matrix (J), wherein the Jacobian matrix is not updated during the iterative minimization process; and

determining the pose of the camera with respect to the real object using the transformation matrix (T) determined at the end of the iterative minimization process.

3. The method of claim 1, wherein the iterative minimization process involves the algorithm of Gauss-Newton or the algorithm of Levenberg-Marquardt.

4. The method of claim 1, wherein the iterative minimization process involves a compositional update process in which the respective updated version of the transformation matrix (T) is computed from a multiplication between a matrix (Δ

T(d)) built with update parameters of the respective previous transformation matrix and the respective previous transformation matrix (T).

5. The method of claim 1, wherein the iterative minimization process involves the algorithm of Gauss-Newton, the iterative minimization process in each iteration loop comprising the steps of:

determining an error vector e;

determining an update vector d by solving the system;

J d=e, with J being the Jacobian matrix; and

determining a respective updated version of the transformation matrix based on a respective previous version of the transformation matrix at the beginning of the respective iteration loop and based on the update vector d;

wherein after determining the respective updated version of the transformation matrix the iterative minimization process returns to determining a new error vector e based on the respective updated version of the transformation matrix.

6. The method of claim 5, wherein the update vector d is computed from:

d=(J^TJ)<sup>−

1</sup>J^Te=J_pe with J_p being computed at the beginning of the iterative minimization process and kept fixed in multiple iteration loops during the iterative minimization process.

7. The method of claim 1, wherein the iterative minimization process involves the algorithm of Levenberg-Marquardt, the iterative minimization process in each iteration loop comprising the steps of:

determining an error vector e;

determining an update vector d by solving the system;

(J^TJ+λ

Id)d=J^Te, with J being the Jacobian matrix; and

determining a respective updated version of the transformation matrix based on a respective previous version of the transformation matrix at the beginning of the respective iteration loop and based on the update vector d;

wherein after determining the respective updated version of the transformation matrix the iterative minimization process returns to determining a new error vector e based on the respective updated version of the transformation matrix or determining a new update vector d.

8. The method of claim 7, wherein the update vector d is computed from:

d=(Q+λ

Id)<sup>−

1</sup> J^Te, with Q=J^TJ being computed at the beginning of the iterative minimization process and kept fixed in multiple iteration loops during the iterative minimization process.

9. The method of claim 1, wherein in the iterative minimization process Lie Algebra parameterization is used.

10. The method of claim 5, further comprising

determining an update matrix Δ

T(d) using the update vector d,

wherein the update matrix Δ

T(d) is parameterized using Lie Algebra.

11. The method of claim 10, wherein the update matrix Δ

T(d) is parameterized using Lie Algebra se(3) associated to the Lie group SE(3).

12. A computer program product adapted to be loaded into the internal memory of a computer, which is coupled with a camera for taking images of a real environment, and comprising software code sections by means of which the steps according to claim 1 are performed when said product is running on said computer.

2. A method for determining the pose of a camera with respect to at least one real object, the method comprising the steps of:

operating the camera for capturing a 3-dimensional image including at least a part of the real object;

providing a transformation matrix (T) which includes information regarding a correspondence between 3-dimensional points (Pi*) associated with the real object and corresponding 3-dimensional points (p_i) of the real object as included in the 3-dimensional image;

determining an initial estimate of the transformation matrix (Tl) as an initial basis for an iterative minimization process used for iteratively refining the transformation matrix,

determining a Jacobian matrix (J) which includes information regarding the initial estimate of the transformation matrix (Tl) and reference values of 3-dimensional points (Pi*) associated with the real object;

in the iterative minimization process, in each one of multiple iteration loops determining a respective updated version of the transformation matrix (T) based on a respective previous version of the transformation matrix (T) and based on the Jacobian matrix (J), wherein the Jacobian matrix is not updated during the iterative minimization process; and

determining the pose of the camera with respect to the real object using the transformation matrix (T) determined at the end of the iterative minimization process.

13. The method of claim 2, wherein

the iterative minimization process involves the algorithm of Gauss-Newton or the algorithm of Levenberg-Marquardt.

14. The method of claim 2, wherein the iterative minimization process involves a compositional update process in which the respective updated version of the transformation matrix (T) is computed from a multiplication between a matrix (Δ

T(d)) built with update parameters of the respective previous transformation matrix and the respective previous transformation matrix (T).

15. The method of claim 2, wherein the iterative minimization process involves the algorithm of Gauss-Newton, the iterative minimization process in each iteration loop comprising the steps of:

determining an error vector e;

determining an update vector d by solving the system;

J d=e, with J being the Jacobian matrix; and

determining a respective updated version of the transformation matrix based on a respective previous version of the transformation matrix at the beginning of the respective iteration loop and based on the update vector d;

wherein after determining the respective updated version of the transformation matrix the iterative minimization process returns to determining a new error vector e based on the respective updated version of the transformation matrix.

16. The method claim 2, wherein the iterative minimization process involves the algorithm of Levenberg-Marquardt, the iterative minimization process in each iteration loop comprising the steps of:

determining an error vector e;

determining an update vector d by solving the system;

(J^TJ+λ

Id)d=J^Te, with J being the Jacobian matrix; and

determining a respective updated version of the transformation matrix based on a respective previous version of the transformation matrix at the beginning of the respective iteration loop and based on the update vector d;

wherein after determining the respective updated version of the transformation matrix the iterative minimization process returns to determining a new error vector e based on the respective updated version of the transformation matrix or determining a new update vector d.

17. The method claim 2, wherein in the iterative minimization process Lie Algebra parameterization is used.

18. A computer program product adapted to be loaded into the internal memory of a computer, which is coupled with a camera for taking images of a real environment, and comprising software code sections by means of which the steps according to claim 2 are performed when said product is running on said computer.