System and method for using texture landmarks for improved markerless tracking in augmented reality applications

• US 7,616,807 B2
• Filed: 02/22/2006
• Issued: 11/10/2009
• Est. Priority Date: 02/24/2005
• Status: Active Grant

• Current Assignee: Siemens Medical Solutions USA Incorporated (Siemens AG)
• Original Assignee: Siemens Corporate Research Incorporated (Siemens AG)
• Inventors: Ramesh, Visvanathan, Zhang, Xiang, Genc, Yakup, Tsin, Yanghai
• Primary Examiner(s): Werner; Brain P
• Assistant Examiner(s): DRENNAN, BARRY T
• Application Number:
  US11/359,673
• Publication Number:
  US 20060188131A1
• Time in Patent Office: 1,357 Days
• Field of Search: 382/154, 382/285, 382/103, 348/578, 348/580, 345419-421, 345/427
• US Class Current: 382/154
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

A method of tracking a pose of a moving camera includes receiving a first image from a camera, receiving a sequence of digitized images from said camera, recording, for each of said sequence of digitized images, the pose and 2D correspondences of landmarks, reconstructing a location and appearance of a 2-dimensional texture patch from 2D correspondences of the landmarks by triangulation and optimization, computing back-projection errors by comparing said reconstructed texture patch with said first received image; and reconstructing said location and appearance of said 2-dimensional texture patch from the 2D correspondences of the landmarks of said sequence of digitized images by triangulation and optimization after eliminating those landmarks with large back-projection errors.

Claims

1. A computer implemented method of tracking a pose of a moving camera, the method performed by the computer comprising the steps of:

receiving a first received digitized image from a camera, said image comprising a plurality of intensities corresponding to a domain of points on a 2-dimensional grid, and including one or more landmarks for tracking;

receiving a sequence of digitized images from said camera, each image comprising a plurality of intensities corresponding to a domain of points on a 2-dimensional grid;

recording, for each of said sequence of digitized images, the pose and 2D correspondences of each of said landmarks;

reconstructing a location and appearance of a 2-dimensional texture patch from the 2D correspondences of the landmarks of said sequence of digitized images by triangulation and optimization;

computing back-projection errors by comparing said reconstructed texture patch with said first received image;

reconstructing said location and appearance of said 2-dimensional texture patch from the 2D correspondences of the landmarks of said sequence of digitized images by triangulation and optimization after eliminating those landmarks with large back-projection errors

searching for a best match of each landmark correspondence in said reconstructed texture patch, wherein a search range is determined by a maximum back-projection error, wherein said best match is found by minimizing a sum of square differences

2. The method of claim 1, wherein said sequence of images comprises a plurality of intensities corresponding to a domain of points on a 3-dimensional (3D) grid.

3. The method of claim 1, wherein recording the pose and 2D correspondences of each of said landmarks comprises recording a texture patch about each said landmark, the camera pose from which said texture patch is obtained, and about each said landmark, a set of 3D coordinates of nearby neighbor features.

4. The method of claim 3, wherein said one or more landmarks are selected based on how well its 3D coordinates can be reconstructed, and wherein said landmarks are evenly distributed among the reconstructed landmarks.

5. The method of claim 4, wherein said one or more landmarks are selected based on a reliability measure r comprising the term b_max/(ε

_avg+ε

_std), wherein b_max is a maximum distance between camera centers, ε

_avg is an average back-projection error distance, and ε

_std is a standard deviation of back-projection errors.

6. The method of claim 4, wherein said landmarks are separated by at least a pre-defined threshold distance in the reconstructed image.

7. The method of claim 1, further comprising deforming the landmark feature coordinates i_c and j_c using an approximate affine transformation

i_c=a₁₁(i₀−

n_p/2)+a₁₂(j₀−

n_p/2)+u_c,

j_c=a₂₁(i₀−

n_p/2)+a₂₂(j₀−

n_p/2)+v_c,

wherein u_c and v_c are landmark coordinates, i₀ and j₀ are land mark feature coordinates in the first image, and a₁₁, a₁₂, a₂₁, and a₂₂ are elements of a matrix that back-projects landmark coordinates u_c, v_c to the corresponding coordinates in the first image.

8. The method of claim 7, further comprising using a RANSAC algorithm to eliminate those landmarks with large back-projection errors.

9. The method of claim 1, wherein said landmarks comprise corners and edges of an object depicted in said first image.

10. A program storage device readable by a computer, tangibly embodying a program of instructions executable by the computer to perform the method steps for tracking a pose of a moving camera, said method comprising the steps of:

receiving a first received digitized image from a camera, said image comprising a plurality of intensities corresponding to a domain of points on a 2-dimensional grid, and including one or more landmarks for tracking;

receiving a sequence of digitized images from said camera, each image comprising a plurality of intensities corresponding to a domain of points on a 2-dimensional grid;

recording, for each of said sequence of digitized images, the pose and 2D correspondences of each of said landmarks, wherein recording the pose and 2D correspondences of each of said landmarks comprises recording a texture patch about each said landmark, the camera pose from which said texture catch is obtained, and about each said landmark, a set of 3D coordinates of nearby neighbor features, wherein said one or more landmarks are selected based on how well its 3D coordinates can be reconstructed, and wherein said landmarks are evenly distributed among the reconstructed landmarks based on a reliability measure r comprising the term b_max/(ε

_avg+ε

_std), wherein b_max is a maximum distance between camera centers, ε

_avg is an average back-projection error distance, and E_std is a standard deviation of back-projection errors;

reconstructing a location and appearance of a 2-dimensional texture patch from the 2D correspondences of the landmarks of said sequence of digitized images by triangulation and optimization;

computing back-projection errors by comparing said reconstructed texture patch with said first received image; and

reconstructing said location and appearance of said 2-dimensional texture patch from the 2D correspondences of the landmarks of said sequence of digitized images by triangulation and optimization after eliminating those landmarks with large back-projection errors.

11. The computer readable program storage device of claim 10, wherein said sequence of images comprises a plurality of intensities corresponding to a domain of points on a 3-dimensional (3D) grid.

12. The computer readable program storage device of claim 10, wherein said landmarks are separated by at least a pre-defined threshold distance in the reconstructed image.

13. The computer readable program storage device of claim 10, the method further comprising searching for a best match of each landmark correspondence in said reconstructed texture patch, wherein a search range is determined by a maximum back-projection error, wherein said best match is found by minimizing a sum of square differences

14. The computer readable program storage device of claim 13, the method further comprising deforming the landmark feature coordinates i_c and j_c using an approximate affine transformation

i_c=a₁₁(i₀−

n_p/2)+a₁₂(j₀−

n_p/2)+u_c,

j_c=a₂₁(i₀−

n_p/2)+a₂₂(j₀−

n_p/2)+v_c,

wherein u_c and v_c are landmark coordinates, i₀ and j₀ are landmark feature coordinates in the first image, and a₁₁, a₁₂, a₂₁, and a₂₂ are elements of a matrix that back-projects landmark coordinates u_c, v_c to the corresponding coordinates in the first image.

15. The computer readable program storage device of claim 14, further comprising using a RANSAC algorithm to eliminate those landmarks with large back-projection errors.

16. The computer readable program storage device of claim 13, the method further comprising determining a likelihood score S_l,k for each feature point k of landmark l from

17. The computer readable program storage device of claim 10, wherein said landmarks comprise corners and edges of an object depicted in said first image.