PLANAR MAPPING AND TRACKING FOR MOBILE DEVICES

• US 20120300979A1
• Filed: 05/03/2012
• Published: 11/29/2012
• Est. Priority Date: 05/27/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Reitmayr, Gerhard, Pirchheim, Christian
• Granted Patent: US 9,020,187 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

Real time tracking and mapping is performed using images of unknown planar object. Multiple images of the planar object are captured. A new image is selected as a new keyframe. Homographies are estimated for the new keyframe and each of a plurality of previous keyframes for the planar object that are spatially distributed. A graph structure is generated using the new keyframe and each of the plurality of previous keyframes and the homographies between the new keyframe and each of the plurality of previous keyframes. The graph structure is used to create a map of the planar object. The planar object is tracked based on the map and subsequently captured images.

Claims

1. A method comprising:

capturing multiple images of a planar object;

selecting a new image as a new keyframe;

calculating homographies between the new keyframe and each of a plurality of previous keyframes for the planar object that are spatially distributed;

generating a graph structure using the new keyframe and each of the plurality of previous keyframes and the homographies between the new keyframe and each of the plurality of previous keyframes;

using the graph structure to create a map of the planar object; and

tracking the planar object based on the map and subsequently captured images.

2. The method of claim 1, wherein the graph structure is a tree based graph structure and the new keyframe and each of the plurality of previous keyframes are nodes and the homographies between the new keyframe and each of the plurality of previous keyframes are edges between the nodes.

3. The method of claim 1, wherein using the graph structure to create the map of the planar object comprises:

selecting a reference keyframe from the plurality of previous keyframes;

calculating a pose for the reference keyframe with respect to the planar object;

calculating poses for a remaining set of keyframes using the pose of the reference keyframe and homographies between the reference keyframe and the remaining set of keyframes; and

using the pose of the reference keyframe and the poses for the remaining set of keyframes to create the map.

4. The method of claim 3, wherein the reference keyframe is selected based on one of closeness to the new keyframe and closeness to a center node of the graph structure.

5. The method of claim 1, wherein selecting the new image as the new keyframe comprises comparing the new image to the plurality of previous keyframes to determine an overlap of the new image with the plurality of previous keyframes and comparing the overlap to a threshold.

6. The method of claim 1, wherein selecting the new image as the new keyframe comprises verifying that the new image is of the planar object.

7. The method of claim 6, wherein verifying that the new image is of the planar object comprises:

determining multiple poses for the new image based on poses of adjacent keyframes and corresponding homographies between the new image and the adjacent keyframes;

comparing the multiple poses to a tracker pose for the new image produced from tracking the new image with respect to the map; and

determining a sum of pairwise pose differences between the multiple poses and the tracker pose is within a threshold indicating that the new image is of the planar object.

8. An apparatus comprising:

a camera capable of capturing multiple images of a planar object; and

a processor coupled to the camera, the processor configured to select a new image as a new keyframe, calculate homographies between the new keyframe and each of a plurality of previous keyframes for the planar object that are spatially distributed, generate a graph structure using the new keyframe and each of the plurality of previous keyframes and the homographies between the new keyframe and each of the plurality of previous keyframes, use the graph structure to create a map of the planar object, and track the planar object based on the map and subsequently captured images.

9. The apparatus of claim 8, wherein the graph structure is a tree based graph structure and the new keyframe and each of the plurality of previous keyframes are nodes and the homographies between the new keyframe and each of the plurality of previous keyframes are edges between the nodes.

10. The apparatus of claim 8, wherein the processor is configured to use the graph structure to create the map of the planar object by being configured to select a reference keyframe from the plurality of previous keyframes, calculate a pose for the reference keyframe with respect to the planar object, calculate poses for a remaining set of keyframes using the pose of the reference keyframe and homographies between the reference keyframe and the remaining set of keyframes, and use the pose of the reference keyframe and the poses for the remaining set of keyframes to create the map.

11. The apparatus of claim 10, wherein the processor is configured to select the reference keyframe based on one of closeness to the new keyframe and closeness to a center node of the graph structure.

12. The apparatus of claim 8, wherein the processor is configured to select the new image as the new keyframe by being configured to compare the new image to the plurality of previous keyframes to determine an overlap of the new image with the plurality of previous keyframes and compare the overlap to a threshold.

13. The apparatus of claim 8, wherein the processor is configured to select the new image as the new keyframe by being configured to verify that the new image is of the planar object.

14. The apparatus of claim 13, wherein the processor is configured to verify that the new image is of the planar object by being configured to determine multiple poses for the new image based on poses of adjacent keyframes and corresponding homographies between the new image and the adjacent keyframes, compare the multiple poses to a tracker pose for the new image produced from tracking the new image with respect to the map, and determine a sum of pairwise pose differences between the multiple poses and the tracker pose is within a threshold indicating that the new image is of the planar object.

15. An apparatus comprising:

means for capturing multiple images of a planar object;

means for selecting a new image as a new keyframe;

means for calculating homographies between the new keyframe and each of a plurality of previous keyframes for the planar object that are spatially distributed;

means for generating a graph structure using the new keyframe and each of the plurality of previous keyframes and the homographies between the new keyframe and each of the plurality of previous keyframes;

means for using the graph structure to create a map of the planar object; and

means for tracking the planar object based on the map and subsequently captured images.

16. The apparatus of claim 15, wherein the graph structure is a tree based graph structure and the new keyframe and each of the plurality of previous keyframes are nodes and the homographies between the new keyframe and each of the plurality of previous keyframes are edges between the nodes.

17. The apparatus of claim 15, wherein the means for using the graph structure to create the map of the planar object comprises:

means for selecting a reference keyframe from the plurality of previous keyframes;

means for calculating a pose for the reference keyframe with respect to the planar object;

means for calculating poses for a remaining set of keyframes using the pose of the reference keyframe and homographies between the reference keyframe and the remaining set of keyframes; and

means for using the pose of the reference keyframe and the poses for the remaining set of keyframes to create the map.

18. The apparatus of claim 15, wherein the means for selecting the new image as the new keyframe comprises means for comparing the new image to the plurality of previous keyframes to determine an overlap of the new image with the plurality of previous keyframes and means for comparing the overlap to a threshold.

19. The apparatus of claim 15, wherein the means for selecting the new image as the new keyframe comprises means for verifying that the new image is of the planar object.

20. The apparatus of claim 19, wherein the means for verifying that the new image is of the planar object comprises:

means for determining multiple poses for the new image based on poses of adjacent keyframes and corresponding homographies between the new image and the adjacent keyframes;

means for comparing the multiple poses to a tracker pose for the new image produced from tracking the new image with respect to the map; and

means for determining a sum of pairwise pose differences between the multiple poses and the tracker pose is within a threshold indicating that the new image is of the planar object.

21. A non-transitory computer-readable medium including program code stored thereon, comprising:

program code to select from captured multiple images of a planar object a new image as a new keyframe;

program code to calculate homographies between the new keyframe and each of a plurality of previous keyframes for the planar object that are spatially distributed;

program code to generate a graph structure using the new keyframe and each of the plurality of previous keyframes and the homographies between the new keyframe and each of the plurality of previous keyframes;

program code to use the graph structure to create a map of the planar object; and

program code to track the planar object based on the map and subsequently captured images.

22. The non-transitory computer-readable medium of claim 21, wherein the graph structure is a tree based graph structure and the new keyframe and each of the plurality of previous keyframes are nodes and the homographies between the new keyframe and each of the plurality of previous keyframes are edges between the nodes.

23. The non-transitory computer-readable medium of claim 21, wherein the program code to use the graph structure to create the map of the planar object comprises:

program code to select a reference keyframe from the plurality of previous keyframes;

program code to calculate a pose for the reference keyframe with respect to the planar object;

program code to calculate poses for a remaining set of keyframes using the pose of the reference keyframe and homographies between the reference keyframe and the remaining set of keyframes; and

program code to use the pose of the reference keyframe and the poses for the remaining set of keyframes to create the map.

24. The non-transitory computer-readable medium of claim 21, wherein the program code to select the new image as the new keyframe comprises program code to compare the new image to the plurality of previous keyframes to determine an overlap of the new image with the plurality of previous keyframes and program code to compare the overlap to a threshold.

25. The non-transitory computer-readable medium of claim 21, wherein the program code to select the new image as the new keyframe comprises program code to verify that the new image is of the planar object.

26. The non-transitory computer-readable medium of claim 25, wherein the program code to verify that the new image is of the planar object comprises:

program code to determine multiple poses for the new image based on poses of adjacent keyframes and corresponding homographies between the new image and the adjacent keyframes;

program code to compare the multiple poses to a tracker pose for the new image produced from tracking the new image with respect to the map; and

program code to determine a sum of pairwise pose differences between the multiple poses and the tracker pose is within a threshold indicating that the new image is of the planar object.