Dynamic extension of map data for object detection and tracking

• US 9,811,731 B2
• Filed: 10/02/2014
• Issued: 11/07/2017
• Est. Priority Date: 10/04/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Lee, Wonwoo, Pan, Qi, Maggio, Emilio, Tallonneau, Romain
• Primary Examiner(s): PARK, SOO JIN
• Application Number:
  US14/505,345
• Publication Number:
  US 20150098615A1
• Time in Patent Office: 1,132 Days
• Field of Search: None
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

A computer-implemented method of tracking a target object in an object recognition system includes acquiring a plurality of images with a camera and simultaneously tracking the target object and dynamically building online map data from the plurality of images. Tracking of the target object is based on the online map data and the offline map data. In one aspect, tracking the target object includes enabling only one of the online map data and offline map data for tracking based on whether tracking is successful. In another aspect, tracking the target object includes fusing the online map data with the offline map data to generate a fused online model.

Claims

1. A computer-implemented method of tracking a target object in an object recognition system, the method comprising:

acquiring a plurality of images with a camera;

acquiring offline map data of the target object; and

simultaneously tracking the target object and dynamically building online map data from the plurality of images, wherein tracking the target object includes selecting between the online map data and the offline map data for tracking the target object based at least in part on a size of the target object relative to an image in which the target object is being tracked.

2. The computer-implemented method of claim 1, wherein tracking the target object includes:

estimating a target pose of the target object with respect to the camera, wherein the target pose estimation may be successful or unsuccessful;

determining whether the target pose estimation is successful; and

if so,

enabling only one of the online map data and offline map data for tracking of the target object in a subsequent image.

3. The computer-implemented method of claim 2, further comprising enabling both online map data and offline map data, if the tracking of the target object is not successful, for tracking of the target object in the subsequent image.

4. The computer-implemented method of claim 2, wherein enabling only one of the online map data and the offline map data includes:

computing a region size of the target object;

enabling the offline map data and disabling the online map data if the region size of the target object is greater than a region threshold; and

enabling the online map data and disabling the offline map data if the region size of the target object is not greater than the region threshold.

5. The computer-implemented method of claim 4, wherein computing the region size of the target object includes projecting a bounding box of offline map data features onto an acquired image.

6. The computer-implemented method of claim 4, wherein the region threshold corresponds to the target object assuming at least half of an image.

7. The computer-implemented method of claim 1, wherein the online map data is based on one or more keyframes corresponding with the plurality of images acquired with the camera.

8. The computer-implemented method of claim 1, wherein building the online map data includes keyframe Simultaneous Localization and Mapping (SLAM).

9. The computer-implemented method of claim 1, wherein building the online map data includes adding a candidate keyframe to one or more keyframes by computing a pose distance between the candidate keyframe and other online keyframes corresponding to images acquired by the camera.

10. The computer-implemented method of claim 1, further comprising fusing the online map data with the offline map data to generate a fused online model of the target object and wherein tracking of the target object uses the fused online model when online map data is selected for tracking the target object.

11. The computer-implemented method of claim 10, further comprising fusing the online map data with the offline map data including:

extracting one or more online features from at least one of the plurality of acquired images;

comparing an online feature with offline features included in the offline map data;

updating a descriptor of an offline feature and adding the updated offline feature to the fused online model if both the online feature and the offline feature correspond to a same 3D point of the target object; and

adding the online feature to the fused online model if the online feature corresponds to a new 3D point on the target object, where the new 3D point does not correspond to any offline feature.

12. A non-transitory computer-readable medium including program code stored thereon for tracking a target object in an object recognition system, the program code comprising instructions to:

acquire a plurality of images with a camera;

acquire offline map data of the target object; and

simultaneously track the target object and dynamically building online map data from the plurality of images, wherein the instruction to track the target object includes instruction to select between the online map data and the offline map data for tracking the target object based at least in part on a size of the target object relative to an image in which the target object is being tracked.

13. The non-transitory computer-readable medium of claim 12, wherein the instructions to track the target object includes instructions to:

estimate a target pose of the target object with respect to the camera, wherein the target pose estimation may be successful or unsuccessful;

determine whether the target pose estimation is successful; and

if so,

enable only one of the online map data and offline map data for tracking of the target object in a subsequent image.

14. The non-transitory computer-readable medium of claim 13, further comprising instructions to enable both online map data and offline map data if the tracking of the target object is not successful for tracking of the target object in the subsequent image.

15. The non-transitory computer-readable medium of claim 13, wherein enabling only one of the online map data and the offline map data includes instructions to:

compute a region size of the target object;

enable the offline map data and disabling the online map data if the region size of the target object is greater than a region threshold; and

enable the online map data and disable the offline map data if the region size of the target object is not greater than the region threshold.

16. The non-transitory computer-readable medium of claim 15, wherein the instructions to compute the region size of the target object includes instructions to project a bounding box of offline map data features onto an acquired image.

17. The non-transitory computer-readable medium of claim 15, wherein the region threshold corresponds to the target object assuming at least half of an image.

18. The non-transitory computer-readable medium of claim 12, wherein the online map data is based on one or more keyframes corresponding with the plurality of images acquired with the camera.

19. The non-transitory computer-readable medium of claim 12, wherein the instructions to build the online map data includes keyframe Simultaneous Localization and Mapping (SLAM).

20. The non-transitory computer-readable medium of claim 12, wherein the instructions to build the online map data includes instructions to add a candidate keyframe to one or more keyframes by computing a pose distance between the candidate keyframe and other online keyframes corresponding to images acquired by the camera.

21. The non-transitory computer-readable medium of claim 12, further comprising instructions to fuse the online map data with the offline map data to generate a fused online model of the target object and wherein tracking of the target object uses the fused online model when online map data is selected for tracking the target object.

22. The non-transitory computer-readable medium of claim 21, further comprising instructions to fuse the online map data with the offline map data including instructions to:

extract one or more online features from at least one of the plurality of acquired images;

compare the one or more online features with offline features included in the offline map data;

update a descriptor of an offline feature and add the updated offline feature to the fused online model if both the online feature and the offline feature correspond to a same 3D point of the target object; and

add the online feature to the fused online model if the online feature corresponds to a new 3D point on the target object, where the new 3D point does not correspond to any offline feature.

23. An apparatus, comprising:

memory adapted to store program code for tracking a target object in an object recognition system; and

one or more processors adapted to access and execute instructions included in the program code, wherein when the instructions are executed by the one or more processors, the one or more processors direct the apparatus to;

acquire a plurality of images with a camera;

acquire offline map data of the target object; and

simultaneously track the target object and dynamically building online map data from the plurality of images, wherein the instruction to track the target object includes instructions to select between the online map data and the offline map data for tracking the target object based at least in part on a size of the target object relative to an image in which the target object is being tracked.

24. The apparatus of claim 23, wherein the instructions to track the target object includes instructions to:

estimate a target pose of the target object with respect to the camera, wherein the target pose estimation may be successful or unsuccessful;

determine whether the target object pose estimation is successful; and

if so,

enable only one of the online map data and offline map data for tracking of the target object in a subsequent image.

25. The apparatus of claim 24, wherein enabling only one of the online map data and the offline map data includes instructions to:

compute a region size of the target object;

enable the offline map data and disabling the online map data if the region size of the target object is greater than a region threshold; and

enable the online map data and disable the offline map data if the region size of the target object is not greater than the region threshold.

26. The apparatus of claim 23, further comprising instructions to fuse the online map data with the offline map data to generate a fused online model of the target object and wherein tracking of the target object uses the fused online model when online map data is selected for tracking the target object.

27. The apparatus of claim 26, further comprising instructions to fuse the online map data with the offline map data including instructions to:

extract one or more online features from at least one of the plurality of acquired images;

compare the one or more online features with offline features included in the offline map data;

update a descriptor of an offline feature and add the updated offline feature to the fused online model if both the online feature and the offline feature correspond to a same 3D point of the target object; and

add the online feature to the fused online model if the online feature corresponds to a new 3D point on the target object, where the new 3D point does not correspond to any offline feature.

28. An apparatus, comprising:

means for acquiring a plurality of images with a camera;

means for acquiring offline map data of the target object; and

means for simultaneously tracking the target object and dynamically building online map data from the plurality of images, wherein means for tracking the target object includes means for selecting between the online map data and the offline map data for tracking the target object based at least in part on a size of the target object relative to an image in which the target object is being tracked.

29. The apparatus of claim 28, further comprising means for fusing the online map data with the offline map data to generate a fused online model of the target object including:

means for extracting one or more online features from at least one of the plurality of acquired images;

means for comparing the one or more online features with offline features included in the offline map data;

means for updating a descriptor of an offline feature and adding the updated offline feature to the fused online model if both the online feature and the offline feature correspond to a same 3D point of the target object; and

means for adding the online feature to the fused online model if the online feature corresponds to a new 3D point on the target object, where the new 3D point does not correspond to any offline feature.

30. The apparatus of claim 28, wherein the means for tracking the target object includes:

means for estimating a target pose of the target object with respect to the camera, wherein the target pose estimation may be successful or unsuccessful;

means for determining whether the target pose estimation is successful; and

means for enabling only one of the online map data and offline map data for tracking of the target object in a subsequent image if tracking of the target object is successful.