DYNAMIC EXTENSION OF MAP DATA FOR OBJECT DETECTION AND TRACKING

• US 20150098615A1
• Filed: 10/02/2014
• Published: 04/09/2015
• Est. Priority Date: 10/04/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: LEE, Wonwoo, PAN, Qi, MAGGIO, Emilio, TALLONNEAU, Romain
• Granted Patent: US 9,811,731 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

A computer-implemented method of tracking a target object in an object recognition system includes acquiring a plurality of images with a camera and simultaneously tracking the target object and dynamically building online map data from the plurality of images. Tracking of the target object is based on the online map data and the offline map data. In one aspect, tracking the target object includes enabling only one of the online map data and offline map data for tracking based on whether tracking is successful. In another aspect, tracking the target object includes fusing the online map data with the offline map data to generate a fused online model.

Claims

1. A computer-implemented method of tracking a target object in an object recognition system, the method comprising:

acquiring a plurality of images with a camera;

acquiring offline map data of the target object; and

simultaneously tracking the target object and dynamically building online map data from the plurality of images, wherein tracking the target object includes tracking the target object based on the online map data and the offline map data.

2. The computer-implemented method of claim 1, wherein tracking the target object based on the online map data and the offline map data includes fusing the online map data with the offline map data to generate a fused online model of the target object and wherein tracking of the target object is based on the fused online model.

3. The computer-implemented method of claim 2, wherein fusing the online map data with the offline map data includes:

extracting one or more online features from at least one of the plurality of acquired images;

comparing an online feature with offline features included in the offline map data;

updating a descriptor of an offline feature and adding the updated offline feature to the fused online model if both the online feature and the offline feature correspond to a same 3D point of the target object; and

adding the online feature to the fused online model if the online feature corresponds to a new 3D point on the target object, where the new 3D point does not correspond to any offline feature.

4. The computer-implemented method of claim 1, wherein tracking the target object based on the online map data and the offline map data includes:

estimating a target pose of the target object with respect to the camera, wherein the target pose estimation may be successful or unsuccessful;

determining whether the target pose estimation is successful; and

if so,

enabling only one of the online map data and offline map data for tracking of the target object in a subsequent image.

5. The computer-implemented method of claim 4, further comprising enabling both online map data and offline map data, if the tracking of the target object is not successful, for tracking of the target object in the subsequent image.

6. The computer-implemented method of claim 4, wherein enabling only one of the online map data and the offline map data includes:

computing a region size of the target object;

enabling the offline map data and disabling the online map data if the region size of the target object is greater than a region threshold; and

enabling the online map data and disabling the offline map data if the region size of the target object is not greater than the region threshold.

7. The computer-implemented method of claim 6, wherein computing the region size of the target object includes projecting a bounding box of offline map data features onto an acquired image.

8. The computer-implemented method of claim 6, wherein the region threshold corresponds to the target object assuming at least half of an image.

9. The computer-implemented method of claim 1, wherein the online map data is based one or more keyframes corresponding with the plurality of images acquired with the camera.

10. The computer-implemented method of claim 1, wherein building the online map data includes keyframe Simultaneous Localization and Mapping (SLAM).

11. The computer-implemented method of claim 1, wherein building the online map data includes adding a candidate keyframe to the one or more keyframes by computing a pose distance between the candidate keyframe and other online keyframes corresponding to images acquired by the camera.

12. A computer-readable medium including program code stored thereon for tracking a target object in an object recognition system, the program code comprising instructions to:

acquire a plurality of images with a camera;

acquire offline map data of the target object; and

simultaneously track the target object and dynamically building online map data from the plurality of images, wherein the instructions to track the target object includes instructions to track the target object based on the online map data and the offline map data.

13. The computer-readable medium of claim 12, wherein the instructions to track the target object based on the online map data and the offline map data includes instructions to fuse the online map data with the offline map data to generate a fused online model of the target object and wherein tracking of the target object is based on the fused online model.

14. The computer-readable medium of claim 13, wherein the instructions to fuse the online map data with the offline map data includes instructions to:

extract one or more online features from at least one of the plurality of acquired images;

compare the one or more online features with offline features included in the offline map data;

update a descriptor of an offline feature and add the updated offline feature to the fused online model if both the online feature and the offline feature correspond to a same 3D point of the target object; and

add the online feature to the fused online model if the online feature corresponds to a new 3D point on the target object, where the new 3D point does not correspond to any offline feature.

15. The computer-readable medium of claim 12, wherein the instructions to track the target object based on the online map data and the offline map data includes instructions to:

estimate a target pose of the target object with respect to the camera, wherein the target pose estimation may be successful or unsuccessful;

determine whether the target pose estimation is successful; and

if so, enable only one of the online map data and offline map data for tracking of the target object in a subsequent image.

16. The computer-readable medium of claim 15, further comprising instructions to enable both online map data and offline map data if the tracking of the target object is not successful for tracking of the target object in the subsequent image.

17. The computer-readable medium of claim 15, wherein enabling only one of the online map data and the offline map data includes instructions to:

compute a region size of the target object;

enable the offline map data and disabling the online map data if the region size of the target object is greater than a region threshold; and

enable the online map data and disable the offline map data if the region size of the target object is not greater than the region threshold.

18. The computer-readable medium of claim 17, wherein the instructions to compute the region size of the target object includes instructions to project a bounding box of offline map data features onto an acquired image.

19. The computer-readable medium of claim 17, wherein the region threshold corresponds to the target object assuming at least half of an image.

20. The computer-readable medium of claim 12, wherein the online map data is based one or more keyframes corresponding with the plurality of images acquired with the camera.

21. The computer-readable medium of claim 12, wherein the instructions to build the online map data includes keyframe Simultaneous Localization and Mapping (SLAM).

22. The computer-readable medium of claim 12, wherein the instructions to build the online map data includes instructions to add a candidate keyframe to the one or more keyframes by computing a pose distance between the candidate keyframe and other online keyframes corresponding to images acquired by the camera.

23. An apparatus, comprising:

memory adapted to store program code for tracking a target object in an object recognition system; and

a processing unit adapted to access and execute instructions included in the program code, wherein when the instructions are executed by the processing unit, the processing unit directs the apparatus to;

acquire a plurality of images with a camera;

acquire offline map data of the target object; and

simultaneously track the target object and dynamically building online map data from the plurality of images, wherein the instructions to track the target object includes instructions to track the target object based on the online map data and the offline map data.

24. The apparatus of claim 23, wherein the instructions to track the target object based on the online map data and the offline map data includes instructions to fuse the online map data with the offline map data to generate a fused online model of the target object and wherein tracking of the target object is based on the fused online model.

25. The apparatus of claim 24, wherein the instructions to fuse the online map data with the offline map data includes instructions to:

extract one or more online features from at least one of the plurality of acquired images;

compare the one or more online features with offline features included in the offline map data;

update a descriptor of an offline feature and add the updated offline feature to the fused online model if both the online feature and the offline feature correspond to a same 3D point of the target object; and

add the online feature to the fused online model if the online feature corresponds to a new 3D point on the target object, where the new 3D point does not correspond to any offline feature.

26. The apparatus of claim 23, wherein the instructions to track the target object based on the online map data and the offline map data includes instructions to:

estimate a target pose of the target object with respect to the camera, wherein the target pose estimation may be successful or unsuccessful;

determine whether the target object pose estimation is successful; and

if so,

enable only one of the online map data and offline map data for tracking of the target object in a subsequent image.

27. The apparatus of claim 26, wherein enabling only one of the online map data and the offline map data includes instructions to:

compute a region size of the target object;

enable the offline map data and disabling the online map data if the region size of the target object is greater than a region threshold; and

enable the online map data and disable the offline map data if the region size of the target object is not greater than the region threshold.

28. An apparatus, comprising:

means for acquiring a plurality of images with a camera;

means for acquiring offline map data of the target object; and

means for simultaneously tracking the target object and dynamically building online map data from the plurality of images, wherein means for tracking the target object includes means for tracking the target object based on the online map data and the offline map data.

29. The apparatus of claim 28, wherein the means for tracking the target object based on the online map data and the offline map data includes means for fusing the online map data with the offline map data to generate a fused online model of the target object, wherein the means for fusing the online map data with the offline map data includes:

means for extracting one or more online features from at least one of the plurality of acquired images;

means for comparing the one or more online features with offline features included in the offline map data;

means for updating a descriptor of an offline feature and adding the updated offline feature to the fused online model if both the online feature and the offline feature correspond to a same 3D point of the target object; and

means for adding the online feature to the fused online model if the online feature corresponds to a new 3D point on the target object, where the new 3D point does not correspond to any offline feature.

30. The apparatus of claim 28, wherein the means for tracking the target object based on the online map data and the offline map data includes:

means for estimating a target pose of the target object with respect to the camera, wherein the target pose estimation may be successful or unsuccessful;

means for determining whether the target pose estimation is successful; and

means for enabling only one of the online map data and offline map data for tracking of the target object in a subsequent image if tracking of the target object is successful.