OBJECT TRACKING BASED ON DYNAMICALLY BUILT ENVIRONMENT MAP DATA

• US 20150098614A1
• Filed: 09/24/2014
• Published: 04/09/2015
• Est. Priority Date: 10/04/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: PAN, Qi, GERVAUTZ, Michael, GEE, Andrew
• Granted Patent: US 9,524,434 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

A computer-implemented method of tracking a target object in an object recognition system includes acquiring a plurality of images with a camera. The method further includes simultaneously tracking the target object and dynamically building environment map data from the plurality of images. The tracking of the target object includes attempting to estimate a target pose of the target object with respect to the camera based on at least one of the plurality of images and based on target map data. Next, the method determines whether the tracking of the target object with respect to the camera is successful. If not, then the method includes inferring the target pose with respect to the camera based on the dynamically built environment map data. In one aspect the method includes fusing the inferred target pose with the actual target pose even if tracking is successful to improve robustness.

Claims

1. A computer-implemented method of tracking a target object in an object recognition system, the method comprising:

acquiring a plurality of images with a camera;

simultaneously tracking the target object and dynamically building environment map data from the plurality of images, wherein tracking the target object includes attempting to estimate a target pose of the target object with respect to the camera based on at least one of the plurality of images and based on target map data;

determining whether the tracking of the target object with respect to the camera is successful; and

, if not, then

inferring the target pose with respect to the camera based on the environment map data.

2. The computer-implemented method of claim 1, further comprising:

inferring the target pose with respect to the camera based on the environment data if it is determined that tracking of the target object with respect to the camera is successful; and

fusing the estimated target pose with the inferred target pose.

3. The computer-implemented method of claim 2, wherein fusing the estimated target pose with the inferred target pose includes applying a Kalman filter.

4. The computer-implemented method of claim 1, wherein inferring the target pose with respect to the camera based on the environment map data includes:

estimating an environment pose of the environment with respect to the camera;

estimating a target pose of the target object with respect to the environment; and

inferring the target pose with respect to the camera based on the estimated target pose with respect to the environment.

5. The computer-implemented method of claim 1, wherein the environment map data is based one or more keyframes corresponding with the one or more plurality of images acquired with the camera.

6. The computer-implemented method of claim 5, wherein building the environment map data includes keyframe Simultaneous Localization and Mapping (SLAM).

7. The computer-implemented method of claim 5, wherein building the environment map data includes adding a candidate keyframe to the one or more keyframes by computing a pose difference between the candidate keyframe and other online keyframes corresponding to images acquired by the camera.

8. The computer-implemented method of claim 7, wherein computing the pose difference between the candidate keyframe and other online keyframes includes skipping offline keyframes when computing the pose difference.

9. The computer-implemented method of claim 7, wherein the pose difference is a minimum pose difference between the candidate keyframe and other online keyframes, the method further comprising adding the candidate keyframe to the other online keyframes if the pose difference is more than a pose threshold.

10. A computer-readable medium including program code stored thereon for tracking a target object in an object recognition system, the program code comprising instructions to:

acquire a plurality of images with a camera;

simultaneously track the target object and dynamically build environment map data from the plurality of images, wherein the instructions to track the target object includes instructions to attempt to estimate a target pose of the target object with respect to the camera based on at least one of the plurality of images and based on target map data;

determine if tracking of the target object with respect to the camera is successful; and

, if not, then

infer the target pose with respect to the camera based on the environment map data.

11. The computer-readable medium of claim 10, wherein the instructions to infer the target pose with respect to the camera based on the environment map data includes instructions to:

estimate an environment pose of the environment with respect to the camera;

estimate a target pose of the target object with respect to the environment; and

infer the target pose with respect to the camera based on the estimated target pose with respect to the environment.

12. The computer-readable medium of claim 10, wherein the environment map data is based one or more keyframes corresponding with the one or more plurality of images acquired with the camera.

13. The computer-readable medium of claim 12, wherein the instructions to build the environment map data includes keyframe Simultaneous Localization and Mapping (SLAM).

14. The computer-readable medium of claim 12, wherein the instructions to build the environment map data includes instructions to add a candidate keyframe to the one or more keyframes and to compute a pose difference between the candidate keyframe and other online keyframes corresponding to images acquired by the camera.

15. The computer-readable medium of claim 14, wherein the instructions to compute the pose difference between the candidate keyframe and other online keyframes includes instructions to skip offline keyframes when computing the pose difference.

16. The computer-readable medium of claim 14, wherein the pose difference is a minimum pose difference between the candidate keyframe and other online keyframes, the medium further instructions to add the candidate keyframe to the other online keyframes if the pose difference is more than a pose threshold.

17. An apparatus, comprising:

memory adapted to store program code for tracking a target object in an object recognition system;

a processing unit adapted to access and execute instructions included in the program code, wherein when the instructions are executed by the processing unit, the processing unit directs the apparatus to;

acquire a plurality of images with a camera;

simultaneously track the target object and dynamically build environment map data from the plurality of images, wherein the instructions to track the target object includes instructions to attempt to estimate a target pose of the target object with respect to the camera based on at least one of the plurality of images and based on target map data;

determine if tracking of the target object with respect to the camera is successful; and

, if not, then

infer the target pose with respect to the camera based on the environment map data.

18. The apparatus of claim 17, wherein the instructions to infer the target pose with respect to the camera based on the environment map data includes instructions to:

estimate an environment pose of the environment with respect to the camera;

estimate a target pose of the target object with respect to the environment; and

infer the target pose with respect to the camera based on the estimated target pose with respect to the environment.

19. The apparatus of claim 17, wherein the environment map data is based one or more keyframes corresponding with the one or more plurality of images acquired with the camera.

20. The apparatus of claim 19, wherein the instructions to build the environment map data includes keyframe Simultaneous Localization and Mapping (SLAM).

21. The apparatus of claim 19, wherein the instructions to build the environment map data includes instructions to add a candidate keyframe to the one or more keyframes and to compute a pose difference between the candidate keyframe and other online keyframes corresponding to images acquired by the camera.

22. The apparatus of claim 21, wherein the instructions to compute the pose difference between the candidate keyframe and other online keyframes includes instructions to skip offline keyframes when computing the pose difference.

23. The apparatus of claim 21, wherein the pose difference is a minimum pose difference between the candidate keyframe and other online keyframes, the medium further instructions to add the candidate keyframe to the other online keyframes if the pose difference is more than a pose threshold.

24. The apparatus of claim 17, further comprising the camera to acquire the plurality of images.

25. An apparatus, comprising:

means for acquiring a plurality of images with a camera;

means for simultaneously tracking the target object and dynamically building environment map data from the plurality of images, wherein the means for tracking the target object includes means for attempting to estimate a target pose of the target object with respect to the camera based on at least one of the plurality of images and based on target map data;

means for determining whether the tracking of the target object with respect to the camera is successful; and

means for inferring the target pose with respect to the camera based on the environment map data if the tracking of the target object with respect to the camera is unsuccessful.

26. The apparatus of claim 25, further comprising:

means for inferring the target pose with respect to the camera based on the environment data if it is determined that tracking of the target object with respect to the camera is successful; and

means for fusing the estimated target pose with the inferred target pose.

27. The apparatus of claim 25, wherein the means for inferring the target pose with respect to the camera based on the environment map data includes:

means for estimating an environment pose of the environment with respect to the camera;

means for estimating a target pose of the target object with respect to the environment; and

means for inferring the target pose with respect to the camera based on the estimated target pose with respect to the environment.

28. The apparatus of claim 25, wherein the environment map data is based one or more keyframes corresponding with the one or more plurality of images acquired with the camera, and wherein the means for building the environment map data includes keyframe Simultaneous Localization and Mapping (SLAM).

29. The apparatus of claim 25, wherein the environment map data is based one or more keyframes corresponding with the one or more plurality of images acquired with the camera, and wherein the means for building the environment map data includes means for adding a candidate keyframe to the one or more keyframes by computing a pose difference between the candidate keyframe and other online keyframes corresponding to images acquired by the camera.

30. The apparatus of claim 29, wherein the pose difference is a minimum pose difference between the candidate keyframe and other online keyframes, the apparatus further comprising means for adding the candidate keyframe to the other online keyframes if the pose difference is more than a pose threshold.