Real time tracking/detection of multiple targets

• US 9,135,514 B2
• Filed: 05/20/2011
• Issued: 09/15/2015
• Est. Priority Date: 05/21/2010
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Wagner, Daniel, Schmalstieg, Dieter
• Primary Examiner(s): NAKHJAVAN, SHERVIN K
• Application Number:
  US13/112,890
• Publication Number:
  US 20110286631A1
• Time in Patent Office: 1,579 Days
• Field of Search: 382/103, 348/169
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks

Abstract

A mobile platform detects and tracks at least one target in real-time, by tracking at least one target, and creating an occlusion mask indicating an area in a current image to detect a new target. The mobile platform searches the area of the current image indicated by the occlusion mask to detect the new target. The use of a mask to instruct the detection system where to look for new targets increases the speed of the detection task. Additionally, to achieve real-time operation, the detection and tracking is performed in the limited time budget of the (inter) frame duration. Tracking targets is given higher priority than detecting new targets. After tracking is completed, detection is performed in the remaining time budget for the frame duration. Detection for one frame, thus, may be performed over multiple frames.

Claims

1. A method of real-time pose estimation and tracking, the method comprising:

tracking at least one target;

creating an occlusion mask based on the at least one tracked target; and

searching the area of the current image indicated by the occlusion mask, which comprises pixels in the current image that are not covered by the occlusion mask, to detect a new target.

2. The method of claim 1, wherein tracking at least one target comprises matching features in the current image to features from a reference image, and wherein creating the occlusion mask based on the at least one tracked target comprises:

generating a plurality of sub-masks, each sub-mask is generated around each feature in the current image that is matched to a feature from the reference image; and

combining all of the plurality of sub-masks to create the occlusion mask.

3. The method of claim 1, wherein searching the area of the current image indicated by the occlusion mask to detect the new target comprises:

detecting keypoints in the area of the current image indicated by the occlusion mask;

describing and matching keypoints in the current image with keypoints from a reference image;

removing outliers and estimating a pose for the new target.

4. The method of claim 3, wherein detecting keypoints in the area of the current image indicated by the occlusion mask comprises searching pixels in the current image that are not covered by the occlusion mask.

5. The method of claim 1, wherein tracking at least one target comprises:

tracking a plurality of targets in each image in a live video stream by matching a number of features in each image to features from a plurality of reference images, wherein the number of features matched in each image is varied to complete the tracking during each frame of the live video stream; and

wherein searching the area of the current image comprises searching the area of the current image from the live video stream over a plurality of frames from the live video stream.

6. An apparatus comprising:

a camera;

a processor connected to the camera configured to receive images from the camera in a live video stream;

a display controlled by the processor; and

memory in communication with the processor, wherein the memory is configured to store instructions configured to cause the processor to track at least one target, to create an occlusion mask based on the at least one tracked target, and search the area of the current image indicated by the occlusion mask, which comprises pixels in the current image that are not covered by the occlusion mask, to detect a new target.

7. The apparatus of claim 6, wherein the instructions configured to cause the processor to track at least one target by matching features in the current image to features from a reference image, and wherein the instructions configured to cause the processor to create the occlusion mask based on the at least one tracked target by causing the processor to:

generate a plurality of sub-masks, each sub-mask is generated around each feature in the current image that is matched to a feature from the reference image; and

combine all of the plurality of sub-masks to create the occlusion mask.

8. The apparatus of claim 6, wherein the instructions configured to cause the processor to search the area of the current image indicated by the occlusion mask to detect the new target by causing the processor to:

detect keypoints in the area of the current image indicated by the occlusion mask;

describe and match keypoints in the current image with keypoints from a reference image;

remove outliers and estimate a pose for the new target.

9. The apparatus of claim 8, wherein the instructions configured to cause the processor searches pixels in the current image that are not covered by the occlusion mask to detect keypoints in the area of the current image indicated by the occlusion mask.

10. The apparatus of claim 6, wherein the instructions configured to cause the processor to track a plurality of targets in each image in the live video stream by matching a number of features in each image to features from a plurality of reference images, wherein the number of features matched in each image is varied to complete the tracking during each frame of the live video stream;

and

wherein the instructions configured to cause the processor to search the area of the current image from the live video stream over a plurality of frames from the live video stream.

11. A system for real-time pose estimation and tracking, the system comprising:

means for tracking at least one target;

means for creating an occlusion mask based on the at least one tracked target; and

means for searching the area of the current image indicated by the occlusion mask, which comprises pixels in the current image that are not covered by the occlusion mask, to detect a new target.

12. The system of claim 11, wherein the means for tracking at least one target matches features in the current image to features from a reference image, and the means for creating the occlusion mask based on the at least one tracked target comprises:

means for generating a plurality of sub-masks, each sub-mask is generated around each feature in the current image that is matched to a feature from the reference image; and

means for combining all of the plurality of sub-masks to create the occlusion mask.

13. The system of claim 11, wherein the means for searching the area of the current image indicated by the occlusion mask to detect the new target comprises:

means for detecting keypoints in the area of the current image indicated by the occlusion mask;

means for describing and matching keypoints in the current image with keypoints from a reference image;

means for removing outliers and estimating a pose for the new target.

14. The system of claim 11, wherein the means for tracking at least one target tracks a plurality of targets in each image in a live video stream by matching a number of features in each image to features from a plurality of reference images, wherein the number of features matched in each image is varied to complete the tracking during each frame of the live video stream;

and

the means for searching the area of the current image searches the area of the current image from the live video stream over a plurality of frames from the live video stream.

15. A non-transitory computer-readable medium including program code stored thereon, comprising:

program code to track at least one target;

program code to create an occlusion mask based on the at least one tracked target; and

program code to search the area of the current image indicated by the occlusion mask, which comprises pixels in the current image that are not covered by the occlusion mask, to detect a new target.

16. The computer-readable medium of claim 15, wherein the program code to track at least one target includes program code to match features in the current image to features from a reference image, and wherein the program code to create the occlusion mask based on the at least one tracked target includes:

program code to generate a plurality of sub-masks, each sub-mask is generated around each feature in the current image that is matched to a feature from the reference image; and

program code to combine all of the plurality of sub-masks to create the occlusion mask.

17. The computer-readable medium of claim 15, wherein the program code to search the area of the current image indicated by the occlusion mask to detect the new target includes:

program code to detect keypoints in the area of the current image indicated by the occlusion mask;

program code to describe and match keypoints in the current image with keypoints from a reference image;

program code to remove outliers and estimate a pose for the new target.

18. The computer-readable medium of claim 15, wherein the program code to track at least one target includes program code to track a plurality of targets in each image in a live video stream by matching a number of features in each image to features from a plurality of reference images, wherein the number of features matched in each image is varied to complete the tracking during each frame of the live video stream;

and

wherein the program code to search the area of the current image includes program code to search the area of the current image from the live video stream over a plurality of frames from the live video stream.

19. A method of real-time pose estimation and tracking, the method comprising:

determining a time budget based in part on a frame duration;

tracking at least one target in each frame in a live video stream by matching a number of features in each frame to features from a plurality of reference images, wherein the tracking the target in each frame comprises varying the number of features to be matched based on the time budget;

determining remaining time in the time budget after the tracking the at least one target; and

detecting a new target in one frame from the live video stream during a plurality of frames in the live video stream, wherein the detecting the new target is at least partially performed during the remaining time.

20. The method of claim 19, further comprising detecting a second new target in a subsequent frame from the live video stream after completing the detecting the new target in the one frame, wherein no target detection is performed on the plurality of frames between the one frame and the subsequent frame.

21. The method of claim 19, further comprising varying the number of features to be matched multiple times during each frame.

22. An apparatus comprising:

a camera;

a processor connected to the camera configured to receive frames from the camera in a live video stream;

a display controlled by the processor; and

memory in communication with the processor, wherein the memory is configured to store instructions configured to cause the processor to determine a time budget based in part on a frame duration, to track at least one target in each frame in the live video stream by matching a number of features in each frame to features from a plurality of reference images, wherein the processor varies the number of features to be matched based on the time budget, to determine a remaining time in the time budget after the tracking the at least one target; and

to detect a new target in one frame from the live video stream during a plurality of frames in the live video stream, wherein the detecting the new target is at least partially performed during the remaining time.

23. The apparatus of claim 22, wherein the instructions configured to cause the processor to detect a second new target in a subsequent frame from the live video stream, wherein no target detection is performed on the plurality of frames between the one frame and the subsequent frame.

24. The apparatus of claim 22, wherein the processor varies the number of features to be matched multiple times during each frame.

25. A system of real-time pose estimation and tracking, the system comprising:

means for determining a time budget based in part on a frame duration;

means for tracking at least one target in each frame in a video stream by matching a number of features in each frame to features from a plurality of reference images, wherein the number of features to be matched is varied based on the time budget;

determining remaining time in the time budget after the tracking the at least one target; and

means for detecting a new target in one frame from the live video stream during a plurality of frames in the live video stream, wherein the detecting the new target is at least partially performed during the remaining time.

26. The system of claim 25, wherein the means for detecting further detects a second new target in a subsequent frame from the live video stream, wherein no target detection is performed on the plurality of frames between the one frame and the subsequent frame.

27. The system of claim 25, wherein the means for tracking varies the number of features to be matched multiple times during each frame.

28. A non-transitory computer-readable medium including program code stored thereon, comprising:

program code to determine a time budget based in part on a frame duration;

program code to track at least one target in each frame in a live video stream by matching a number of features in each frame to features from a plurality of reference images, wherein the number of features to be matched is varied based on the time budget;

program code to determine remaining time in the time budget after the tracking the at least one target; and

program code to detect a new target in one frame from the live video stream during a plurality of frames in the live video stream, wherein the detecting the new target is at least partially performed during the remaining time.

29. The computer-readable medium of claim 28, wherein the program code to detect a new target detects a second new target in a subsequent frame from the live video stream, wherein no target detection is performed on the plurality of frames between the one frame and the subsequent frame.

30. The computer-readable medium of claim 28, wherein the number of features to be matched is varied multiple times during each frame.

31. A method of real-time pose estimation and tracking, the method comprising:

determining a time budget based in part on image acquisition time and frame duration;

tracking at least one target in each frame in a video stream by matching a number of features in each frame to features from a plurality of reference images, wherein the number of features to be matched for the least one target is varied based on the time budget;

determining remaining time in the time budget after the tracking at least one target is completed; and

detecting a new target in one frame from the video stream during a plurality of frames in the video stream, wherein the detecting the new target is partially performed during the remaining time if the remaining time is greater than zero.