Object tracking and processing

• US 9,349,066 B2
• Filed: 08/06/2012
• Issued: 05/24/2016
• Est. Priority Date: 01/06/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Koo, Hyung-Il, You, Kisun, Baik, Young-Ki
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): TORRES, JOSE
• Application Number:
  US13/567,412
• Publication Number:
  US 20130177203A1
• Time in Patent Office: 1,387 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 30/10 Character recognition

Abstract

A method includes tracking an object in each of a plurality of frames of video data to generate a tracking result. The method also includes performing object processing of a subset of frames of the plurality of frames selected according to a multi-frame latency of an object detector or an object recognizer. The method includes combining the tracking result with an output of the object processing to produce a combined output.

Claims

1. A method comprising:

tracking an object with respect to a plurality of frames of video data to generate a tracking result;

generating an output based on processing a subset of frames of the plurality of frames by an object processor, the object processor including an object detector or an object recognizer, wherein the object is tracked with respect to at least a first frame of the plurality of frames and a second frame of the plurality of frames and the output is generated based on processing the first frame of the plurality of frames; and

after generating the tracking result and the output, combining the tracking result with the output to produce a combined output.

2. The method of claim 1, wherein:

a plurality of tracking results including the tracking result are generated by a tracker at a first frequency which is different than a second frequency at which a plurality of outputs including the output are generated,

the tracking results and the outputs are combined at a temporal filter, and

the temporal filter is accessed asynchronously by the tracker and the object processor.

3. The method of claim 1, wherein:

if the output is generated by performing object detection of the subset of frames, the subset of frames is selected based on a first latency of the object detector; and

if the output is generated by performing object recognition of the subset of frames, the subset of frames is selected based on a second latency of the object recognizer.

4. The method of claim 1, wherein the subset of frames includes the first frame and the second frame of the plurality of frames.

5. The method of claim 1, wherein the subset of frames is selected based on a first latency of the object detector or a second latency of the object recognizer, the first latency corresponding to a period of time for a tracker to perform tracking with respect to at least two frames of the plurality of frames, and the second latency corresponding to a period of time for the object recognizer to process the first frame of the subset of frames.

6. The method of claim 1, further comprising updating state information of the object based on the output, motion history of the object, or any combination thereof.

7. The method of claim 1, wherein tracking the object and generating the output partially overlap in time.

8. The method of claim 7, wherein the object is tracked with respect to at least the first frame of the plurality of frames and the second frame of the plurality of frames while the output is generated based on processing the first frame of the plurality of frames.

9. The method of claim 7, wherein combining the tracking result with the output does not overlap in time with tracking the object to generate the tracking result, and wherein combining the tracking result with the output does not overlap in time with generating the output.

10. The method of claim 1, further comprising updating state information of the object based on the combined output, wherein the state information includes:

a location of the object; and

a definition of the object, a type of the object, historical data of the object, or any combination thereof.

11. The method of claim 1, further comprising updating state information of the object based on the combined output, wherein the object corresponds to a text box, and wherein the state information includes:

recognized text in the text box; and

a definition of the object, a type of the object, historical data of the object, or any combination thereof.

12. The method of claim 11, further comprising:

updating the state information, wherein the state information includes augmented content; and

displaying image content at least partially based on the state information at a display device of an image capture device.

13. The method of claim 1, wherein combining the tracking result with the output comprises integrating the tracking result with the output to obtain temporal information of the object.

14. The method of claim 13, wherein integrating the tracking result with the output comprises generating location information of the object based on the tracking result using at least one of a Kalman filter, a Particle filter, or an Extended Kalman filter.

15. The method of claim 13, wherein combining the tracking result with the output comprises integrating the tracking result with the output, and wherein integrating the tracking result with the output comprises generating identification information including text data of the object based on the output using at least one of a maximum-likelihood estimator or a maximum a posteriori estimator.

16. The method of claim 1, wherein the plurality of frames are generated by an image capture device, and wherein the image capture device comprises a camera, a video recording device, a wireless device, a portable electronic device, or any combination thereof.

17. The method of claim 1, further comprising:

prior to detecting a completion of object processing of the first frame of the subset of frames;

estimating motion of the object between the first frame and the second frame of the subset of frames; and

updating state information of the object based on the motion.

18. The method of claim 1, wherein the subset of frames is selected based on a first latency of the object detector or a second latency of the object recognizer, wherein the object is tracked by a tracker that has a third latency, wherein the third latency is less than the first latency, and wherein the third latency is less than the second latency.

19. The method of claim 1, wherein tracking the object is performed by a tracker to generate a particular tracking result that is not provided to the object processor, and wherein processing the subset of frames is performed by the object processor to generate a particular output that is not provided to the tracker.

20. An apparatus comprising:

a temporal filter;

a tracker configured to track an object with respect to each of a plurality of frames of video data to generate a tracking result and to provide the tracking result to the temporal filter; and

an object processor configured to process the object in a subset of frames of the plurality of frames to generate an output and to provide the output to the temporal filter, wherein the tracker is configured to track the object with respect to at least a first frame of the plurality of frames and a second frame of the plurality of frames and the object processor is configured to generate the output based on processing the first frame of the plurality of frames;

wherein the temporal filter is configured to receive the tracking result from the tracker, wherein the temporal filter is configured to receive the output from the object processor, and wherein the temporal filter is further configured to combine the tracking result and the output to produce a combined output.

21. The apparatus of claim 20, wherein the object processor includes:

an object detector configured to detect the object in the subset of frames; and

an object recognizer configured to recognize the object in the subset of frames.

22. The apparatus of claim 20, wherein the subset of frames is selected based on a latency of the object processor, and wherein the latency of the object processor corresponds to a period of time for the tracker to perform tracking with respect to at least two frames of the plurality of frames.

23. The apparatus of claim 20, wherein the temporal filter is further configured to update state information of the object, wherein the object corresponds to a text box, and wherein the state information includes recognized text in the text box and a location of the text box.

24. The apparatus of claim 23, wherein the temporal filter includes a Kalman filter and a maximum-likelihood estimator.

25. The apparatus of claim 24, wherein the Kalman filter is configured to determine a location of the text, and wherein the location of the text includes coordinates of the text box.

26. The apparatus of claim 24, wherein the maximum-likelihood estimator is configured to:

generate proposed text data via optical character recognition (OCR); and

access a dictionary to verify the proposed text data.

27. The apparatus of claim 26, wherein the proposed text data includes multiple text candidates and confidence data associated with the multiple text candidates, and wherein a particular text candidate corresponding to an entry of the dictionary is selected as verified text according to a confidence value associated with the text candidate.

28. An apparatus comprising:

means for tracking an object with respect to a plurality of frames of video data to generate a tracking result;

means for processing the object in a subset of frames of the plurality of frames to generate an output, wherein the means for tracking is configured to track the object with respect to at least a first frame of the plurality of frames and a second frame of the plurality of frames and the means for processing is configured to generate output based on processing the first frame of the plurality of frames; and

means for combining the tracking result of the means for tracking with the output of the means for processing to produce a combined output, wherein the means for combining is configured to receive the tracking result from the means for tracking, and wherein the means for combining is further configured to receive the output from the means for processing.

29. The apparatus of claim 28, wherein the means for processing the object in the subset of frames includes means for detecting the object in the subset of frames.

30. The apparatus of claim 28, wherein the means for processing the object in the subset of frames includes means for recognizing the object in the subset of frames.

31. The apparatus of claim 28, wherein the subset of frames is selected based on a latency of the means for processing, wherein the means for tracking performs tracking with respect to at least two frames of the plurality of frames during a first period of time, and wherein the latency corresponds to the first period of time.

32. The apparatus of claim 31, wherein the means for combining is configured to update state information of the object, wherein the object corresponds to a text box, and wherein the state information includes recognized text in the text box and a location of the text box.

33. The apparatus of claim 31, further comprising means for generating the plurality of frames and means for displaying image content at least partially based on state information of the object.

34. The apparatus of claim 28, wherein the means for combining includes a Kalman filter and a maximum-likelihood estimator.

35. A non-transitory computer-readable medium including program code that, when executed by a processor, causes the processor to perform operations including:

tracking an object with respect to a plurality of frames of video data to generate a tracking result;

generating an output based on processing a subset of frames of the plurality of frames by an object processor, the object processor including an object detector or an object recognizer, wherein the object is tracked with respect to at least a first frame of the plurality of frames and a second frame of the plurality of frames and the output is generated based on processing the first frame of the plurality of frames; and

after generating the tracking result and the output, combining the tracking result with the output to produce a combined output.

36. The non-transitory computer-readable medium of claim 35, wherein the subset of frames is selected based on a first latency of the object detector or a second latency of the object recognizer, wherein the output is generated by performing object detection of the object, and wherein the subset of frames is selected based on the first latency of the object detector.

37. The non-transitory computer-readable medium of claim 35, wherein the subset of frames is selected based on a first latency of the object detector or a second latency of the object recognizer, wherein the output is generated by performing object recognition of the object, and wherein the subset of frames is selected based on the second latency of the object recognizer.

38. The non-transitory computer-readable medium of claim 35, wherein the subset of frames is selected based on a first latency of the object detector or a second latency of the object recognizer, and wherein the first latency corresponds to a period of time for a tracker to perform tracking with respect to at least two frames of the plurality of frames.

39. The non-transitory computer-readable medium of claim 38, further comprising updating state information of the object based on the combined output, wherein the object corresponds to a text box, and wherein the state information includes recognized text in the text box and a location of the text box.

40. The non-transitory computer-readable medium of claim 35, further comprising program code that, when executed by the processor, causes the processor to integrate the tracking result with the output to obtain temporal information of the object.

41. A method comprising:

tracking a region that includes text with respect to a first frame of a plurality of frames of video data to generate a tracking result;

performing text processing on the first frame, wherein the text processing includes performing text recognition on detected text data by an object processor to generate recognized text data, wherein the region is tracked with respect to at least one of the first frame and a second frame of the plurality of frames and the recognized text data is generated based on processing the first frame of the plurality of frames; and

after generating the tracking result and the recognized text data, combining the tracking result and the recognized text data to generate a combined output.

42. The method of claim 41, wherein performing the text processing on the first frame includes performing text detection on the first frame to identify the detected text data.

43. The method of claim 41, further comprising performing the text processing on the second frame of the plurality of frames to identify second recognized text data, wherein performing the text processing on the second frame includes performing text recognition on the second frame, and wherein the second frame is a frame immediately subsequent to the first frame.

44. The method of claim 41, further comprising updating state information of an object based on the combined output, wherein the object corresponds to a text box, and wherein the state information includes recognized text in the text box and a location of the text box.

45. The method of claim 41, wherein the region is tracked by a tracker that has a first latency, wherein the object processor has a second latency, and wherein the first latency is equal to the second latency.